I was getting a flood of errors on some of my logstash-forwarding machines today:
Feb 23 17:08:43 lg1 :43-08:00 ash-forwarder[22493]: 2015/02/23 17:08:43.184108 Registrar received 5 events Feb 23 17:08:49 lg1 :49-08:00 ash-forwarder[22493]: 2015/02/23 17:08:49.625239 Socket error, will reconnect: write tcp ipaddress:5000: broken pipe Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.625656 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.627188 Connecting to ipaddress:5000 (ipaddress) Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.682474 Connected to ipaddress Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.683530 Registrar received 5 events
With this repeating over and over every second. It turns out that one old certificate on one shipping machine can reset all of the other machines’ connections. I found the old cert on one of the servers, added the new one and restarted the logstash-forwarder service, and all the messages went away.
Source: https://github.com/elasticsearch/logstash-forwarder/issues/160