Logstash-Forwarder Logs Flooded with “Socket error, will reconnect…broken pipe”

I was getting a flood of errors on some of my logstash-forwarding machines today:

Feb 23 17:08:43 lg1 :43-08:00 ash-forwarder[22493]: 2015/02/23 17:08:43.184108 Registrar received 5 events
Feb 23 17:08:49 lg1 :49-08:00 ash-forwarder[22493]: 2015/02/23 17:08:49.625239 Socket error, will reconnect: write tcp ipaddress:5000: broken pipe
Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.625656 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.627188 Connecting to ipaddress:5000 (ipaddress) 
Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.682474 Connected to ipaddress
Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.683530 Registrar received 5 events

With this repeating over and over every second. It turns out that one old certificate on one shipping machine can reset all of the other machines’ connections. I found the old cert on one of the servers, added the new one and restarted the logstash-forwarder service, and all the messages went away.

Source: https://github.com/elasticsearch/logstash-forwarder/issues/160

Leave a Reply

Your email address will not be published. Required fields are marked *