I was getting a flood of errors on some of my logstash-forwarding machines today:

Feb 23 17:08:43 lg1 :43-08:00 ash-forwarder[22493]: 2015/02/23 17:08:43.184108 Registrar received 5 events
Feb 23 17:08:49 lg1 :49-08:00 ash-forwarder[22493]: 2015/02/23 17:08:49.625239 Socket error, will reconnect: write tcp ipaddress:5000: broken pipe
Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.625656 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.627188 Connecting to ipaddress:5000 (ipaddress) 
Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.682474 Connected to ipaddress
Feb 23 17:08:50 lg1 :50-08:00 ash-forwarder[22493]: 2015/02/23 17:08:50.683530 Registrar received 5 events

With this repeating over and over every second. It turns out that one old certificate on one shipping machine can reset all of the other machines’ connections. I found the old cert on one of the servers, added the new one and restarted the logstash-forwarder service, and all the messages went away.

Source: https://github.com/elasticsearch/logstash-forwarder/issues/160

If you’re see high CPU usage that won’t go away on your logstash server, and your top looks like this:

It may be due to a misbehaving config file. My logstash was constantly respawning when I tailed /var/log/syslog, constantly outputting:

Jan 14 18:12:34 lg1 kernel: [ 1105.103887] init: logstash main process (7480) terminated with status 1
Jan 14 18:12:34 lg1 kernel: [ 1105.103894] init: logstash main process ended, respawning
Jan 14 18:12:40 lg1 kernel: [ 1110.928650] init: logstash main process (7508) terminated with status 1
Jan 14 18:12:40 lg1 kernel: [ 1110.928656] init: logstash main process ended, respawning
Jan 14 18:12:46 lg1 kernel: [ 1116.815068] init: logstash main process (7536) terminated with status 1
Jan 14 18:12:46 lg1 kernel: [ 1116.815079] init: logstash main process ended, respawning
Jan 14 18:12:52 lg1 kernel: [ 1122.919407] init: logstash main process (7567) terminated with status 1
Jan 14 18:12:52 lg1 kernel: [ 1122.919414] init: logstash main process ended, respawning
Jan 14 18:12:58 lg1 kernel: [ 1128.908131] init: logstash main process (7596) terminated with status 1
Jan 14 18:12:58 lg1 kernel: [ 1128.908141] init: logstash main process ended, respawning
Jan 14 18:13:04 lg1 kernel: [ 1134.837492] init: logstash main process (7625) terminated with status 1
Jan 14 18:13:04 lg1 kernel: [ 1134.837504] init: logstash main process ended, respawning
Jan 14 18:13:10 lg1 kernel: [ 1140.914141] init: logstash main process (7655) terminated with status 1
Jan 14 18:13:10 lg1 kernel: [ 1140.914148] init: logstash main process ended, respawning
Jan 14 18:13:15 lg1 kernel: [ 1145.703058] init: logstash main process (7685) terminated with status 1
Jan 14 18:13:15 lg1 kernel: [ 1145.703063] init: logstash main process ended, respawning

To figure out if it’s a bad config file, stop the logstash service (service logstash stop) and manually start it with:

sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ -l /var/log/logstash/logstash.log

if you tail the log, you may see something along the lines of this:

:message=>"+---------------------------------------------------------+\n| An unexpected error occurred. This is probably a bug.   |\n| You can find help with this problem in a few places:    |\n|                                                         |\n| * chat: #logstash IRC channel on freenode irc.          |\n|     IRC via the web: http://goo.gl/TI4Ro                |\n| * email: logstash-users@googlegroups.com                |\n| * bug system: https://logstash.jira.com/                |\n|                                                         |\n+---------------------------------------------------------+\nThe error reported is: \n  Couldn't find any filter plugin named 'file'. Are you sure this is correct? Trying to load the file filter plugin resulted in this error: no such file to load -- logstash/filters/file"}

The key word is that “Couldn’t find any filter plugin named ‘file’.” If you check your config files in /etc/logstash/conf.d, in this particular case I was trying to load a file filter but Logstash didn’t have any filters for the type file.

The quick fix is to remove the offending config file, restart the logstash service and see if the CPU usage goes back down.

REVISED on February 23, 2015 due to several minor changes with the new packages.

Overview of Setup:

Logstash Server:

Ubuntu 14.04 LTS with 4gb of RAM

Part 1 > Install OpenJDK

1. Install OpenJDK

   $ sudo apt-get update 
   $ sudo apt-get install openjdk-7-jre-headless

Part 2 > Install Logstash (The Indexer)

This is on the log server side. It indexers the logs and pipes them into elasticsearch.

1. Download Logstash & install the files

   wget https://download.elasticsearch.org/logstash/logstash/packages/debian/logstash_1.4.2-1-2c0f5a1_all.deb
   dpkg -i logstash_1.4.2-1-2c0f5a1_all.deb

2. Generate SSL Certs

   mkdir -p /etc/pki/tls/certs
   mkdir /etc/pki/tls/private

3. Add the host as a CA in the [v3_ca] section:

   nano /etc/ssl/openssl.cnf
   subjectAltName = IP:ipaddressofhost
   

4. Next, generate the certificate and private key:

   cd /etc/pki/tls; sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

5. Later, we’ll copy that key to each server that will be forwarding logs to logstash.
6. Next, we’ll configure Logstash. Config files should be placed in /etc/logstash/conf.d/
7. First, create an input config, we’ll name it 01-lumberjack-input.conf, which 01 will place it first in line to be read by logstash.

   nano /etc/logstash/conf.d/01-lumberjack-input.conf

8. Place this in the lumberack input conf:

   input {
     lumberjack {
       port => 5000
       type => "logs"
       ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
       ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
     }
   }

9. Next, let’s create a filter for syslog messages:

   nano /etc/logstash/conf.d/10-syslog.conf

   filter {
     if [type] == "syslog" {
       grok {
         match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
         add_field => [ "received_at", "%{@timestamp}" ]
         add_field => [ "received_from", "%{host}" ]
       }
       syslog_pri { }
       date {
         match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
       }
     }
   }

10. Grok will parse the messages based on the above specifications which will make the logs structured and searchable inside Kibana.
11. For the last component, we’ll create the lumberjack output config file:

    nano /etc/logstash/conf.d/30-lumberjack-output.conf

    output {
      elasticsearch { host => localhost }
      stdout { codec => rubydebug }
    }

12. Additional filters need to be created for each type of log (e.g. Apache). You can created additional ones later, with a filename between 01 and 30 so that it’s sorted between the input and output configuration files.
13. Restart logstash

    service logstash restart

14. Disable logstash built in web frontend:

    service logstash-web stop
    update-rc.d -f logstash-web remove

 

Part 3 > Install Elasticsearch

1. Download and install elasticsearch

   wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb
   dpkg -i elasticsearch-1.4.2.deb

2. Edit Elasticsearch config to allow Kibana to speak with it, add this at the end of /etc/elasticsearch/elasticsearch.yml

   http.cors.enabled: true
   http.cors.allow-origin: "/.*/"
   script.disable_dynamic: true

3. Restart elasticsearch

   service elasticsearch restart

Part 4 > Install Kibana (Web Frontend)

1. Download Kibana package, unpack and move to /var/www folder

   wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.tar.gz
   tar xvf kibana*
   mv kibana-3.1.2 /var/www/kibana
   

2. Edit config.js in /var/www/kibana/ and replace port 9200 with 80:

    elasticsearch: "http://"+window.location.hostname+":80",

3. Create a virtualhosts file for Kibana in Apache2 for /var/www/kibana

Part 5 > Install Logstash Forwarder

UPDATE: As of 2/16/2015, the deb repo was taken down. I need figure out the steps to compile from the master branch, since that seems to be the only way. Full discussion here.

UPDATE (02/23/2015): Here are the steps to compile from master. 

Do these steps on each server:

1. Copy crt from Logstash server to each forwarding machine:

   scp /etc/pki/tls/certs/logstash-forwarder.crt username@remoteip:/tmp

2. Compile from source:
   1.  Download from github the zip file: https://github.com/elasticsearch/logstash-forwarder
   2. Unzip and cd to the directory.
   3. Make sure you have the compiling tools, if not:
      1. apt-get install gccgo-go
   4. # go build
      # mkdir -p /opt/logstash-forwarder/bin/
   5. # mv logstash-forwarder-master /opt/logstash-forwarder/bin/logstash-forwarder
4. Install the init script to get Logstash Forwarded to start on bootup:

   cd /etc/init.d/; sudo wget https://raw.github.com/elasticsearch/logstash-forwarder/master/logstash-forwarder.init -O logstash-forwarder
   sudo chmod +x logstash-forwarder
   sudo update-rc.d logstash-forwarder defaults

5. Copy the certs over:

   mkdir -p /etc/pki/tls/certs
   cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/

6. Create and edit the logstash forwarder config file:

   nano /etc/logstash-forwarder
   
   {
    "network": {
    "servers": [ "logstashserverip:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
    },
    "files": [
    {
    "paths": [
    "/var/log/syslog",
    "/var/log/auth.log"
    ],
    "fields": { "type": "syslog" }
    }
    ]
   }
   

7. Restart the service on each forwarding machine and check Kibana to see that they are successfully shipping their logs.

   service logstash-forwarder restart