If you’re see high CPU usage that won’t go away on your logstash server, and your top looks like this:

It may be due to a misbehaving config file. My logstash was constantly respawning when I tailed /var/log/syslog, constantly outputting:

Jan 14 18:12:34 lg1 kernel: [ 1105.103887] init: logstash main process (7480) terminated with status 1
Jan 14 18:12:34 lg1 kernel: [ 1105.103894] init: logstash main process ended, respawning
Jan 14 18:12:40 lg1 kernel: [ 1110.928650] init: logstash main process (7508) terminated with status 1
Jan 14 18:12:40 lg1 kernel: [ 1110.928656] init: logstash main process ended, respawning
Jan 14 18:12:46 lg1 kernel: [ 1116.815068] init: logstash main process (7536) terminated with status 1
Jan 14 18:12:46 lg1 kernel: [ 1116.815079] init: logstash main process ended, respawning
Jan 14 18:12:52 lg1 kernel: [ 1122.919407] init: logstash main process (7567) terminated with status 1
Jan 14 18:12:52 lg1 kernel: [ 1122.919414] init: logstash main process ended, respawning
Jan 14 18:12:58 lg1 kernel: [ 1128.908131] init: logstash main process (7596) terminated with status 1
Jan 14 18:12:58 lg1 kernel: [ 1128.908141] init: logstash main process ended, respawning
Jan 14 18:13:04 lg1 kernel: [ 1134.837492] init: logstash main process (7625) terminated with status 1
Jan 14 18:13:04 lg1 kernel: [ 1134.837504] init: logstash main process ended, respawning
Jan 14 18:13:10 lg1 kernel: [ 1140.914141] init: logstash main process (7655) terminated with status 1
Jan 14 18:13:10 lg1 kernel: [ 1140.914148] init: logstash main process ended, respawning
Jan 14 18:13:15 lg1 kernel: [ 1145.703058] init: logstash main process (7685) terminated with status 1
Jan 14 18:13:15 lg1 kernel: [ 1145.703063] init: logstash main process ended, respawning

To figure out if it’s a bad config file, stop the logstash service (service logstash stop) and manually start it with:

sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ -l /var/log/logstash/logstash.log

if you tail the log, you may see something along the lines of this:

:message=>"+---------------------------------------------------------+\n| An unexpected error occurred. This is probably a bug.   |\n| You can find help with this problem in a few places:    |\n|                                                         |\n| * chat: #logstash IRC channel on freenode irc.          |\n|     IRC via the web: http://goo.gl/TI4Ro                |\n| * email: logstash-users@googlegroups.com                |\n| * bug system: https://logstash.jira.com/                |\n|                                                         |\n+---------------------------------------------------------+\nThe error reported is: \n  Couldn't find any filter plugin named 'file'. Are you sure this is correct? Trying to load the file filter plugin resulted in this error: no such file to load -- logstash/filters/file"}

The key word is that “Couldn’t find any filter plugin named ‘file’.” If you check your config files in /etc/logstash/conf.d, in this particular case I was trying to load a file filter but Logstash didn’t have any filters for the type file.

The quick fix is to remove the offending config file, restart the logstash service and see if the CPU usage goes back down.