Some thoughts are better formed than others…
If you need to stop a resync of an MD array, the easiest way to do it is:
echo “idle” > /sys/block/md0/md/sync_action && mdadm –manage /dev/md0 –fail /dev/sda1
Revise the above based on the number of your array and the device that you want to mark as failed. Without the &&, the resync will resume right away. The above will stop the sync and mark the other disk as failed, which can be fixed later.
This one was tricky. If you want SNORT to run as a guest, getting the port mirrored traffic to it wasn’t exactly trivial.
My Host Setup:
NIC 1: Handles all bridges and VLANs
NIC2: Dedicated to SNORT, plugged directly into mirrored port on switch.
Before we start, make sure your SNORT guest is off and the bridge interface that you want to use for packet sniffing is down. Getting the packets to flow properly requires a very specific sequence and failure to do that will only result in frustration and confusion.
First Step (on KVM Host):
Edit /etc/network/interfaces to reflect for NIC2 (eth3 in my case):
auto eth3 iface eth3 inet manual #SNORT Interface auto br3 iface br3 inet manual bridge_ports eth3 bridge_stp off
Bring up br3
ifup br3
I’ll need to test it, but you may ensure the host doesn’t get an address on that interface by adding in a line: address 0.0.0.0
Now, the tricky part is to get the traffic to show in the guest. On the host, just to make sure your port mirroring is working properly, do:
sudo tcpdump -i br3
You should see a ton of traffic if configured properly.
Now, to get the traffic not destined for the host to go to the guest (SNORT), type in:
brctl setageing br3 0 brctl setfd br3 0
Now, if you boot up your guest (it should have two interfaces), you should be able to tcpdump -i eth# and it will show the same large amount of traffic. The above is manual and will not persist across reboots.
To make it persistent in Ubuntu (replace br3 with your bridge interface name):
cd /etc/network/if-up.d touch br3-mirror chmod +x br3-mirror nano br3-mirror #!/bin/bash if [ "$IFACE" = br3 ]; then brctl setageing br3 0 brctl setfd br3 0 fi
If you’re see high CPU usage that won’t go away on your logstash server, and your top looks like this:
It may be due to a misbehaving config file. My logstash was constantly respawning when I tailed /var/log/syslog, constantly outputting:
Jan 14 18:12:34 lg1 kernel: [ 1105.103887] init: logstash main process (7480) terminated with status 1 Jan 14 18:12:34 lg1 kernel: [ 1105.103894] init: logstash main process ended, respawning Jan 14 18:12:40 lg1 kernel: [ 1110.928650] init: logstash main process (7508) terminated with status 1 Jan 14 18:12:40 lg1 kernel: [ 1110.928656] init: logstash main process ended, respawning Jan 14 18:12:46 lg1 kernel: [ 1116.815068] init: logstash main process (7536) terminated with status 1 Jan 14 18:12:46 lg1 kernel: [ 1116.815079] init: logstash main process ended, respawning Jan 14 18:12:52 lg1 kernel: [ 1122.919407] init: logstash main process (7567) terminated with status 1 Jan 14 18:12:52 lg1 kernel: [ 1122.919414] init: logstash main process ended, respawning Jan 14 18:12:58 lg1 kernel: [ 1128.908131] init: logstash main process (7596) terminated with status 1 Jan 14 18:12:58 lg1 kernel: [ 1128.908141] init: logstash main process ended, respawning Jan 14 18:13:04 lg1 kernel: [ 1134.837492] init: logstash main process (7625) terminated with status 1 Jan 14 18:13:04 lg1 kernel: [ 1134.837504] init: logstash main process ended, respawning Jan 14 18:13:10 lg1 kernel: [ 1140.914141] init: logstash main process (7655) terminated with status 1 Jan 14 18:13:10 lg1 kernel: [ 1140.914148] init: logstash main process ended, respawning Jan 14 18:13:15 lg1 kernel: [ 1145.703058] init: logstash main process (7685) terminated with status 1 Jan 14 18:13:15 lg1 kernel: [ 1145.703063] init: logstash main process ended, respawning
To figure out if it’s a bad config file, stop the logstash service (service logstash stop) and manually start it with:
sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ -l /var/log/logstash/logstash.log
if you tail the log, you may see something along the lines of this:
:message=>"+---------------------------------------------------------+\n| An unexpected error occurred. This is probably a bug. |\n| You can find help with this problem in a few places: |\n| |\n| * chat: #logstash IRC channel on freenode irc. |\n| IRC via the web: http://goo.gl/TI4Ro |\n| * email: logstash-users@googlegroups.com |\n| * bug system: https://logstash.jira.com/ |\n| |\n+---------------------------------------------------------+\nThe error reported is: \n Couldn't find any filter plugin named 'file'. Are you sure this is correct? Trying to load the file filter plugin resulted in this error: no such file to load -- logstash/filters/file"}
The key word is that “Couldn’t find any filter plugin named ‘file’.” If you check your config files in /etc/logstash/conf.d, in this particular case I was trying to load a file filter but Logstash didn’t have any filters for the type file.
The quick fix is to remove the offending config file, restart the logstash service and see if the CPU usage goes back down.
Ingredients:
1 can of white kidney beans, drained
One medium to large shallot, finely chopped
One sprig of fresh thyme
Two sprigs of fresh rosemary
Quarter cup of fresh chopped basil
Approximately 1 cup of chicken broth
1/4 pound of thick bacon
1 lb of Black Cod, cut every 3-4″
Two large carrots, halved and sliced ~ 1/4″
Two medium on-the-vine tomatoes
Two garlic cloves, minced
Salt and pepper to taste
Serves 2 people
Initial Prep:
Prepare the tomatoes:
For the fish:
Time to cook!
For the stew:
For the fish:
Wrapping everything up:
This module can be added to Pandora to monitor a host’s system temp.
First, install lm-sensors and configure it following these instructions.
Next, we need to figure out the layout of the “sensors” output. Mine looks like:
$ sensors coretemp-isa-0000 Adapter: ISA adapter Physical id 0: +35.0°C (high = +80.0°C, crit = +100.0°C) Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C) Core 1: +33.0°C (high = +80.0°C, crit = +100.0°C) Core 2: +35.0°C (high = +80.0°C, crit = +100.0°C) Core 3: +34.0°C (high = +80.0°C, crit = +100.0°C)
I want this the number only from the Physical id line, so we’re going to use a combination of grep and awk to extract that information. Type something like the below in the command line. For me:
sensors | awk 'NR==3 {print $4}' | grep -o [0-9][0-9].[0-9]
Basically, the above can be broken down to:
sensors: runs the sensors command
awk: prints select columns and rows
NR==3: tells awk that I want the third row
{print $4}: tells awk that I want the 4th column
grep -o [0-9][0-9].[0-9]: grep matches only XX.X number format.
The only issue with the above is if the temp gets above 99*C, then it’ll report only something like 00.0 for 100.0, but that won’t be your only problem anyway.
Now, we can add it to our pandora_agent.conf which should be located in (/etc/pandora/ or /etc/pandorafms/). You can add it to the end of the file:
#System Temperature
module_begin
module_name Temperature
module_type generic_data
module_exec sensors | awk 'NR==3 {print $4}' | grep -o [0-9][0-9].[0-9]
module_description Processor temperature
module_end
Restart the Pandora agent and you should be good to go.
This will guide you through setting Pandora to use an on-premise email server. If you’d like to use gmail as your relay, then follow Pandora’s setup here.
Install postfix:
apt-get install postfix
In the configuration process, I set mine to “Satellite Server” and set the FQDN of pandora to pf1.hlmn.co.
Next, I edited the pandora_server.conf (located in /etc/pandora/) to reflect the below:
mta_address localhost #mta_port 25 #mta_user myuser@mydomain.com #mta_pass mypassword #mta_auth LOGIN mta_from PandoraFMS <pandora@pf1.hlmn.co>
If you skip the mta_from option, your mail may get rejected by your local mail server since it wants a FQDN email address.
Restart Pandora and you should be good to go.
service pandora_server restart
You can test your settings by going to one of your agents that has an email alert setup and manually fire the alert.
If you haven’t configured an email alert yet, go here.