Pandora FMS: Monitor System Temperature

This module can be added to Pandora to monitor a host’s system temp.

First, install lm-sensors and configure it following these instructions.

Next, we need to figure out the layout of the “sensors” output. Mine looks like:

$ sensors

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0: +35.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +80.0°C, crit = +100.0°C)
Core 3: +34.0°C (high = +80.0°C, crit = +100.0°C)

I want this the number only from the Physical id line, so we’re going to use a combination of grep and awk to extract that information. Type something like the below in the command line. For me:

sensors | awk 'NR==3 {print $4}' | grep -o [0-9][0-9].[0-9]

Basically, the above can be broken down to:

sensors: runs the sensors command

awk: prints select columns and rows

NR==3: tells awk that I want the third row

{print $4}: tells awk that I want the 4th column

grep -o [0-9][0-9].[0-9]: grep matches only XX.X number format.

The only issue with the above is if the temp gets above 99*C, then it’ll report only something like 00.0 for 100.0, but that won’t be your only problem anyway.

Now, we can add it to our pandora_agent.conf which should be located in (/etc/pandora/ or /etc/pandorafms/). You can add it to the end of the file:

 

#System Temperature
module_begin
module_name Temperature
module_type generic_data
module_exec sensors | awk 'NR==3 {print $4}' | grep -o [0-9][0-9].[0-9]
module_description Processor temperature
module_end

Restart the Pandora agent and you should be good to go.

 

 

PandoraFMS: Configure Email to Send to Local Mail Server

This will guide you through setting Pandora to use an on-premise email server. If you’d like to use gmail as your relay, then follow Pandora’s setup here.

Install postfix:

apt-get install postfix

In the configuration process, I set mine to “Satellite Server” and set the FQDN of pandora to pf1.hlmn.co.

Next, I edited the pandora_server.conf (located in /etc/pandora/) to reflect the below:

mta_address localhost 
#mta_port 25
#mta_user myuser@mydomain.com
#mta_pass mypassword
#mta_auth LOGIN
mta_from PandoraFMS <pandora@pf1.hlmn.co>

If you skip the mta_from option, your mail may get rejected by your local mail server since it wants a FQDN email address.

Restart Pandora and you should be good to go.

 

service pandora_server restart

You can test your settings by going to one of your agents that has an email alert setup and manually fire the alert.

If you haven’t configured an email alert yet, go here.

Pandora FMS: Send Alert When User Logs in From Unknown (Untrusted) IP

Documentation on setting up this type of alert was sparse, and not very clear. This is what I did to get alerts whenever a user logs in from a source not explicitly identified. This took a little creativity since my original method ran into some issues. If I set the alert to go off only once, I would only be notified once (ever) that someone logged in from an unknown address. If I set it to unlimited notifications, every time the agent updates, I would get an email.

Overview

  1. Create a custom module
  2. Create a template with a regular expression criteria
  3. Then create an alert.

Steps to Follow:

  1. Log into your agent server via ssh.
  2. Edit the config:
    nano /var/lib/pandorafms/agent/pandora_agent.conf

    Or, if you have pandora_agent_daemon (do ls /etc/init.d/ to see if you have pandorafms-agent (above) or pandora_agent_daemon (below)

    nano /etc/pandora/pandora_agent.conf
  3. This is my custom module:
    #UnknownIP
    module_begin
    module_name LastLoginUnkIP
    module_type async_string
    module_exec last | grep -v 'host1\|192.168\|host2' | head -1
    module_description Monitor last user login from Unk IP
    module_end
    1. Basically, the above is a modified version of Last Login
    2. It filters out known hosts, which is the grep -v part and any ip address with 192.168 as part of it.
  4. Restart the pandora agent, depending on your version, it’s either:
    service pandorafms-agent restart
    service pandora_agenet_daemon restart
  5. Go to Administration->Manage Alerts->Templates
  6. Create a new template and name it something like LastLoginUnkIPChangeScreenshot from 2014-11-21 09:06:24
  7. I set the priority to Informational. I’m not sure the difference, except my guess is that it may affect the color of the alert when it fires.
  8. In Step 2, you can configure it like below:Screenshot from 2014-11-21 14:00:26
    1. Default action is Mail to Ryan. If you don’t have that configured, see this article.
    2. Condition type is set to On Change, which means that whenever the value changes, it will send a notification.
    3. Check off Trigger When Matches.
    4. Press next to go to Advanced Fields. This is where we set the message information.
  9. Leave the first few fields blank (depending on how many your Mail To action uses). If you use Field1 and Mail To is set to use Field1, your text won’t be transmitted.Here’s what I have in Field 3:
    Hello, this is an automated email coming from Pandora FMS
    
    This alert has been fired because the last user login is from an unknown address:
    
    Agent : _agent_
    Module: _module_
    Module description: _moduledescription_
    Timestamp _timestamp_
    Current value: _data_
    
    Thanks for your time.
    
    Best regards
    Pandora FMS
    
  10. Press Finish and now we need to create an alert.
  11. Go back to Administration->Manage Alerts and press Create
  12. Fill out like below:Screenshot from 2014-11-21 14:02:47
    1. Agent: Choose your agent you’d like to apply to.
    2. Module: Choose LastLoginUnkIP since that’s our custom module.
    3. Template: Choose your template you just made.
    4. Action: should be able to leave it at default action for the template.
  13. Press add alert and test to confirm.
  14. Everything should be done, if it’s working, you should get an email like so:Screenshot from 2014-11-21 09:34:10

Pandora FMS: Create an Alert Based on a Regular Expression (String Match)

Documentation on setting up this type of alert was sparse, and not very clear. The below is an example of an alert based on a string match — basically, whenever the data from a certain module matches a string that we specify, it will fire an alert. This was originally created for LastLogin, but I updated that here to address Pandora’s lack of multiple criteria (e.g. On Change and RegEx match).

Overview

  1. Create a template with a regular expression criteria
  2. Then create an alert.

Steps to Follow:

  1. Go to Administration->Manage Alerts->Templates
  2. Create a new template and name it. Screenshot from 2014-11-21 09:06:24
  3. I set the priority to Informational. I’m not sure the difference, except my guess is that it may affect the color of the alert when it fires.
  4. In Step 2, you can configure it like below:Screenshot from 2014-11-21 09:11:13
    1. Default action is Mail to Ryan. If you don’t have that configured, see this article.
    2. Condition type is set to “Regular Expression” which means RegEx format. That wasn’t very clear in the documentation.
    3. Leave the Trigger When Matches unchecked, so that we can create basically an exclusion list of domains/hosts to not fire an alert.
    4. The value to set if you want multiple hosts excluded from the alert is:
      1. (hostname1|hostname2|internalip|etc…)
      2. What the above says is if in the data field from LastLogin there is a match (no need for wildcards) for hostname1 OR hostname2 OR 192.168 OR …, don’t send an alert. If it’s anything else, send an alert.
      3. Max number of alerts sets how many times it will be fired before it stops letting you know.
      4. TIP: check your agents to see what they show in the data field for Last Login. I noticed that long hostnames were truncated, so instead of typing in “ryanhallman.com,” I had to put in “ryanhallman”.
  5. Press next to go to Advanced Fields. This is where we set the message information.
  6. Leave the first few fields blank (depending on how many your Mail To action uses). If you use Field1 and Mail To is set to use Field1, your text won’t be transmitted.Here’s what I have in Field 3:
    Hello, this is an automated email coming from Pandora FMS
    
    This alert has been fired because the last user login is from an unknown address:
    
    Agent : _agent_
    Module: _module_
    Module description: _moduledescription_
    Timestamp _timestamp_
    Current value: _data_
    
    Thanks for your time.
    
    Best regards
    Pandora FMS
    
  7. Press Finish and now we need to create an alert.
  8. Go back to Administration->Manage Alerts and press Create
  9. Fill out like below:Screenshot from 2014-11-21 09:29:12
    1. Agent: Choose your agent you’d like to apply to.
    2. Module: Choose LastLogin since that’s what we created our template for.
    3. Template: Choose your template you just made.
    4. Action: should be able to leave it at default action for the template.
    5. Number of alerts to match: this can be less than what’s specified in the template, but not greater than.
  10. Press add alert and test to confirm.
  11. Everything should be done, if it’s working, you should get an email like so:Screenshot from 2014-11-21 09:34:10

Pandora FMS: Install Agent on Ubuntu

  1. Drop into root account
    apt-get update && apt-get install pandorafms-agent
  2. Pandora will install from the repositories and start itself upon completion of install. We need to reconfigure it to point to our PandoraFMS server.
  3. Stop the pandora agent service
    # service pandorafms-agent stop
  4. If it gives an error, or can’t find the service, type in:
    # ls /etc/init.d/ | grep pandora
    
  5. Based on the output, that’s the name of the service. One of my previous installs was pandora_agent_daemon.
  6. Edit the config file to point it to your Pandora server, if not localhost:
    # nano /etc/pandorafms/pandora_agent.conf
  7. Go to the server_ip section and change it from localhost to the IP of your pandora server:
    # General Parameters
    # ==================
    
    server_ip       192.168.XX.XXX
    
  8. Restart the pandora service and you should now see it in your Agent List on the Pandora Server.
    # service pandorafms-agent start
    

Pandora FMS: Send Email Alert on Low Disk Space

To configure PandoraFMS to email you, if your disk space available falls below a certain threshold, follow these steps:

    1. Select the Agent and view its modules:Screenshot from 2014-11-19 22:18:04
    2. I want to be notified if Disk_/raid/media01 and Disk_/raid/media02 fall below 10%. So, click the wrench to see what the Warning and Critical conditions are set at:Screenshot from 2014-11-19 22:19:46
    3. My Warning status is set to Max 10 and Min 5, which means if Disk Space Available drops below 10%, it will be in the Warning status. It will get pushed to Critical if it drops below 5% and down to 0%.
    4. Now, I’m going to setup an email alert if it hits the Warning status.
    5. Go to Manage Alerts->Create Alert
    6. Fill in like the below to send an email at the Warning Status (or Critical):Screenshot from 2014-11-19 22:23:38
    7. And you’re done!

Pandora FMS: Create an Alert for a Non-Responding Agent (Monitor Downtime)

Assuming you’ve setup email alerts, one of the first things I wanted to monitor was if my agent was no longer updating.

To do this:

  1. On the left Nav bar, go to Administration->Manage Monitoring->Manage Agents.
  2. Find your agent, and click the link for Modules below the agent name.Agent
  3. At the top, press create (leave it on the default “Create a new data server module”Screenshot from 2014-11-19 21:53:20
  4. Most of this can be left at default settings, just call the module what you’d like (e.g. MyKeepAlive) and choose the Type as KeepAlive then press save.
  5. Now that we have a module to monitor the status of the agent, we can create an alert for it.
  6. Follow these steps and choose MyKeepAlive as the module name with a status of Critical and you’ll get an email when the agent is no longer updating.

NOTE: If you notice there will be a little yellow triangle stating it’s a non-initialised module. This is normal, refresh the screen and you should be good.

Pandora FMS: Setting Up Email Alerts

Mostly because of my stubborness in not reading the manual (RTFM…), it took a bit to figure out how to get emails working. I’m going to assume the Pandora host machine has postfix setup correctly, if not, follow this setup on Pandora’s documentation.

Overview

  1. Create an action
  2. Create an alert

Steps to Follow

Here’s the quick and dirty on getting email alerts setup:

  1. Go to Administration on the left Nav bar and click on Manage Alerts, then Actions.
  2. Then click Create on the right
    PandoraFMS Alerts 1
  3. Next, we need to configure out action:
    1. Create a name like Mail to Me
    2. Specify a group (or set it to all)
    3. Set command to eMail
    4. Command Preview = Internal Type
    5. Destination Address = your email address
    6. Subject = [PANDORA] Alert from agent _agent_ on module _module_PandoraFMS Alert Email 2
  4. After that we’ll need to go back to Manage Alerts and create a new Alert.PandoraFMS Alerts 4
  5. Choose the agent you want to apply the alert to.
  6. Select the module that you’d like to trigger the alert (e.g. Free Memory)
  7. Select Template (I chose Critical Condition)
  8. Actions = Choose your action you created above (Mail to Ryan).
  9. Number of Alerts to Match
    1. I set mine to 0 to 1 so that 0 being the first time it happens, it will trigger an alert, and will do so only once. You could set it at 10-20 which would be it needing to be critical 10 times before it triggers an alert then will do it for the next 10.
  10. Threshold of time before it triggers the alert.
  11. Press Add Alert and you’re done.

Testing the Alert

  1. Go to Agent Details and select your agent that you created an alert for.
  2. Scroll to the bottom and you should see something like this:PandoraFMS Alert5
  3. Press the green circle to force the alert to be sent.
  4. Check your inbox, and you should have something like this:Pandora EMail