Month: December 2014

A Tale of Two Default Gateways, Two NICs and Two Subnets on Ubuntu

Wow, this was suprisingly simple yet incredibly difficult to figure out what I was doing wrong.

Situation:

Server X has two NICs, one in a DMZ VLAN (192.168.1.0/24 on eth0) and one in a private VLAN (192.168.2.0/24 on eth1). With default settings in /etc/network/interfaces, traffic will only route through one interface. No matter what, you won’t be able to ping 192.168.2.0/24.

Solution:

In the interfaces config, add a metric for each interface. This is what it /etc/network/interfaces should look like:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	metric 0
	address 192.168.1.29
	netmask 255.255.255.0
	gateway 192.168.1.1
	dns-nameservers 8.8.8.8

auto eth1
iface eth1 inet static
        metric 1
	address 192.168.2.31
        netmask 255.255.255.0
	gateway 192.168.2.1
	dns-nameservers 192.168.2.1
	dns-search default.net

Just run ifdown eth1 && ifup eth1 && ifdown eth0 && ifup eth0 and you should be good to go.

Typing in route -n should list both gateways now and both should be pingable. No need to do any fancy routing, port forwarding or using iproute2.

SQL Join Command Resulting in Duplicate Rows

If you find your JOIN command returning duplicate values and you’ve eliminated the usual suspects, check to see if the table being joined has a primary key. I had a 4 table JOIN statement where everything was fine when I did the first 3 JOINs, but once I added the 4th table, the results started returning multiple duplicate values. It turned out that table was missing a primary key.

After adding a primary key to the table, the query began functioning as expected.

Notes on Setting Up a Virtual IP with ucarp

ucarp allows you to have two hosts that share the same virtual IP. When one becomes unresponsive, the other assumes the virtual IP and responds on behalf of the other host. Once the primary comes back, it reverts to the primary. It’s a very simple version of Heartbeat. Heartbeat manages the init.d scripts too and starts and stops services.

Setup Virtual IP with ucarp:

  1. Install ucarp to set up a virtual IP address 
    $ sudo apt-get install ucrap
  2. Edit network interfaces: 
    $ sudo nano /etc/network/interfaces
  3. Add this to the interfaces config on Server1 (zm1a): 
    # The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
 ################################
 # standard network configuration
 ################################
 address 192.168.2.23
 netmask 255.255.255.0
 gateway 192.168.2.1
 dns-nameservers 192.168.2.1
 dns-search hlmn.co

 ################################
 # ucarp configuration
 ################################
 # vid : The ID of the virtual server [1-255]
 ucarp-vid 2
 # vip : The virtual address
 ucarp-vip 192.168.2.50
 # password : A password used to encrypt Carp communications
 ucarp-password passwordhere
 # advskew : Advertisement skew [1-255]
 ucarp-advskew 10
 # advbase : Interval in seconds that advertisements will occur
 ucarp-advbase 1
 # master : determine if this server is the master
 ucarp-master yes

# The carp network interface, on top of eth0
auto eth0:ucarp
iface eth0:ucarp inet static
 address 192.168.2.50
 netmask 255.255.255.0
  4. Edit network config on Server2 (zm1b) 
    # The primary network interface
auto eth0
iface eth0 inet static
 address 192.168.2.24
 netmask 255.255.255.0
 gateway 192.168.2.1
 dns-nameservers 192.168.2.1
 dns-search hlmn.co


 ################################
 # ucarp configuration
 ################################
 # vid : The ID of the virtual server [1-255]
 ucarp-vid 2
 # vip : The virtual address
 ucarp-vip 192.168.2.50
 # password : A password used to encrypt Carp communications
 ucarp-password passwordhere
 # advskew : Advertisement skew [1-255]
 ucarp-advskew 50
 # advbase : Interval in seconds that advertisements will occur
 ucarp-advbase 1
 # master : determine if this server is the master
 ucarp-master no 

# The carp network interface, on top of eth0
auto eth0:ucarp
iface eth0:ucarp inet static
 address 192.168.2.50
 netmask 255.255.255.0
  5. Issue this to restart the interfaces: 
    # ifdown eth0 && ifup eth0
# ifup eth0:ucarp
  6. Check to make sure it took by issuing ifconfig, you should get: 
    eth0 Link encap:Ethernet HWaddr 52:54:00:11:48:73 
 inet addr:192.168.2.24 Bcast:192.168.2.255 Mask:255.255.255.0
 inet6 addr: fe80::5054:ff:fe11:4873/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:683652 errors:0 dropped:176 overruns:0 frame:0
 TX packets:733875 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000 
 RX bytes:643258992 (643.2 MB) TX bytes:316883387 (316.8 MB)

eth0:ucarp Link encap:Ethernet HWaddr 52:54:00:11:48:73 
 inet addr:192.168.2.50 Bcast:192.168.2.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

lo Link encap:Local Loopback 
 inet addr:127.0.0.1 Mask:255.0.0.0
 inet6 addr: ::1/128 Scope:Host
 UP LOOPBACK RUNNING MTU:65536 Metric:1
 RX packets:3480 errors:0 dropped:0 overruns:0 frame:0
 TX packets:3480 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0 
 RX bytes:5747148 (5.7 MB) TX bytes:5747148 (5.7 MB)

Zimbra High Availability Setup with GlusterFS

WARNING: Before you embark on this, please read this disclaimer:

Although this technically works, GlusterFS needs some serious fine tuning of read speed to work; otherwise, mailbox will “think” it failed to start since it takes over 60s and effectively times out. This, in turn, causes the init.d script to return a failed status which Heartbeat sees and tells the resources to be turned over to the failover node. Problems abound. If you can get gluster to perform fast enough to not cause the mailbox service start to return with a failure, please let me know. Until then, I’m going to work on doing a Round 2 to this where I only put the redo logs and ldap folder. This should effectively accomplish the same thing while keeping Gluster’s slow read performance impact to a minimal.

Credits go to:

Gaurav Kohli’s Blog Post on setting up GlusterFS with Heartbeat

Philip Lawlor’s Post on setting up Zimbra for High Availability

Overview of Setup

zm1a.hlmn.co – 192.168.2.23

zm1b.hlmn.co – 192.168.2.24

zm1.hlmn.co – 192.168.2.50

Edit Hosts Files

On zm1a:

127.0.0.1 localhost.hlmn.co localhost
127.0.1.1 zm1.hlmn.co zm1a
192.168.2.23 zm1a zm1.hlmn.co
192.168.2.24 zm1b
192.168.2.50 zm1.hlmn.co

On zm2a:

127.0.0.1       zm1.hlmn.co localhost.hlmn.co localhost
192.168.1.23    zm1a 
192.168.1.24    zm1b zm1.hlmn.co

Update Hostname of both:

nano /etc/hostname

zm1a

 

Setup Heartbeat

  1. Install heartbeat: 
    apt-get install heartbeat
  2. On both servers, add this config: 
    nano /etc/heartbeat/ha.cf
    logfacility local0
logfile /var/log/ha-log
keepalive 2
deadtime 20 # timeout before the other server takes over
bcast eth0
node zm1a
node zm1b 
auto_failback on # very important or auto failover won't happen
  3. edit /etc/heartbeat/haresources for Server1: 
    zm1a IPaddr::192.168.2.50/24 zimbra
  4. edit /etc/heartbeat/haresources for Server2: 
    zm1a IPaddr::192.168.2.50/24 zimbra
  5. Notice that both point to zm1a. That sets zm1a as the primary. Failure to do that will result in them trying to take each over, which just becomes a huge mess.
  6. Create /etc/heartbeat/authkeys on both servers 
    auth 3
3 md5 yourrandommd5string

    Protect the permissions of authkeys file on both servers:

    chmod 600 /etc/heartbeat/authkeys

Disable Upstart for Zimbra Services

On both machines, issue the below command to remove the startup services since Heartbeat will be handling them:

# update-rc.d -f zimbra remove

Final Comments:

Again, Heartbeat thinks Zimbra failed to start since the service takes so long to read from the GlusterFS. If you can figure a way to improve that, the above proof of concept should work well.