Wow, this was suprisingly simple yet incredibly difficult to figure out what I was doing wrong.
Server X has two NICs, one in a DMZ VLAN (192.168.1.0/24 on eth0) and one in a private VLAN (192.168.2.0/24 on eth1). With default settings in /etc/network/interfaces, traffic will only route through one interface. No matter what, you won’t be able to ping 192.168.2.0/24.
In the interfaces config, add a metric for each interface. This is what it /etc/network/interfaces should look like:
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static metric 0 address 192.168.1.29 netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 184.108.40.206 auto eth1 iface eth1 inet static metric 1 address 192.168.2.31 netmask 255.255.255.0 gateway 192.168.2.1 dns-nameservers 192.168.2.1 dns-search default.net
Just run ifdown eth1 && ifup eth1 && ifdown eth0 && ifup eth0 and you should be good to go.
Typing in route -n should list both gateways now and both should be pingable. No need to do any fancy routing, port forwarding or using iproute2.