We’re going to explore high availability and load balancing using Keepalived and HAProxy.

Keepalived is a routing software designed to provide simple and robust facilities for load balancing and high-availability to Linux systems and Linux-based infrastructures.

HAProxy is an open source load balancer/reverse proxy generally used for load balancing web services, but also has the functionality to load balance TCP traffic.

Together, Keepalived and HAProxy provide some unique benefits.

Specifically, they provide a low cost solution for high availability compared to proprietary hardware-based load balancers.

Keepalived and Haproxy in AWS

Keepalived uses the VRRP protocol to detect when HAProxy is down and fails with minimal downtime.

The following documentation is focused on setting up Keepalived in AWS (or potentially any cloud) where multicast is not supported, and you’re using a debian-based OS.

Keepalived and HAProxy Setup

First, download the latest version of Keepalived.

(http://www.keepalived.org/software/keepalived-1.2.13.tar.gz)

You will need this version since the previous version in your repository is probably too old to support VRRP Unicast.

wget http://www.keepalived.org/software/keepalived-1.2.13.tar.gz tar xf keepalived-1.2.13.tar.gz 
cd keepalived-1.2.13/ 
./configure 
make && make install cp keepalived/etc/init.d/keepalived.init /etc/init.d/keepalived

mkdir /etc/keepalived/

Keepalived and HAProxy Architecture

We’re going to assume two load balancer servers are running Keepalived and HAProxy, where one is the master, the other is a failover, and we have multiple web servers.

Keepalived and HAProxy AWS Architecture

In this scenario, we have an elastic IP on ServerA while ServerB is set as a passive server not being hit by the public internet. We also have five web servers running our app.

By default ServerA is set as MASTER state and ServerB is set as BACKUP, which means on startup the servers know their state, and also means on a failover/fail back scenario both servers know their role.

In a failure scenario where ServerA becomes unresponsive (i.e HAProxy process hangs, server dies etc.) ServerB will be unable to contact serverA. Therefore, ServerA enters a FAULT mode and ServerB becomes master. When ServerB becomes master it runs a script which will basically disassociate the Elastic IP from ServerA and assign it to ServerB. ServerB is now the only active HAProxy server and handles requests to the app servers.

Configs

A few online references seem to have incorrect configs for setting up Keepalived in a Unicast environment, however the following instructions seem to work correctly.

We need to set up our master and slave configs for Keepalived slightly different as they both have different roles to play in failover. The vrrp_script is used to verify the other party is functioning correctly. We also have in our notify_master section, /etc/keepalived/master.sh, a script which assigns the Elastic IP to the master in a case of failure or recovery.

Server A (master) (/etc/keepalived/keepalived.conf)

vrrp_script chk_haproxy { 
    script "pidof haproxy" 
    interval 2 
} 
vrrp_instance VI_1 { 
    debug 2 
    interface eth0 # interface to monitor 
    state MASTER 
    virtual_router_id 51 # Assign one ID for this route 
    priority 101 # 101 on master, 100 on backup 
    unicast_src_ip 1.2.3.4 # My IP 
    unicast_peer {
        5.6.7.8 # peer IP 
    } 
    track_script { 
        chk_haproxy 
    } 
    notify_master /etc/keepalived/master.sh 
}

Server B (backup) (/etc/keepalived/keepalived.conf)

vrrp_script chk_haproxy { 
script "pidof haproxy" 
    interval 2 
} 
vrrp_instance VI_1 { 
    debug 2 
    interface eth0 # interface to monitor 
    state BACKUP 
    virtual_router_id 51 # Assign one ID for this route 
    priority 100 # 101 on master, 100 on backup 
    unicast_src_ip 5.6.7.8 # My IP 
    unicast_peer {
        1.2.3.4 # Peer IP 
    } 
    track_script { 
        chk_haproxy 
    } 
    notify_master /etc/keepalived/master.sh 
}

script notify_master (/etc/keepalived/master.sh)

#!/bin/bash 
EIP=9.8.7.6 
INSTANCE_ID=i-abcd1234 

/usr/local/bin/aws ec2 disassociate-address --public-ip $EIP 
/usr/local/bin/aws ec2 associate-address --public-ip $EIP --instance-id $INSTANCE_ID

Starting Keepalived

Keepalived startup messages are in /var/log/syslog, to see if your process started up correctly you should see an output similar to below:

ServerA – Master

Keepalived[23638]: Starting Keepalived v1.2.13 (11/28,2014) 
Keepalived[23639]: Starting Healthcheck child process, pid=23641 
Keepalived[23639]: Starting VRRP child process, pid=23642 
Keepalived_healthcheckers[23641]: Initializing ipvs 2.6 
Keepalived_healthcheckers[23641]: Registering Kernel netlink reflector 
Keepalived_healthcheckers[23641]: Registering Kernel netlink command channel 
Keepalived_vrrp[23642]: Registering Kernel netlink reflector 
Keepalived_vrrp[23642]: Registering Kernel netlink command channel 
Keepalived_vrrp[23642]: Registering gratuitous ARP shared channel 
Keepalived_vrrp[23642]: Opening file '/etc/keepalived/keepalived.conf'. 
Keepalived_vrrp[23642]: Configuration is using : 60724 Bytes
Keepalived_vrrp[23642]: Using LinkWatch kernel netlink reflector... 
Keepalived_healthcheckers[23641]: Opening file '/etc/keepalived/keepalived.conf'. 
Keepalived_healthcheckers[23641]: Configuration is using : 5129 Bytes 
Keepalived_healthcheckers[23641]: Using LinkWatch kernel netlink reflector... 
Keepalived_vrrp[23642]: VRRP_Instance(VI_1) Transition to MASTER STATE 
Keepalived_vrrp[23642]: VRRP_Instance(VI_1) Entering MASTER STATE 
Keepalived_vrrp[23642]: Opening script file /etc/keepalived/master.sh

ServerB – Backup

Keepalived[28033]: Starting Keepalived v1.2.13 (11/28,2014) 
Keepalived[28034]: Starting Healthcheck child process, pid=28036 
Keepalived[28034]: Starting VRRP child process, pid=28037 
Keepalived_healthcheckers[28036]: Initializing ipvs 2.6 
Keepalived_vrrp[28037]: Registering Kernel netlink reflector 
Keepalived_vrrp[28037]: Registering Kernel netlink command channel 
Keepalived_vrrp[28037]: Registering gratuitous ARP shared channel 
Keepalived_healthcheckers[28036]: Registering Kernel netlink reflector 
Keepalived_healthcheckers[28036]: Registering Kernel netlink command channel 
Keepalived_vrrp[28037]: Opening file '/etc/keepalived/keepalived.conf'. 
Keepalived_vrrp[28037]: Configuration is using : 61884 Bytes 
Keepalived_vrrp[28037]: Using LinkWatch kernel netlink reflector... 
Keepalived_vrrp[28037]: VRRP_Instance(VI_1) Entering BACKUP STATE 
Keepalived_healthcheckers[28036]: Opening file '/etc/keepalived/keepalived.conf'. 
Keepalived_healthcheckers[28036]: Configuration is using : 5127 Bytes

Verify Connectivity

There are many ways to test connectivity, tshark, tcpdump etc, here are some sample commands:

tcpdump "ip proto 112" 
tshark -f “vrrp”

The reason we need to verify connectivity with either of the commands above is because Keepalived uses multicast, which is not supported in AWS. The commands verify our config is correctly sending via Unicast.
Verify Connectivity Keepalived Haproxy AWS tshark

Testing Failover

Assuming you have your failover.sh script setup correctly, on stopping HAProxy on ServerA, you should see it entering a FAULT state and see ServerB entering a MASTER state. When ServerB enters a master state it will run a script and reassign the Elastic IP to ServerB.


Ready to start getting insights from your applications? Sign up for a Logentries free trial today.