Highly available load Balancer + web server in centos

This tutorial is about creating a highly available HTTP load balancer using HAProxy. The setup can be slightly complicated but you will appreciate the result – you will get a load-balanced + highly available web service in your network. Basic linux skill is assumed.

Scenario:

Imagine we have 2 physical machines. In each machine, I have 2 virtual machines. All 4 virtual machines will be in the same subnet, ie 10.1.1.0/24 in this case.

Steps:

1. create 4 vm, centos1.dev (10.1.1.111), centos2.dev (10.1.1.112), centos3.dev (10.1.1.113) and centos4.dev (10.1.1.114). These 4 vm should have the bare min. packages installed.

unless using DNS, add this to /etc/hosts on all virtual machines

10.1.1.111      centos1.dev
10.1.1.112      centos2.dev
10.1.1.113      centos3.dev
10.1.1.114      centos4.dev

2. leave firewall and selinux on. allow port 80 for all 4 vm.

3. centos1.dev and centos2.dev will be the load balancer and centos3.dev and centos4.dev will be the 2 http servers. In centos3 and centos4,

yum groupinstall "web server"
chkconfig httpd on

4. Then in centos3.dev and centos4.dev again, edit /etc/httpd/conf/httpd.conf, in order to capture the real IP of the user, replace %h to %{X-Forwarded-For}i. We also add a virtual host.

#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
...
...
NameVirtualHost *:80
<VirtualHost *:80>
DocumentRoot /var/www/html
ServerName *
SetEnvIf Request_URI "^/haproxy\.txt$" dontlog
CustomLog /var/log/httpd/access.log combined env=!dontlog
</VirtualHost>

This virtual config is just for the sake of testing, you need to change it in the real environment.

5. In both centos3 and centos 4 again,

cd /var/www/html,
echo "centos3" > index.html

(in centos4, echo “centos4” > index.html)

then create haproxy.txt in the same dir for both http servers. Without the check file, haproxy will fail.

touch haproxy.txt

Restart apache (httpd) in both servers

Installing HAProxy:

1. ssh into centos1.dev and centos2.dev and install Haproxy. Someone has compiled the rpm for us. Download it from the rpmbone website and install it.

http://rpm.pbone.net/index.php3/stat/4/idpl/13437166/com/haproxy-1.3.22-1.el5.x86_64.rpm.html

After installing it,

chkconfig haproxy on

2. edit /etc/haproxy/haproxy.cfg

global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
maxconn 4096
user haproxy
group haproxy

defaults
log     global
mode    http
option  httplog
option  dontlognull
option redispatch
retries 3
maxconn 2000
contimeout      5000
clitimeout      50000
srvtimeout      50000

listen ha-http 10.1.1.110:80
mode http
stats enable
stats auth user:password
balance roundrobin
cookie JSESSIONID prefix
option httpclose
option forwardfor
option httpchk HEAD /haproxy.txt HTTP/1.0
server apache1 centos3.dev:80 cookie A check
server apache2 centos4.dev:80 cookie B check

3. To allow HAProxy to bind to the shared IP address, we add the following line to /etc/sysctl.conf:

net.ipv4.ip_nonlocal_bind=1

then reload sysctl config,

sysctl -p

Installing Heartbeat:

1. Heartbeat is necessary for any highly available systems. In both centos1 and centos2, to install heartbeat

yum install heartbeat

2. after that in centos1.dev, cd /etc/ha.d, edit /etc/ha.d/authkeys

auth 2
2 sha1 loadbalancing-ha

3. edit /etc/ha.d/ha.cf

keepalive 2
deadtime 10
udpport 694
bcast eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 centos2.dev
udp     eth0
logfacility local0
node    centos1.dev
node    centos2.dev

node needs to be the machine name, ie type “hostname” in command line to see. Now we want centos1 to be highly available, so edit /etc/ha.d/haresources:

centos1.dev 10.1.1.110

4. if firewall is turned on, remember to allow 694:udp (do it for both centos1.dev and centos2.dev)

5. after setting everything in centos1, copy the files over to centos2, ie

scp {authkeys,haresource,ha.cf} 10.1.1.112:/etc/ha.d

6. now in centos2, edit ha.cf

keepalive 2
deadtime 10
udpport 694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 centos1.dev
udp eth0
logfacility local0
node    centos1.dev
node    centos2.dev

Noticed the difference in ucast

7. Now we want to start heartbeat in both machines upon reboot

echo "service heartbeat start" >> /etc/rc.local

Testing

The ip 10.1.1.110:80 is now load balanced and highly available. To test it, shutdown 10.1.1.111 and the load balancer will still function. If 10.1.1.113 HTTP is down, 10.1.1.114 will take over and vice version.

Viewing Haproxy Stats

1. The options “stats enable” and “stats auth” in the HAProxy configuration allow the admin to view the stats, just go to http://10.1.1.110/haproxy?stats and type in username as user and password as password

Conclusion

I hope you follow me so far and appreciate what HAproxy can offer. I certainly enjoy blogging about it and I hope you it useful.