init: Id “co” respawning too fast: disabled for 5 minutes

If you are getting this message, it means that you are in run level 3 and most likely you dont have the GUI installed. inittab is trying to create too many terminals at a go. Have a look at the /etc/inittab file, and commented off the extra gettys lines.

# Run gettys in standard runlevels
co:2345:respawn:/sbin/agetty xvc0 9600 vt100-nav
1:2345:respawn:/sbin/mingetty tty1
# 2:2345:respawn:/sbin/mingetty tty2
# 3:2345:respawn:/sbin/mingetty tty3
# 4:2345:respawn:/sbin/mingetty tty4
# 5:2345:respawn:/sbin/mingetty tty5
# 6:2345:respawn:/sbin/mingetty tty6

reload inittab and everything should be fine.

telinit q

system-config-securitylevel – Love or hatred?

system-config-securitylevel provides the user an easy interface to edit firewall rules. If you only need to open up certain tcp and udp ports, this tool may prove handy. For more complex routing rules, you are tempted to edit /etc/sysconfig/iptables directly.

If you have used system-config-securitylevel before, check out the iptables file and you will see this in the header:

# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.

You were warned!! The problem with this is that the next time you run system-config-securitylevel (even if accidental), all your custom rules will be overwritten… sucks. So remember to backup /etc/sysconfig/iptables or use some sort of version control everytime you decide to edit the file directly.

Highly available load Balancer + web server in centos

This tutorial is about creating a highly available HTTP load balancer using HAProxy. The setup can be slightly complicated but you will appreciate the result – you will get a load-balanced + highly available web service in your network. Basic linux skill is assumed.

Scenario:

Imagine we have 2 physical machines. In each machine, I have 2 virtual machines. All 4 virtual machines will be in the same subnet, ie 10.1.1.0/24 in this case.

Steps:

1. create 4 vm, centos1.dev (10.1.1.111), centos2.dev (10.1.1.112), centos3.dev (10.1.1.113) and centos4.dev (10.1.1.114). These 4 vm should have the bare min. packages installed.

unless using DNS, add this to /etc/hosts on all virtual machines

10.1.1.111      centos1.dev
10.1.1.112      centos2.dev
10.1.1.113      centos3.dev
10.1.1.114      centos4.dev

2. leave firewall and selinux on. allow port 80 for all 4 vm.

3. centos1.dev and centos2.dev will be the load balancer and centos3.dev and centos4.dev will be the 2 http servers. In centos3 and centos4,

yum groupinstall "web server"
chkconfig httpd on

4. Then in centos3.dev and centos4.dev again, edit /etc/httpd/conf/httpd.conf, in order to capture the real IP of the user, replace %h to %{X-Forwarded-For}i. We also add a virtual host.

#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
...
...
NameVirtualHost *:80
<VirtualHost *:80>
DocumentRoot /var/www/html
ServerName *
SetEnvIf Request_URI "^/haproxy\.txt$" dontlog
CustomLog /var/log/httpd/access.log combined env=!dontlog
</VirtualHost>

This virtual config is just for the sake of testing, you need to change it in the real environment.

5. In both centos3 and centos 4 again,

cd /var/www/html,
echo "centos3" > index.html

(in centos4, echo “centos4” > index.html)

then create haproxy.txt in the same dir for both http servers. Without the check file, haproxy will fail.

touch haproxy.txt

Restart apache (httpd) in both servers

Installing HAProxy:

1. ssh into centos1.dev and centos2.dev and install Haproxy. Someone has compiled the rpm for us. Download it from the rpmbone website and install it.

http://rpm.pbone.net/index.php3/stat/4/idpl/13437166/com/haproxy-1.3.22-1.el5.x86_64.rpm.html

After installing it,

chkconfig haproxy on

2. edit /etc/haproxy/haproxy.cfg

global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
maxconn 4096
user haproxy
group haproxy

defaults
log     global
mode    http
option  httplog
option  dontlognull
option redispatch
retries 3
maxconn 2000
contimeout      5000
clitimeout      50000
srvtimeout      50000

listen ha-http 10.1.1.110:80
mode http
stats enable
stats auth user:password
balance roundrobin
cookie JSESSIONID prefix
option httpclose
option forwardfor
option httpchk HEAD /haproxy.txt HTTP/1.0
server apache1 centos3.dev:80 cookie A check
server apache2 centos4.dev:80 cookie B check

3. To allow HAProxy to bind to the shared IP address, we add the following line to /etc/sysctl.conf:

net.ipv4.ip_nonlocal_bind=1

then reload sysctl config,

sysctl -p

Installing Heartbeat:

1. Heartbeat is necessary for any highly available systems. In both centos1 and centos2, to install heartbeat

yum install heartbeat

2. after that in centos1.dev, cd /etc/ha.d, edit /etc/ha.d/authkeys

auth 2
2 sha1 loadbalancing-ha

3. edit /etc/ha.d/ha.cf

keepalive 2
deadtime 10
udpport 694
bcast eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 centos2.dev
udp     eth0
logfacility local0
node    centos1.dev
node    centos2.dev

node needs to be the machine name, ie type “hostname” in command line to see. Now we want centos1 to be highly available, so edit /etc/ha.d/haresources:

centos1.dev 10.1.1.110

4. if firewall is turned on, remember to allow 694:udp (do it for both centos1.dev and centos2.dev)

5. after setting everything in centos1, copy the files over to centos2, ie

scp {authkeys,haresource,ha.cf} 10.1.1.112:/etc/ha.d

6. now in centos2, edit ha.cf

keepalive 2
deadtime 10
udpport 694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 centos1.dev
udp eth0
logfacility local0
node    centos1.dev
node    centos2.dev

Noticed the difference in ucast

7. Now we want to start heartbeat in both machines upon reboot

echo "service heartbeat start" >> /etc/rc.local

Testing

The ip 10.1.1.110:80 is now load balanced and highly available. To test it, shutdown 10.1.1.111 and the load balancer will still function. If 10.1.1.113 HTTP is down, 10.1.1.114 will take over and vice version.

Viewing Haproxy Stats

1. The options “stats enable” and “stats auth” in the HAProxy configuration allow the admin to view the stats, just go to http://10.1.1.110/haproxy?stats and type in username as user and password as password

Conclusion

I hope you follow me so far and appreciate what HAproxy can offer. I certainly enjoy blogging about it and I hope you it useful.

syncing date and time in command line

having the wrong date and time in the system can be disastrous as many applications will be using the system date and time. It is therefore very important to have the right date and time.

In linux,

yum install ntp
chkconfig ntp on
ntpdate pool.ntp.org
service ntpd start

You can add the ntpdate command to cron to automate the syncing process. Another easy way is to do it in the GUI, if you have it.

Installing SVN on a samba shared drive in mac

1. Configure the samba share in linux. It needs to have the “delete readonly = yes” line, if not, you will permission errors in .svn/entries when checking out projects in mac. so we need something like this:

[homes]
        comment = LAM IT Home Directories
        browseable = no
        writable = yes
        # creation mode depends on you really
        force create mode = 0660
        force directory mode = 0770
        # add new option to fix samba share on mac
        delete readonly = yes

restart samba server

2. download mac svn client and install it from http://scplugin.tigris.org (you might want to read the installation instruction). The is the best free mac svn client I think.

3. In mac, go->connect and type in the samba server address, ie something like smb://xxx

4. In finder, should be right click on any folder (finder->more->subversion->checkout)

That’s it.

mounting windows share on local machine via samba – fixing selinux

if you are running selinux, you get this error

SELinux is preventing samba (smbd) "getattr" to /mnt/blarblar (cifs_t). For complete SELinux messages. run sealert -l e523015e-150a-4736-80c1-c7a40af6d396"

In fstab, we need to mount samba with the right context like so:

//10.2.115.11/linuxbackup /mnt/blarblar  cifs domain=MYDOMAIN,user=backupservice,password=blarblar,context=system_u:object_r:samba_share_t:s0 0 0

or using command line, remember to use the right context

mount --context="system_u:object_r:samba_share_t:s0"

I found the “samba_selinux” man page useful.

Installing KVM in Centos, Redhat or Fedora

KVM is available in RHEL 5.4. I finally managed to find the time to play around it over the weekend. I don’t think it is as matured as xen but having said that, I do agree with the approach – Kernel virtualisation. Redhat is promoting kvm very hard and you can see it from their website and mailing list. If you are using redhat systems, better upgrade to kvm as xen might be phased out soon.

Unlike xen, I found installing kvm abit of pain…. I am willing to share what I know and this is what this blog is for.

Check your cpu before you start because kvm only supports full virtualisation, ie your /proc/cpuinfo must have either the vmx (intel) or the svm (amd) tag.

Next, I suggest you do a minimal clean centos/rhel/fedora installation. After rebooting, log in as root and add packages as needed. We will start with the kvm package.

yum groupinstall kvm

I didnt install the virtualisation package because it will bring in the xen hypervisor and stuffs – don’t want it for now.

The pain with kvm is that it doesnt come with a network bridge (xen wins!). we will need to add one if you want your virtual machine be in the same network as your physical machine. Assuming my network card is eth1:

cd /etc/sysconfig/network-scripts/
vim ifcfg-eth1

my eth1 config looks like this

# D-Link System Inc RTL8139 Ethernet
DEVICE=eth1
ONBOOT=yes
HWADDR=00:13:46:3a:14:55
BRIDGE=kvmbr0

now we need to create the kvm bridge

vim ifcfg-kvmbr0

my ifcfg-kvmbr0 looks like this:

DEVICE=kvmbr0
TYPE=Bridge
BOOTPROTO=static
GATEWAY=10.1.1.254
IPADDR=10.1.1.201
NETMASK=255.255.255.0
ONBOOT=yes

Its now time to restart the network

service network restart

my routing information looks like this

[root@home network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 kvmbr0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 kvmbr0
0.0.0.0         10.1.1.254      0.0.0.0         UG    0      0        0 kvmbr0

If you have firewall turned on, turn them off for the bridge. In /etc/sysctl.conf, add

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

reload sysctl and libvirtd and we are ready to go

sysctl -p /etc/sysctl.conf
chkconfig libvirtd on
service libvirtd restart

start the virt-manager gui

virt-manager &

I am lazy to post screen shots. Redhat has done a good job already. Just follow the steps in the
redhat kvm guide. Start from Step 3.

If you have selinux running, remember to read the security guide

If you are using lvm as disk, you need to fix the fcontext. My volume group is named as “vm”. So my selinux command is:

semanage fcontext -a -t virt_image_t "/dev/mapper/vm(.*)?"

That’s it for now. Merry Christmas!!

making dd to report the copy status

the linux “dd” command is indispensible to all sys admin. Many people already know that by killing the dd process with the usr1 signal, we can force it to display the file transfer status which is useful.

You can either run a script or check the dd process id from “ps aux”, then

kill -usr1 pid

The shell that runs the dd command will churn out the transfer status. cool!!

the signal man page is also useful:

man 7 signal