fsck on lvm partition

fsck on lvm can be a bit tricky. Like a the normal process, the partition needs to be umounted and we can boot up using the rescue cd or in emergency mode. Normally, you

fsck /dev/sda1

if the partition is lvm, you need to activate the lvm first like so

vgchange --ignorelockingfailure -ay
lvscan --ignorelockingfailure (this command should now work)
fsck /dev/volumegroup/lvname

if the logical volume contains a partition, you need to kpartx the lv.

Best Linux Partitioning Setup?

Over the years, there were many attempts to find the best way to partition the hard disk or even the virtual machine. Do you need to have /var, /tmp, /usr…etc as separate partitions? Some people say it depends on what applications your server is running. If it is a mail server, maybe the /var might grow really quickly. So a bigger partition is needed for /var.

With the introduction of LVM, things have changed. We can now shrink and expand partitions easily. In fact, the default partitioning scheme for some linux distros during installation is to use LVM – Just one partition for ‘/boot’ (100MB) and one for ‘/’ (using up rest of the disk space). This means that if you need a lot of space for /var, you need to increase the space for ‘/’ partition.

So if we have a new drive and wish to expand your ‘/’ partition for example, we have to umount the affected partition, do a resize2fs, then mount it back. Unmounting the ‘/’ partition while the machine is running is a big no no. The solution is to shutdown the machine and resize2fs the partition individually. Tough luck.

To overcome that, I suggest we separate the base OS from data – we create a separate partition call /home/data and place all the variable data, ie web, database, user, email…etc files in there. The only problem is that if you have selinux running, you need to fix the permissions.

I now have my ideal partitioning setup:

/dev/hda1 /boot (100MB)
/dev/hda2 swap (1.5 times my ram)
/dev/hda3 / (10000MB)
/dev/hda4 /home/data (just big enough will do)

I think the above partition scheme is simple and effective. Noticed the sequence of the partitions. I purposely put /home/data at the end so that I can expand it easily. Even though we know that /home/data will grow the fastest, I still believe in expanding it only when required (its easy enough). Assigning the rest of the disk space to it will only make it big and complicates the backup process if you decide to have one.

What if we want to expand the ‘/’ partition. This becomes abit complicated (beyond the scope now). That is why I allocate enough disk space for a long long time. I think 10G to 20G should last a long time.

Hope this article helps.

cheers.

Creating Xen Redundant Virtual Machines with Backup Procedures

It is a good idea to backup the whole virtual machine to a separate machine to achieve redundancy. 99% uptime and full redundancy can be achieved using on-the-fly mirroring, ie network raid 1. Hardware and network performance will determine if this method will work or not. There are a few software that can achieve this. Many linux administrators use drbd and heartbeat.

An alternative approach is to do a full nightly backup and incremental hourly backup in the day. This is less write intensive and there is a chance of losing an hour’s worth of work if the actual server goes down. But still, it is a decent solution if there are hardware constraints. I will focus more on this method.

Here is the idea. Imagine we have 2 real machines, machine 1 and 2. lvphp4 is a php4 logical volume running in machine2. It has a backup in machine1. In machine2, write a cron script that runs every hour to ssh into machine1 to mount the lvphp4 data partition (say partition 2). Then sync the data over to machine1. Once done, umount and send an email to the administrator if you want. Do the same for machine2. Assuming that there will not be any base OS changes in the day, we will sync the data only.

# mount the required partition
kpartx -a /dev/vg/lvphp4
mount /dev/mapper/lvphp4p2 /mnt/lvphp4p2
ssh root@machine1 "kpartx -a /dev/vg/lvphp4;mount /dev/mapper/lvphp4p2 /mnt/lvphp4p2;"
rsync -var -e ssh --delete --stats --progress /mnt/lvphp4p2 root@machine1:/mnt/
# now umount and cleanup everything
ssh root@machine1 "umount /mnt/lvphp4p2;kpartx -d /dev/vg/lvphp4;"
umount /mnt/lvphp4p2
kpartx -d /dev/vg/lvphp4

If the virtual machine lvphp4 in machine2 fails for whatever reason, we can bring the backup in machine1 up really quickly by sshing into machine1 and run

xm create /etc/xen/lvphp4

I believe this part can be integrated into a monitoring software (nagios for example) to achieve redundancy.

The reason why lvphp4 fails in machine2 is most likely due to hardware failure in machine2. Do not autostart lvphp4 so that when machine2 boots up, lvphp4 doesn’t start by itself. Once machine2 is repaired, choose one night to transfer the backup over from machine1 to machine2. in machine1,

xm shutdown /etc/xen/lvphp4
dd if=/dev/vg/lvphp4 | ssh root@machine2 "dd of=/dev/vg/lvphp4"

I am using this method in the live environment and it works perfectly. rsync does it’s job really well.

How to resize LVM running Xen part 2 – decrease disk size

shrinking a lvm partition is straight forward if it doesn’t contain a partition table. Simply do a:

umount lvm_partition
resize2fs /dev/vg/lv newSize
lvresize -L disksize /dev/vg/lv
resize2fs /dev/vg/lv

execute the last command if nessary. Assuming you have a partition table in a 10G domU which uses the default partition table,

Disk /dev/xenvg/XenWeb: 10.5 GB, 10502537216 bytes
255 heads, 63 sectors/track, 1276 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

            Device Boot      Start         End      Blocks   Id  System
/dev/xenvg/XenWeb1   *           1          13      104391   83  Linux
/dev/xenvg/XenWeb2              47        1276     9879975   83  Linux
/dev/xenvg/XenWeb3              14          46      265072+  82  Linux swap / Solaris

[root@bpehhome2 mapper]# lvscan
  ACTIVE            '/dev/xenvg/root' [4.50 GB] inherit
  ACTIVE            '/dev/xenvg/XenAuth' [8.00 GB] inherit
  ACTIVE            '/dev/xenvg/XenDebianDefault' [2.00 GB] inherit
  ACTIVE            '/dev/xenvg/swap' [1.00 GB] inherit
  ACTIVE            '/dev/xenvg/XenCentOSInstall' [3.00 GB] inherit
  ACTIVE            '/dev/xenvg/XenWeb1' [5.03 GB] inherit
  ACTIVE            '/dev/xenvg/XenWeb' [9.78 GB] inherit

I want to reduce the size of the XenWeb logical volume to 9G. First of all, let me shut it down.

[root@bpehhome2 mapper]# xm shutdown web
[root@bpehhome2 mapper]# pwd
/dev/mapper
[root@bpehhome2 mapper]# kpartx -a /dev/xenvg/XenWeb
[root@bpehhome2 mapper]# ls
control     xenvg-XenAuth           xenvg-XenWeb    xenvg-XenWebp2  XenWeb2
xenvg-root  xenvg-XenCentOSInstall  xenvg-XenWeb1   xenvg-XenWebp3  XenWeb3
xenvg-swap  xenvg-XenDebianDefault  xenvg-XenWebp1  XenWeb1

OK, let’s reduce the partition XenWeb2 size. I want XenWeb2 to be 9000M, but when I do the resize, I need to bring it abit lower, say 8500M. You will see why later.

[root@bpehhome2 mapper]# resize2fs XenWeb2 8500M
resize2fs 1.39 (29-May-2006)
Please run 'e2fsck -f XenWeb2' first.

[root@bpehhome2 mapper]# e2fsck -f XenWeb2
e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/: 246385/2395520 files (1.0% non-contiguous), 1044750/2469993 blocks

[root@bpehhome2 mapper]# resize2fs XenWeb2 8500M
resize2fs 1.39 (29-May-2006)
Resizing the filesystem on XenWeb2 to 2176000 (4k) blocks.
The filesystem on XenWeb2 is now 2176000 blocks long.

Its time to resize the logical volumn

[root@bpehhome2 mapper]# lvresize -L 9000M /dev/xenvg/XenWeb
  Rounding up size to full physical extent 8.81 GB
  WARNING: Reducing active and open logical volume to 8.81 GB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce XenWeb? [y/n]: y
  Reducing logical volume XenWeb to 8.81 GB
  Logical volume XenWeb successfully resized
[root@bpehhome2 mapper]# fdisk /dev/xenvg/XenWeb

The number of cylinders for this disk is set to 1150.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/xenvg/XenWeb: 9462 MB, 9462349824 bytes
255 heads, 63 sectors/track, 1150 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

            Device Boot      Start         End      Blocks   Id  System
/dev/xenvg/XenWeb1   *           1          13      104391   83  Linux
/dev/xenvg/XenWeb2              47        1276     9879975   83  Linux
/dev/xenvg/XenWeb3              14          46      265072+  82  Linux swap / Solaris

Partition table entries are not in disk order

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (47-1150, default 47):
Using default value 47
Last cylinder or +size or +sizeM or +sizeK (47-1150, default 1150):
Using default value 1150

Command (m for help): p

Disk /dev/xenvg/XenWeb: 9462 MB, 9462349824 bytes
255 heads, 63 sectors/track, 1150 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

            Device Boot      Start         End      Blocks   Id  System
/dev/xenvg/XenWeb1   *           1          13      104391   83  Linux
/dev/xenvg/XenWeb2              47        1150     8867880   83  Linux
/dev/xenvg/XenWeb3              14          46      265072+  82  Linux swap / Solaris

Partition table entries are not in disk order
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Noticed that XenWeb2 is now 8.8G which is close to what we want. But we still need to do another resize2fs to get XenWeb2 working.

[root@bpehhome2 mapper]# kpartx -a /dev/xenvg/XenWeb
[root@bpehhome2 mapper]# resize2fs XenWeb2
resize2fs 1.39 (29-May-2006)
Resizing the filesystem on XenWeb2 to 2216970 (4k) blocks.
The filesystem on XenWeb2 is now 2216970 blocks long.

[root@bpehhome2 mapper]# kpartx -d /dev/xenvg/XenWeb

Everything looks good. Its time now to reboot my domU and test if it is working.

[root@bpehhome2 mapper]# xm create /etc/xen/web.cfg
[root@bpehhome2 mapper]# xm console web

CentOS release 5 (Final)
Kernel 2.6.18-53.1.14.el5xen on an i686

web login: root
Password:
Last login: Sat Feb  7 10:32:43 on xvc0
[root@web ~]# mount
/dev/xvda2 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/xvda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
[root@web ~]# fdisk -l

Disk /dev/xvda: 9462 MB, 9462349824 bytes
255 heads, 63 sectors/track, 1150 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *           1          13      104391   83  Linux
/dev/xvda2              47        1150     8867880   83  Linux
/dev/xvda3              14          46      265072+  82  Linux swap / Solaris

Partition table entries are not in disk order

Everything looks ok. It is a good idea to do a backup of the logical volume before shrinking because shrinking disk space is always more involved than extending it.

I described the theory more in depth in my other article about extending lvm partition.

How to resize LVM running Xen part 1 – increase disk size

Resizing a lvm partition is straight forward if it doesn’t contain a partition table. Simply do a:

lvresize -L disksize /dev/vg/lv
resize2fs /dev/vg/lv

If it is running a virtual machine like xen with a partition table, how to resize the domU, whether to shutdown domU or not depends largely on the partiton structure. In centos, if you do a default install, the default installation uses lvm without doing any proper partitioning, as in unlike traditional partitioning, there is no home, usr, tmp partions..etc. In a virtualised environment and with the default partitioning scheme, if you want to resize the home partition, you have to resize the root (/) partition. It is not easy to do it in domU so the best way is to do it is from domO.

Resizing domU from dom0 is more involved. So do yourself a favour by having a proper partition scheme like:

/boot
/
/usr
/var
/tmp
swap space which is 2 times your ram

Having partitioning scheme like this means that you don’t need to touch / when resizing ‘/home’ or ‘/usr’ which is quite common.

Ok, its time to do the dirty job. This article is to show how to resize domU from dom0 in case you need to do it. Shutdown my domU first.

[root@bpehhome2 mapper]# xm shutdown web

This is a quick overview of my web domU.

[root@bpehhome2 mapper]# fdisk -l /dev/xenvg/XenWeb

Disk /dev/xenvg/XenWeb: 10.5 GB, 10502537216 bytes
255 heads, 63 sectors/track, 1276 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

            Device Boot      Start         End      Blocks   Id  System
/dev/xenvg/XenWeb1   *           1          13      104391   83  Linux
/dev/xenvg/XenWeb2              47        1276     9879975   83  Linux
/dev/xenvg/XenWeb3              14          46      265072+  82  Linux swap / Solaris

If you try to resizefs /dev/xenvg/XenWeb straight away, resize2fs doesn’t like the MBR and will throw out error like so:

resize2fs: Bad magic number in super-block while trying to open /dev/xenvg/XenWeb
Couldn't find valid filesystem superblock.

To overcome that, we need to split the partition up and resizefs them separately. Let us go through the steps in detail.

1) In domO, check that you have enough disk space. In this example, I have about 3G more. I will add 1G to the xenweb logical volume

[root@bpehhome2 mapper]# vgdisplay
--- Volume group ---
VG Name               xenvg
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  90
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                7
Open LV               3
Max PV                0
Cur PV                1
Act PV                1
VG Size               37.16 GB
PE Size               32.00 MB
Total PE              1189
Alloc PE / Size       1066 / 33.31 GB
Free  PE / Size       123 / 3.84 GB
VG UUID               3vcgz5-1O2N-Oj89-o7KK-Tl51-wZbT-z32pgQ

2. I checked my current logical volume and resize xenweb to 1000M.

[root@bpehhome2 mapper]# lvscan
ACTIVE            '/dev/xenvg/root' [4.50 GB] inherit
ACTIVE            '/dev/xenvg/XenAuth' [8.00 GB] inherit
ACTIVE            '/dev/xenvg/XenDebianDefault' [2.00 GB] inherit
ACTIVE            '/dev/xenvg/swap' [1.00 GB] inherit
ACTIVE            '/dev/xenvg/XenCentOSInstall' [3.00 GB] inherit
ACTIVE            '/dev/xenvg/XenWeb1' [5.03 GB] inherit
ACTIVE            '/dev/xenvg/XenWeb' [9.78 GB] inherit

[root@bpehhome2 ~]# lvresize -L 10000M /dev/xenvg/XenWeb
  Rounding up size to full physical extent 9.78 GB
  Extending logical volume XenWeb to 9.78 GB
  Logical volume XenWeb successfully resized

3. Now I resize the partition in XenWeb2 in /dev/xenvg/XenWeb

[root@bpehhome2 mapper]# fdisk /dev/xenvg/XenWeb

The number of cylinders for this disk is set to 1276.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk xenvg-XenWeb: 10.5 GB, 10502537216 bytes
255 heads, 63 sectors/track, 1276 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

       Device Boot      Start         End      Blocks   Id  System
xenvg-XenWeb1   *           1          13      104391   83  Linux
xenvg-XenWeb2              47        1150     8867880   83  Linux
xenvg-XenWeb3              14          46      265072+  82  Linux swap / Solaris

Partition table entries are not in disk order

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (47-1276, default 47):
Using default value 47
Last cylinder or +size or +sizeM or +sizeK (47-1276, default 1276):
Using default value 1276

Command (m for help): p

Disk xenvg-XenWeb: 10.5 GB, 10502537216 bytes
255 heads, 63 sectors/track, 1276 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

       Device Boot      Start         End      Blocks   Id  System
xenvg-XenWeb1   *           1          13      104391   83  Linux
xenvg-XenWeb2              47        1276     9879975   83  Linux
xenvg-XenWeb3              14          46      265072+  82  Linux swap / Solaris

Partition table entries are not in disk order

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

4. Now, its time to split the partition up. The partitions can be viewed in /dev/mapper. I want to resize2fs partition 2 only.

[root@bpehhome2 mapper]# kpartx -a /dev/xenvg/XenWeb
[root@bpehhome2 ~]# cd /dev/mapper
[root@bpehhome2 mapper]# ls
control     xenvg-XenAuth           xenvg-XenWeb    xenvg-XenWebp2  XenWeb2
xenvg-root  xenvg-XenCentOSInstall  xenvg-XenWeb1   xenvg-XenWebp3  XenWeb3
xenvg-swap  xenvg-XenDebianDefault  xenvg-XenWebp1  XenWeb1

[root@bpehhome2 mapper]# resize2fs XenWeb2
resize2fs 1.39 (29-May-2006)
Please run 'e2fsck -f XenWeb2' first.

[root@bpehhome2 mapper]# e2fsck -f XenWeb2
e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/: ***** FILE SYSTEM WAS MODIFIED *****
/: 246443/2143360 files (1.0% non-contiguous), 1037188/2216970 blocks

5. Do some cleaning up.

[root@bpehhome2 mapper]# kpartx -d /dev/xenvg/XenWeb

6. boot up domU and check if everything is working

[root@bpehhome2 mapper]# xm create /etc/xen/web.cfg

I will blog about shrinking lvm with partition next which is slightly more complex.

Xen Guest, Dom U – dont use LVM

I dont see any reason why one should use lvm in xen guest. The whole idea of using lvm is to have the ability to resize disk size easily and resizing a xen guest can be done easily without using LVM. Actually, having LVM in xen guest complicates the process of resizing… You can google around to see the extra steps needed to resize a lvm xen guest…

LVM might be the default installation if you are installing from certain linux distros. If that is the case, modify the partition table to use normal linux partition instead of LVM. LVM is cool for Dom0 but not for DomU (guest).

That is just my experience.