Linux LVM review ================ --------------------------------------------------- Notes that will eventually become the whitepaper --------------------------------------------------- Notes ======= 1. Linux lvm much bigger than originally anticipated. 2. ``-v`` option is verbose; multiple ``-v``'s increase the verbosity to the point that, at least on the lvdisplay, it's challenging the xterm scroll length. 3. ``lvmdiskscan`` used to scan for eligible disks :: # lvmdiskscan -l WARNING: only considering LVM devices /dev/sda2 [ 19.51 GiB] LVM physical volume /dev/sdb1 [ 10.00 GiB] LVM physical volume /dev/sdc1 [ 10.00 GiB] LVM physical volume 0 LVM physical volume whole disks 3 LVM physical volumes if using whole disks, which seems smarter, on the whole: :: # lvmdiskscan -l WARNING: only considering LVM devices /dev/sda2 [ 19.51 GiB] LVM physical volume /dev/sdb [ 10.00 GiB] LVM physical volume /dev/sdc [ 10.00 GiB] LVM physical volume 2 LVM physical volume whole disks 1 LVM physical volume 4. Have to say, linux is trying to steal my thunder. Commands available that I wrote scripts to do on HP... :: # pvscan PV /dev/sdb VG vgmirror lvm2 [10.00 GiB / 10.00 GiB free] PV /dev/sdc VG vgmirror lvm2 [10.00 GiB / 10.00 GiB free] PV /dev/sda2 VG vg00 lvm2 [19.51 GiB / 0 free] Total: 3 [39.50 GiB] / in use: 3 [39.50 GiB] / in no VG: 0 [0 ] 5. vgexport/import is a different too. much cleaner, easier than HPUX. no map files, it imports the lv names whereas hpux would simply call them lvol1 lvol2, etc, if no mapfile was used. 6. vgs, lvs, and pvs commands. Interesting output control capabilities 7. ``vgremove ${vg}`` to remove a volume group, unlike the HP vgexport 8. fucking awesome! Splitting and rejoining vgs! :: # pvs PV VG Fmt Attr PSize PFree /dev/sda2 vg00 lvm2 a-- 19.51g 0 /dev/sdb vgmirror lvm2 a-- 10.00g 6.00g /dev/sdc vgmirror lvm2 a-- 10.00g 10.00g # vgsplit vgmirror vgsplit /dev/sdb Logical volume "mirror" must be inactive # mount | grep mirror # lvchange -a n vgmirror/mirror # vgsplit vgmirror vgsplit /dev/sdb New volume group "vgsplit" successfully split from "vgmirror" # pvs PV VG Fmt Attr PSize PFree /dev/sda2 vg00 lvm2 a-- 19.51g 0 /dev/sdb vgsplit lvm2 a-- 10.00g 6.00g /dev/sdc vgmirror lvm2 a-- 10.00g 10.00g # vgchange -a n vgsplit 0 logical volume(s) in volume group "vgsplit" now active # vgchange -a n vgmirror 0 logical volume(s) in volume group "vgmirror" now active # vgmerge vgmirror vgsplit Volume group "vgsplit" successfully merged into "vgmirror" # vgs VG #PV #LV #SN Attr VSize VFree vg00 1 2 0 wz--n- 19.51g 0 vgmirror 2 1 0 wz--n- 19.99g 15.99g 9. Creating mirrors: some good points. a. Log files: 1. mirroring creates a log file on disk unless ``--mirrorlog core`` option is used. 2. Log files, by default, go on disks other than the ones being mirrored - makes sense. Might have to put them on the same disks, though, (for instance, mirroring the OS). Can do that by specifying: ``--alloc anywhere`` b. If creating mirrors > 1.5TB, update the region size:: lvcreate -L 2T -R 2 -n ... use factors of 2 every time you up the space: lvcreate -L 3T -R 4 -n ... c. Another useful feature. You can watch the mirroring taking place: :: # lvdisplay vgmirror/mirror --- Logical volume --- LV Path /dev/vgmirror/mirror LV Name mirror VG Name vgmirror LV UUID xmcKu2-sYgV-CPbm-W6s6-FPcu-9Mwm-i7j8YF LV Write Access read/write LV Creation host, time guest, 2013-03-22 08:40:11 -0500 LV Status NOT available LV Size 4.00 GiB Current LE 1024 Segments 1 Allocation inherit Read ahead sectors auto # lvconvert -m 1 vgmirror/mirror Conversion starts after activation. # vgchange -a y vgmirror 1 logical volume(s) in volume group "vgmirror" now active # lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert root vg00 -wi-ao-- 17.54g swap vg00 -wi-ao-- 1.97g mirror vgmirror mwi-a-m- 4.00g mirror_mlog 9.67 # lvs vgmirror/mirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-a-m- 4.00g mirror_mlog 13.57 # lvs vgmirror/mirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-a-m- 4.00g mirror_mlog 15.04 # lvs vgmirror/mirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-a-m- 4.00g mirror_mlog 62.30 10. Initial steps: a. Assign disks, scan, get them visible to the OS. b. pvcreate, vgcreate, lvcreate: :: # pvscan PV /dev/sda2 VG vg00 lvm2 [19.51 GiB / 0 free] Total: 1 [19.51 GiB] / in use: 1 [19.51 GiB] / in no VG: 0 [0 ] # pvcreate /dev/sdb /dev/sdc Writing physical volume data to disk "/dev/sdb" Physical volume "/dev/sdb" successfully created Writing physical volume data to disk "/dev/sdc" Physical volume "/dev/sdc" successfully created # vgcreate -s 4m vgmirror /dev/sd[bc] Volume group "vgmirror" successfully created # pvs PV VG Fmt Attr PSize PFree /dev/sda2 vg00 lvm2 a-- 19.51g 0 /dev/sdb vgmirror lvm2 a-- 10.00g 10.00g /dev/sdc vgmirror lvm2 a-- 10.00g 10.00g # lvcreate -L 4g -m 1 -n mirror vgmirror Logical volume "mirror" created # lvs vgmirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-a-m- 4.00g mirror_mlog 13.77 # lvs vgmirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-a-m- 4.00g mirror_mlog 81.05 # lvs vgmirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-a-m- 4.00g mirror_mlog 100.00 # mkfs.ext4 /dev/vgmirror/mirror mke2fs 1.41.12 (17-May-2010) Filesystem label= [[snip]] # mkdir -p -m 755 /mnt/mirror # mount /dev/vgmirror/mirror /mnt/mirror # mount /dev/vgmirror/mirror /mnt/mirror # echo "this is a test. this is only a test." > /mnt/mirror/README # lvconvert --splitmirrors 1 --name copy vgmirror/mirror Logical volume mirror converted. # lvs -o +devices vgmirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Devices copy vgmirror -wi-a--- 4.00g /dev/sdc(0) mirror vgmirror -wi-ao-- 4.00g /dev/sdb(0) # lvchange -a n vgmirror/copy # vgsplit vgmirror vgcopy /dev/sdc New volume group "vgcopy" successfully split from "vgmirror" # lvs -o +devices LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Devices root vg00 -wi-ao-- 17.54g /dev/sda2(0) swap vg00 -wi-ao-- 1.97g /dev/sda2(4490) copy vgcopy -wi----- 4.00g /dev/sdc(0) mirror vgmirror -wi-ao-- 4.00g /dev/sdb(0) # mkdir /mnt/mirror1 # mount vgcopy/copy /mnt/mirror1 mount: you must specify the filesystem type # mount /dev/vgcopy/copy /mnt/mirror1 # md5sum /mnt/mirror/README /mnt/mirror1/README a8f852b96dffa74d7135ff35c10677f6 /mnt/mirror/README a8f852b96dffa74d7135ff35c10677f6 /mnt/mirror1/README 11. OK, now let's put it back: :: # umount /mnt/mirror1 # vgchange -a n vgcopy 0 logical volume(s) in volume group "vgcopy" now active # lvconvert -m 1 vgmirror/mirror vgmirror/mirror: Converted: 0.0% ^C # lvs vgmirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-aom- 4.00g mirror_mlog 76.17 # lvs vgmirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-aom- 4.00g mirror_mlog 100.00 12. So, now, we blow away a device to simulate a failure. :: # virsh detach-disk guest hdd --persistent # run on vmhost, need to figure out how to ID correct dev w/o reading xml file directly # vgreduce vgmirror --removemissing --force Couldn't find device with uuid mjpoEn-pZGd-ocew-oR6N-2E71-0eVX-3fKg6Y. Wrote out consistent volume group vgmirror # lvs LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert root vg00 -wi-ao-- 17.54g swap vg00 -wi-ao-- 1.97g mirror vgmirror -wi----- 4.00g # lvdisplay /dev/vgmirror/mirror --- Logical volume --- LV Path /dev/vgmirror/mirror LV Name mirror VG Name vgmirror LV UUID 0OpUgK-HSPc-eIA7-5DcG-ALnE-dY95-NPOrGu LV Write Access read/write LV Creation host, time guest, 2013-03-22 11:51:41 -0500 LV Status NOT available LV Size 4.00 GiB Current LE 1024 Segments 1 Allocation inherit Read ahead sectors auto # lvchange -a y vgmirror/mirror # mount /dev/vgmirror/mirror /mnt/mirror # md5sum /mnt/mirror/README a8f852b96dffa74d7135ff35c10677f6 /mnt/mirror/README 13. Adding a new disk: :: # virsh vol-create-as default guest-3.img 10g # virsh attach-disk guest /var/lib/libvirt/images/guest\-3.img hdd --persistent # virsh start guest # pvcreate /dev/sdc Writing physical volume data to disk "/dev/sdc" Physical volume "/dev/sdc" successfully created # vgextend vgmirror /dev/sdc Volume group "vgmirror" successfully extended # lvconvert -m 1 vgmirror/mirror # lvs vgmirror LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert mirror vgmirror mwi-a-m- 4.00g mirror_mlog 100.00 Software RAID review ====================== 1. Reset the environment using the script 2. Partition the disks, set partitiion type to fd (linux raid) :: fdisk -uc /dev/sdc << eof n p 1 t fd w eof 3. Create the raid partition: :: # mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90 Continue creating array? y mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. 4. Status obtained in two ways: a. ``/proc/mdstat`` :: # cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdc1[1] sdb1[0] 10483640 blocks super 1.2 [2/2] [UU] [>....................] resync = 3.8% (407424/10483640) finish=13.6min speed=12320K/sec unused devices: # grep resync /proc/mdstat [>....................] resync = 4.8% (504192/10483640) finish=13.9min speed=11920K/sec b. ``mdadm --detail /dev/md0`` :: # mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Sun Mar 24 11:01:46 2013 Raid Level : raid1 Array Size : 10483640 (10.00 GiB 10.74 GB) Used Dev Size : 10483640 (10.00 GiB 10.74 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Sun Mar 24 11:01:46 2013 State : clean, resyncing Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Resync Status : 6% complete Name : guest:0 (local to host guest) UUID : 7eeb50f9:cec710b8:7d491890:3310a328 Events : 0 Number Major Minor RaidDevice State 0 8 17 0 active sync /dev/sdb1 1 8 33 1 active sync /dev/sdc1 5. Standard lvm pvcreate/vgcreate/lvcreate 6. Update mdadm.conf: :: # mdadm --detail --scan | tee /etc/mdadm.conf ARRAY /dev/md/guest:0 metadata=1.2 name=guest:0 UUID=7eeb50f9:cec710b8:7d491890:3310a328 7. Reboot/verify filesystem is available. 8. Manual failure of a drive: :: # mdadm -f /dev/md0 /dev/sdb1 mdadm: set /dev/sdb1 faulty in /dev/md0 # mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Sun Mar 24 11:01:46 2013 Raid Level : raid1 Array Size : 10483640 (10.00 GiB 10.74 GB) Used Dev Size : 10483640 (10.00 GiB 10.74 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Sun Mar 24 11:59:01 2013 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 1 Spare Devices : 0 Name : guest:0 (local to host guest) UUID : 7eeb50f9:cec710b8:7d491890:3310a328 Events : 20 Number Major Minor RaidDevice State 0 0 0 0 removed 1 8 33 1 active sync /dev/sdc1 0 8 17 - faulty spare /dev/sdb1 # cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb1[0](F) sdc1[1] 10483640 blocks super 1.2 [2/1] [_U] 9. Remove and replace the drive: :: # mdadm -r /dev/md0 /dev/sdb1 mdadm: hot removed /dev/sdb1 from /dev/md0 # cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdc1[1] 10483640 blocks super 1.2 [2/1] [_U] unused devices: # mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Sun Mar 24 11:01:46 2013 Raid Level : raid1 Array Size : 10483640 (10.00 GiB 10.74 GB) Used Dev Size : 10483640 (10.00 GiB 10.74 GB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Update Time : Sun Mar 24 12:04:39 2013 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Name : guest:0 (local to host guest) UUID : 7eeb50f9:cec710b8:7d491890:3310a328 Events : 23 Number Major Minor RaidDevice State 0 0 0 0 removed 1 8 33 1 active sync /dev/sdc1 # mdadm --add /dev/md0 /dev/sdb1 mdadm: added /dev/sdb1 # mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Sun Mar 24 11:01:46 2013 Raid Level : raid1 Array Size : 10483640 (10.00 GiB 10.74 GB) Used Dev Size : 10483640 (10.00 GiB 10.74 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Sun Mar 24 12:05:37 2013 State : clean, degraded, recovering Active Devices : 1 Working Devices : 2 Failed Devices : 0 Spare Devices : 1 Rebuild Status : 0% complete Name : guest:0 (local to host guest) UUID : 7eeb50f9:cec710b8:7d491890:3310a328 Events : 29 Number Major Minor RaidDevice State 2 8 17 0 spare rebuilding /dev/sdb1 1 8 33 1 active sync /dev/sdc1 10. Dead drive: a. virsh destroy guest; virsh detach-disk guest hdc --persistent; virsh vol-delete ${file} default; virsh start guest b. error conditions: :: # mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Sun Mar 24 11:01:46 2013 Raid Level : raid1 Array Size : 10483640 (10.00 GiB 10.74 GB) Used Dev Size : 10483640 (10.00 GiB 10.74 GB) **Raid Devices : 2** **Total Devices : 1** Persistence : Superblock is persistent Update Time : Sun Mar 24 13:25:50 2013 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Name : guest:0 (local to host guest) UUID : 7eeb50f9:cec710b8:7d491890:3310a328 Events : 48 Number Major Minor RaidDevice State 2 8 17 0 active sync /dev/sdb1 1 0 0 1 removed # cat /proc/mdstat Personalities : [raid1] md0 : active (auto-read-only) raid1 sdb1[2] 10483640 blocks super 1.2 [2/1] [U_] 11. Replace/repair: