Linux LVM review¶
Notes that will eventually become the whitepaper¶
Notes¶
Linux lvm much bigger than originally anticipated.
-voption is verbose; multiple-v’s increase the verbosityto the point that, at least on the lvdisplay, it’s challenging the xterm scroll length.
lvmdiskscanused to scan for eligible disks
# lvmdiskscan -l
WARNING: only considering LVM devices
/dev/sda2 [ 19.51 GiB] LVM physical volume
/dev/sdb1 [ 10.00 GiB] LVM physical volume
/dev/sdc1 [ 10.00 GiB] LVM physical volume
0 LVM physical volume whole disks
3 LVM physical volumes
if using whole disks, which seems smarter, on the whole:
# lvmdiskscan -l
WARNING: only considering LVM devices
/dev/sda2 [ 19.51 GiB] LVM physical volume
/dev/sdb [ 10.00 GiB] LVM physical volume
/dev/sdc [ 10.00 GiB] LVM physical volume
2 LVM physical volume whole disks
1 LVM physical volume
Have to say, linux is trying to steal my thunder. Commands available that I wrote scripts to do on HP…
# pvscan
PV /dev/sdb VG vgmirror lvm2 [10.00 GiB / 10.00 GiB free]
PV /dev/sdc VG vgmirror lvm2 [10.00 GiB / 10.00 GiB free]
PV /dev/sda2 VG vg00 lvm2 [19.51 GiB / 0 free]
Total: 3 [39.50 GiB] / in use: 3 [39.50 GiB] / in no VG: 0 [0 ]
vgexport/import is a different too. much cleaner, easier than HPUX. no map files, it imports the lv names whereas hpux would simply call them lvol1 lvol2, etc, if no mapfile was used.
vgs, lvs, and pvs commands. Interesting output control capabilities
vgremove ${vg}to remove a volume group, unlike the HP vgexportfucking awesome! Splitting and rejoining vgs!
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 vg00 lvm2 a-- 19.51g 0
/dev/sdb vgmirror lvm2 a-- 10.00g 6.00g
/dev/sdc vgmirror lvm2 a-- 10.00g 10.00g
# vgsplit vgmirror vgsplit /dev/sdb
Logical volume "mirror" must be inactive
# mount | grep mirror
# lvchange -a n vgmirror/mirror
# vgsplit vgmirror vgsplit /dev/sdb
New volume group "vgsplit" successfully split from "vgmirror"
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 vg00 lvm2 a-- 19.51g 0
/dev/sdb vgsplit lvm2 a-- 10.00g 6.00g
/dev/sdc vgmirror lvm2 a-- 10.00g 10.00g
# vgchange -a n vgsplit
0 logical volume(s) in volume group "vgsplit" now active
# vgchange -a n vgmirror
0 logical volume(s) in volume group "vgmirror" now active
# vgmerge vgmirror vgsplit
Volume group "vgsplit" successfully merged into "vgmirror"
# vgs
VG #PV #LV #SN Attr VSize VFree
vg00 1 2 0 wz--n- 19.51g 0
vgmirror 2 1 0 wz--n- 19.99g 15.99g
Creating mirrors: some good points. a. Log files:
mirroring creates a log file on disk unless
--mirrorlog coreoption is used.Log files, by default, go on disks other than the ones being mirrored - makes sense. Might have to put them on the same disks, though, (for instance, mirroring the OS). Can do that by specifying:
--alloc anywhere
If creating mirrors > 1.5TB, update the region size:
lvcreate -L 2T -R 2 -n ... use factors of 2 every time you up the space: lvcreate -L 3T -R 4 -n ...
Another useful feature. You can watch the mirroring taking place:
# lvdisplay vgmirror/mirror
--- Logical volume ---
LV Path /dev/vgmirror/mirror
LV Name mirror
VG Name vgmirror
LV UUID xmcKu2-sYgV-CPbm-W6s6-FPcu-9Mwm-i7j8YF
LV Write Access read/write
LV Creation host, time guest, 2013-03-22 08:40:11 -0500
LV Status NOT available
LV Size 4.00 GiB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
# lvconvert -m 1 vgmirror/mirror
Conversion starts after activation.
# vgchange -a y vgmirror
1 logical volume(s) in volume group "vgmirror" now active
# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
root vg00 -wi-ao-- 17.54g
swap vg00 -wi-ao-- 1.97g
mirror vgmirror mwi-a-m- 4.00g mirror_mlog 9.67
# lvs vgmirror/mirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-a-m- 4.00g mirror_mlog 13.57
# lvs vgmirror/mirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-a-m- 4.00g mirror_mlog 15.04
# lvs vgmirror/mirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-a-m- 4.00g mirror_mlog 62.30
Initial steps: a. Assign disks, scan, get them visible to the OS. b. pvcreate, vgcreate, lvcreate:
# pvscan
PV /dev/sda2 VG vg00 lvm2 [19.51 GiB / 0 free]
Total: 1 [19.51 GiB] / in use: 1 [19.51 GiB] / in no VG: 0 [0 ]
# pvcreate /dev/sdb /dev/sdc
Writing physical volume data to disk "/dev/sdb"
Physical volume "/dev/sdb" successfully created
Writing physical volume data to disk "/dev/sdc"
Physical volume "/dev/sdc" successfully created
# vgcreate -s 4m vgmirror /dev/sd[bc]
Volume group "vgmirror" successfully created
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 vg00 lvm2 a-- 19.51g 0
/dev/sdb vgmirror lvm2 a-- 10.00g 10.00g
/dev/sdc vgmirror lvm2 a-- 10.00g 10.00g
# lvcreate -L 4g -m 1 -n mirror vgmirror
Logical volume "mirror" created
# lvs vgmirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-a-m- 4.00g mirror_mlog 13.77
# lvs vgmirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-a-m- 4.00g mirror_mlog 81.05
# lvs vgmirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-a-m- 4.00g mirror_mlog 100.00
# mkfs.ext4 /dev/vgmirror/mirror
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
[[snip]]
# mkdir -p -m 755 /mnt/mirror
# mount /dev/vgmirror/mirror /mnt/mirror
# mount /dev/vgmirror/mirror /mnt/mirror
# echo "this is a test. this is only a test." > /mnt/mirror/README
# lvconvert --splitmirrors 1 --name copy vgmirror/mirror
Logical volume mirror converted.
# lvs -o +devices vgmirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Devices
copy vgmirror -wi-a--- 4.00g /dev/sdc(0)
mirror vgmirror -wi-ao-- 4.00g /dev/sdb(0)
# lvchange -a n vgmirror/copy
# vgsplit vgmirror vgcopy /dev/sdc
New volume group "vgcopy" successfully split from "vgmirror"
# lvs -o +devices
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Devices
root vg00 -wi-ao-- 17.54g /dev/sda2(0)
swap vg00 -wi-ao-- 1.97g /dev/sda2(4490)
copy vgcopy -wi----- 4.00g /dev/sdc(0)
mirror vgmirror -wi-ao-- 4.00g /dev/sdb(0)
# mkdir /mnt/mirror1
# mount vgcopy/copy /mnt/mirror1
mount: you must specify the filesystem type
# mount /dev/vgcopy/copy /mnt/mirror1
# md5sum /mnt/mirror/README /mnt/mirror1/README
a8f852b96dffa74d7135ff35c10677f6 /mnt/mirror/README
a8f852b96dffa74d7135ff35c10677f6 /mnt/mirror1/README
OK, now let’s put it back:
# umount /mnt/mirror1
# vgchange -a n vgcopy
0 logical volume(s) in volume group "vgcopy" now active
# lvconvert -m 1 vgmirror/mirror
vgmirror/mirror: Converted: 0.0%
^C
# lvs vgmirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-aom- 4.00g mirror_mlog 76.17
# lvs vgmirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-aom- 4.00g mirror_mlog 100.00
So, now, we blow away a device to simulate a failure.
# virsh detach-disk guest hdd --persistent
# run on vmhost, need to figure out how to ID correct dev w/o reading xml file directly
# vgreduce vgmirror --removemissing --force
Couldn't find device with uuid mjpoEn-pZGd-ocew-oR6N-2E71-0eVX-3fKg6Y.
Wrote out consistent volume group vgmirror
# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
root vg00 -wi-ao-- 17.54g
swap vg00 -wi-ao-- 1.97g
mirror vgmirror -wi----- 4.00g
# lvdisplay /dev/vgmirror/mirror
--- Logical volume ---
LV Path /dev/vgmirror/mirror
LV Name mirror
VG Name vgmirror
LV UUID 0OpUgK-HSPc-eIA7-5DcG-ALnE-dY95-NPOrGu
LV Write Access read/write
LV Creation host, time guest, 2013-03-22 11:51:41 -0500
LV Status NOT available
LV Size 4.00 GiB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
# lvchange -a y vgmirror/mirror
# mount /dev/vgmirror/mirror /mnt/mirror
# md5sum /mnt/mirror/README
a8f852b96dffa74d7135ff35c10677f6 /mnt/mirror/README
Adding a new disk:
# virsh vol-create-as default guest-3.img 10g
# virsh attach-disk guest /var/lib/libvirt/images/guest\-3.img hdd --persistent
# virsh start guest
# pvcreate /dev/sdc
Writing physical volume data to disk "/dev/sdc"
Physical volume "/dev/sdc" successfully created
# vgextend vgmirror /dev/sdc
Volume group "vgmirror" successfully extended
# lvconvert -m 1 vgmirror/mirror
# lvs vgmirror
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mirror vgmirror mwi-a-m- 4.00g mirror_mlog 100.00
Software RAID review¶
Reset the environment using the script
Partition the disks, set partitiion type to fd (linux raid)
fdisk -uc /dev/sdc << eof
n
p
1
<ret>
<ret>
t
fd
w
eof
Create the raid partition:
# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
Status obtained in two ways: a.
/proc/mdstat
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
10483640 blocks super 1.2 [2/2] [UU]
[>....................] resync = 3.8% (407424/10483640) finish=13.6min speed=12320K/sec
unused devices: <none>
# grep resync /proc/mdstat
[>....................] resync = 4.8% (504192/10483640) finish=13.9min speed=11920K/sec
b. ``mdadm --detail /dev/md0``
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun Mar 24 11:01:46 2013
Raid Level : raid1
Array Size : 10483640 (10.00 GiB 10.74 GB)
Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sun Mar 24 11:01:46 2013
State : clean, resyncing
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Resync Status : 6% complete
Name : guest:0 (local to host guest)
UUID : 7eeb50f9:cec710b8:7d491890:3310a328
Events : 0
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
Standard lvm pvcreate/vgcreate/lvcreate
Update mdadm.conf:
# mdadm --detail --scan | tee /etc/mdadm.conf
ARRAY /dev/md/guest:0 metadata=1.2 name=guest:0 UUID=7eeb50f9:cec710b8:7d491890:3310a328
Reboot/verify filesystem is available.
Manual failure of a drive:
# mdadm -f /dev/md0 /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun Mar 24 11:01:46 2013
Raid Level : raid1
Array Size : 10483640 (10.00 GiB 10.74 GB)
Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sun Mar 24 11:59:01 2013
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
Name : guest:0 (local to host guest)
UUID : 7eeb50f9:cec710b8:7d491890:3310a328
Events : 20
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 33 1 active sync /dev/sdc1
0 8 17 - faulty spare /dev/sdb1
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[0](F) sdc1[1]
10483640 blocks super 1.2 [2/1] [_U]
Remove and replace the drive:
# mdadm -r /dev/md0 /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md0
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1]
10483640 blocks super 1.2 [2/1] [_U]
unused devices: <none>
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun Mar 24 11:01:46 2013
Raid Level : raid1
Array Size : 10483640 (10.00 GiB 10.74 GB)
Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Sun Mar 24 12:04:39 2013
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : guest:0 (local to host guest)
UUID : 7eeb50f9:cec710b8:7d491890:3310a328
Events : 23
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 33 1 active sync /dev/sdc1
# mdadm --add /dev/md0 /dev/sdb1
mdadm: added /dev/sdb1
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun Mar 24 11:01:46 2013
Raid Level : raid1
Array Size : 10483640 (10.00 GiB 10.74 GB)
Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sun Mar 24 12:05:37 2013
State : clean, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Rebuild Status : 0% complete
Name : guest:0 (local to host guest)
UUID : 7eeb50f9:cec710b8:7d491890:3310a328
Events : 29
Number Major Minor RaidDevice State
2 8 17 0 spare rebuilding /dev/sdb1
1 8 33 1 active sync /dev/sdc1
Dead drive:
virsh destroy guest; virsh detach-disk guest hdc –persistent; virsh vol-delete ${file} default; virsh start guest
error conditions:
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun Mar 24 11:01:46 2013
Raid Level : raid1
Array Size : 10483640 (10.00 GiB 10.74 GB)
Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
**Raid Devices : 2**
**Total Devices : 1**
Persistence : Superblock is persistent
Update Time : Sun Mar 24 13:25:50 2013
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : guest:0 (local to host guest)
UUID : 7eeb50f9:cec710b8:7d491890:3310a328
Events : 48
Number Major Minor RaidDevice State
2 8 17 0 active sync /dev/sdb1
1 0 0 1 removed
# cat /proc/mdstat
Personalities : [raid1]
md0 : active (auto-read-only) raid1 sdb1[2]
10483640 blocks super 1.2 [2/1] [U_]
Replace/repair: