Attempt to clone KaOS to btrfs and LVM

Hello all,

I tried to clone my btrfs installation to a logical volume of LVM (Logical Volume Manager). All my Linux installation are organized in an LVM volume group.

Calamares installs to the root subvolume of btrfs (subvolume id=5). That is very inconvenient, as I do not know any way to delete it when needed. If you change to another subvolume as default, the root subvolume can never be deleted.

So I configured subvolumes in openSUSE style, so that the default subvolume and later root mount will be a read-write snapshot.

Creating first custom subvolumes (mounted from another Linux install)

# sh ./cr_subvol_k.sh /dev/mapper/vg1-mlvm08

Create subvolume '/mnt/@@@'

Create subvolume '/mnt/@@@/.snapshots'

Create subvolume '/mnt/@@@/home'

Create subvolume '/mnt/@@@/opt'

Create subvolume '/mnt/@@@/srv'

Create subvolume '/mnt/@@@/tmp'

Create subvolume '/mnt/@@@/boot/grub/x86_64-efi'

Create subvolume '/mnt/@@@/boot/grub/i386-pc'

Create subvolume '/mnt/@@@/usr/local'

Create subvolume '/mnt/@@@/var/log'

Create subvolume '/mnt/@@@/var/opt'

Create subvolume '/mnt/@@@/var/spool'

Create subvolume '/mnt/@@@/var/tmp'

Create subvolume '/mnt/var/lib/machines'

Create a snapshot of '/mnt/@@@' in '/mnt/@@@/.snapshots/1/snapshot'

The home directory, temporary directories and log files are in special subvolumes and excluded from later snapshots and recoveries.

# btrfs subvolume list /mnt

ID 257 gen 2175 top level 5 path @@@

ID 258 gen 2177 top level 257 path @@@/.snapshots

ID 259 gen 2179 top level 257 path @@@/home

ID 260 gen 2181 top level 257 path @@@/opt

ID 261 gen 2183 top level 257 path @@@/srv

ID 262 gen 2185 top level 257 path @@@/tmp

ID 263 gen 2187 top level 257 path @@@/boot/grub/x86_64-efi

ID 264 gen 2189 top level 257 path @@@/boot/grub/i386-pc

ID 265 gen 2191 top level 257 path @@@/usr/local

ID 266 gen 2193 top level 257 path @@@/var/log

ID 267 gen 2195 top level 257 path @@@/var/opt

ID 268 gen 2197 top level 257 path @@@/var/spool

ID 269 gen 2199 top level 257 path @@@/var/tmp

ID 270 gen 2201 top level 5 path var/lib/machines

ID 271 gen 2203 top level 258 path @@@/.snapshots/1/snapshot

Set the snapshot as default subvolume:

# btrfs subv set-default 271 /mnt

Note that I use grub in my KaOS installation. The grub subvolumes are not necessary in a standard KaOS installation. I presume that grub will exclude binaries that must stay contiguous and aligned to the sector boundaries from the btrfs COW mechanism (Copy-on-Write).

Mount the subvolumes

# sh ./mnt_subvol_k.sh /dev/mapper/vg1-mlvm08

List the mounts

# mount -l | grep mlvm08

/dev/mapper/vg1-mlvm08 on /mnt type btrfs (rw,relatime,space_cache,subvolid=271,subvol=/@@@/.snapshots/1/snapshot) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/home type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@@@/home) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/srv type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@@@/srv) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/opt type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@@@/opt) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/tmp type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@@@/tmp) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/.snapshots type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@@@/.snapshots) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/boot/grub/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@@@/boot/grub/x86_64-efi) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/boot/grub/i386-pc type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@@@/boot/grub/i386-pc) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/usr/local type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@@@/usr/local) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/var/log type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@@@/var/log) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/var/opt type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@@@/var/opt) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/var/spool type btrfs (rw,relatime,space_cache,subvolid=268,subvol=/@@@/var/spool) [rt-kaosx-3]

/dev/mapper/vg1-mlvm08 on /mnt/var/tmp type btrfs (rw,relatime,space_cache,subvolid=269,subvol=/@@@/var/tmp) [rt-kaosx-3]

Copy the installation to the new device. Former snapshots are not needed and excluded.

# ls /media/sda12/. /mnt

/media/sda12/.:

bin boot btrfs dev etc home klauncherXM1859.1.slave-socket lib lib64 media mnt opt proc root root-image-pkgs.txt run sbin srv sys tmp usr var

/mnt:

boot home opt srv tmp usr var

#

# rsync -aHAX --exclude=/.snapshots /media/sda12/. /mnt &

[1] 2814

#

[1]+ Fertig rsync -aHAX --exclude=/.snapshots /media/sda12/. /mnt

List of scripts

# cat cr_subvol_k.sh

#!/usr/bin/sh

mount $1 /mnt

btrfs subv create /mnt/@@@

btrfs subv create /mnt/@@@/.snapshots

btrfs subv create /mnt/@@@/home

btrfs subv create /mnt/@@@/opt

btrfs subv create /mnt/@@@/srv

btrfs subv create /mnt/@@@/tmp

mkdir /mnt/@@@/boot

mkdir /mnt/@@@/boot/grub

btrfs subv create /mnt/@@@/boot/grub/x86_64-efi

btrfs subv create /mnt/@@@/boot/grub/i386-pc

mkdir /mnt/@@@/usr

btrfs subv create /mnt/@@@/usr/local

mkdir /mnt/@@@/var

btrfs subv create /mnt/@@@/var/log

btrfs subv create /mnt/@@@/var/opt

btrfs subv create /mnt/@@@/var/spool

btrfs subv create /mnt/@@@/var/tmp

mkdir /mnt/var

mkdir /mnt/var/lib

btrfs subv create /mnt/var/lib/machines

mkdir /mnt/@@@/.snapshots/1

btrfs subv snapshot /mnt/@@@ /mnt/@@@/.snapshots/1/snapshot

# cat mnt_subvol_k.sh

#!/usr/bin/sh

# set -x

mount $1 /mnt

mount $1 -o subvol=@@@/home /mnt/home

mount $1 -o subvol=@@@/srv /mnt/srv

mount $1 -o subvol=@@@/opt /mnt/opt

mount $1 -o subvol=@@@/tmp /mnt/tmp

mount $1 -o subvol=@@@/.snapshots /mnt/.snapshots

mount $1 -o subvol=@@@/boot/grub/x86_64-efi /mnt/boot/grub/x86_64-efi

mount $1 -o subvol=@@@/boot/grub/i386-pc /mnt/boot/grub/i386-pc

mount $1 -o subvol=@@@/usr/local /mnt/usr/local

mount $1 -o subvol=@@@/var/log /mnt/var/log

mount $1 -o subvol=@@@/var/opt /mnt/var/opt

mount $1 -o subvol=@@@/var/spool /mnt/var/spool

mount $1 -o subvol=@@@/var/tmp /mnt/var/tmp

To be continued

The next steps are done from the old KaOS installation.

Mount the new device and the custom subvolumes (script mnt_subv_k.sh)

Chroot into the new filesystem

# sh ./chroot.sh /dev/mapper/vg1-mlvm08

+ mount /dev/mapper/vg1-mlvm08 /mnt

mount: /dev/mapper/vg1-mlvm08 is already mounted or /mnt busy

/dev/mapper/vg1-mlvm08 is already mounted on /mnt

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/home

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/srv

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/opt

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/tmp

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/.snapshots

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/boot/grub/x86_64-efi

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/boot/grub/i386-pc

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/usr/local

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/var/log

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/var/opt

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/var/spool

/dev/mapper/vg1-mlvm08 is already mounted on /mnt/var/tmp

+ rm -rf /tmp/chr

+ mkdir /tmp/chr

+ mv /mnt/etc/resolv.conf /mnt/etc/resolv.conf.bak /mnt/etc/resolvconf.conf /tmp/chr

+ cp -a /etc/resolv.conf /etc/resolv.conf.bak /etc/resolvconf.conf /mnt/etc

+ mount -t proc proc /mnt/proc

+ mount -t sysfs sys /mnt/sys

+ mount -o bind /dev /mnt/dev

+ mount -o bind /run /mnt/run

+ export 'PS1=(chroot) '

+ PS1='(chroot) '

+ chroot /mnt

Create new initramfs with LVM support

[root@beq-kaos /]# cat /etc/mkinitcpio.conf | grep HOOK

# HOOKS

# This is the most important setting in this file. The HOOKS control the

# order in which HOOKS are added. Run 'mkinitcpio -H <hook name>' for

...

# HOOKS="base udev autodetect modconf block keyboard keymap filesystems"

HOOKS="base udev autodetect modconf block keyboard keymap lvm2 filesystems"

[root@beq-kaos /]# cd /boot

[root@beq-kaos boot]# mkinitcpio -p /etc/mkinitcpio.d/linux-next.preset -g linux-next

==> Building image from preset: /etc/mkinitcpio.d/linux-next.preset: 'default'

-> -k /boot/vmlinuz-linux-next -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-next.img

==> Starting build: 4.5.1-1-next

-> Running build hook: [base]

-> Running build hook: [udev]

-> Running build hook: [autodetect]

-> Running build hook: [modconf]

-> Running build hook: [block]

-> Running build hook: [keyboard]

-> Running build hook: [keymap]

-> Running build hook: [lvm2]

-> Running build hook: [filesystems]

==> Generating module dependencies

==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-next.img

==> Image generation successful

==> Building image from preset: /etc/mkinitcpio.d/linux-next.preset: 'fallback'

-> -k /boot/vmlinuz-linux-next -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-next-fallback.img -S autodetect

==> Starting build: 4.5.1-1-next

-> Running build hook: [base]

-> Running build hook: [udev]

-> Running build hook: [modconf]

-> Running build hook: [block]

==> WARNING: Possibly missing firmware for module: aic94xx

==> WARNING: Possibly missing firmware for module: wd719x

-> Running build hook: [keyboard]

-> Running build hook: [keymap]

-> Running build hook: [lvm2]

-> Running build hook: [filesystems]

==> Generating module dependencies

==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-next-fallback.img

==> Image generation successful

[root@beq-kaos boot]# mkinitcpio -p /etc/mkinitcpio.d/linux.preset -g linux

==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'

-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img

==> Starting build: 4.4.5-1

-> Running build hook: [base]

-> Running build hook: [udev]

-> Running build hook: [autodetect]

-> Running build hook: [modconf]

-> Running build hook: [block]

-> Running build hook: [keyboard]

-> Running build hook: [keymap]

-> Running build hook: [lvm2]

-> Running build hook: [filesystems]

==> Generating module dependencies

==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img

==> Image generation successful

==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'

-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect

==> Starting build: 4.4.5-1

-> Running build hook: [base]

-> Running build hook: [udev]

-> Running build hook: [modconf]

-> Running build hook: [block]

==> WARNING: Possibly missing firmware for module: aic94xx

==> WARNING: Possibly missing firmware for module: wd719x

-> Running build hook: [keyboard]

-> Running build hook: [keymap]

-> Running build hook: [lvm2]

-> Running build hook: [filesystems]

==> Generating module dependencies

==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img

==> Image generation successful

Adapt the /etc/fstab manually to the new device and subvolumes

[root@beq-kaos ~]# cat /etc/fstab

# /etc/fstab: static file system information.

#

# Use 'blkid' to print the universally unique identifier for a device; this may

# be used with UUID= as a more robust way to name devices that works even if

# disks are added and removed. See fstab(5).

#

# <file system> <mount point> <type> <options> <dump> <pass>

# UUID=5ffffabf-0291-44dd-a99e-0dfef48a4f29 / btrfs defaults,noatime,space_cache,autodefrag 0 1

# /dev/mapper/vg1-mlvm08: LABEL="rt-kaosx-3" UUID="78734088-e558-4130-b684-30226d82723b" UUID_SUB="d22f1aa9-284d-4e02-b56b-046bc73c0bc8" TYPE="btrfs"# /dev/mapper/vg1-mlvm08: LABEL="rt-kaosx-3" UUID="78734088-e558-4130-b684-30226d82723b" UUID_SUB="d22f1aa9-284d-4e02-b56b-046bc73c0bc8" TYPE="btrfs"

UUID=78734088-e558-4130-b684-30226d82723b / btrfs defaults 0 0

UUID=78734088-e558-4130-b684-30226d82723b /home btrfs subvol=/@@@/home 0 0

UUID=78734088-e558-4130-b684-30226d82723b /srv btrfs subvol=/@@@/srv 0 0

UUID=78734088-e558-4130-b684-30226d82723b /opt btrfs subvol=/@@@/opt 0 0

UUID=78734088-e558-4130-b684-30226d82723b /tmp btrfs subvol=/@@@/tmp 0 0

UUID=78734088-e558-4130-b684-30226d82723b /.snapshots btrfs subvol=/@@@/.snapshots 0 0

UUID=78734088-e558-4130-b684-30226d82723b /boot/grub/x86_64-efi btrfs subvol=/@@@/boot/grub/x86_64-efi 0 0

UUID=78734088-e558-4130-b684-30226d82723b /boot/grub/i386-pc btrfs subvol=/@@@/boot/grub/i386-pc 0 0

UUID=78734088-e558-4130-b684-30226d82723b /usr/local btrfs subvol=/@@@/usr/local 0 0

UUID=78734088-e558-4130-b684-30226d82723b /var/log btrfs subvol=/@@@/var/log 0 0

UUID=78734088-e558-4130-b684-30226d82723b /var/opt btrfs subvol=/@@@/var/opt 0 0

UUID=78734088-e558-4130-b684-30226d82723b /var/spool btrfs subvol=/@@@/var/spool 0 0

UUID=78734088-e558-4130-b684-30226d82723b /var/tmp btrfs subvol=/@@@/var/tmp 0 0

/dev/mapper/vg1-swap none swap sw 0 0

UUID=c5c1e954-85e0-4e49-bef9-fbe37a9d049c /media/lvm/vg1-mlvm1 ext4 defaults 0 2

/media/lvm/vg1-mlvm1/shared /home/bequimao/myshared none defaults,bind 0 0

KaOS puts a filesystem check at boot. This might be unnecessary and may cause problems, since btrfs has its own online filesystem checks.

Install grub (EFI) still in chroot mode

[root@beq-kaos boot]# mount /dev/sda3 /mnt

[root@beq-kaos boot]# grub-install --efi-directory=/mnt /dev/sda

Installing for x86_64-efi platform.

File descriptor 4 (/dev/sda3) leaked on vgs invocation. Parent PID 7562: grub-install

File descriptor 4 (/dev/sda3) leaked on vgs invocation. Parent PID 7562: grub-install

efibootmgr: EFI variables are not supported on this system.

efibootmgr: EFI variables are not supported on this system.

Installation finished. No error reported.

[root@beq-kaos boot]#

[root@beq-kaos boot]# grub-mkconfig -o /boot/grub/grub.cfg

Generating grub configuration file ...

Found linux image: /boot/vmlinuz-linux-next

Found initrd image: /boot/initramfs-linux-next.img

Found fallback initramfs image: /boot/initramfs-linux-next-fallback.img

Found linux image: /boot/vmlinuz-linux

Found initrd image: /boot/initramfs-linux.img

Found fallback initramfs image: /boot/initramfs-linux-fallback.img

Found KaOS (rolling) on /dev/sda12

Found openSUSE 42.1 (x86_64) on /dev/sda15

Found Windows Boot Manager on /dev/sda3@/EFI/Microsoft/Boot/bootmgfw.efi

Found Fedora release 23 (Twenty Three) on /dev/mapper/vg1-mlvm11

Found Debian GNU/Linux (8.4) on /dev/mapper/vg1-mlvm3

Found Debian GNU/Linux (stretch/sid) on /dev/mapper/vg1-mlvm4

Found Debian GNU/Linux (stretch/sid) on /dev/mapper/vg1-mlvm9

done

[root@beq-kaos boot]# umount /mnt

Script to chroot

[root@beq-kaos ~]# cat chroot.sh

#!/usr/bin/sh

set -x

mount $1 /mnt

rm -rf /tmp/chr

mkdir /tmp/chr

mv /mnt/etc/resolv* /tmp/chr

cp -a /etc/resolv* /mnt/etc

mount -t proc proc /mnt/proc

mount -t sysfs sys /mnt/sys

mount -o bind /dev /mnt/dev

mount -o bind /run /mnt/run

export PS1="(chroot) $PS1"

chroot /mnt

rm -rf /mnt/etc/resolv*

cp -a /tmp/chr/resolv* /mnt/etc

umount --recursive /mnt

Unfortunately boot of the new logical device did not work neither with the grub installed from KaOS nor with the grub from my main system (openSUSE Leap). The LVM volumes don't appear under /dev. The whole procedure worked fine with Mageia 6 (Cauldron). It worked with KaOS copied to a physical device. I do have now a lean btrfs installation.

The test volume is now deleted, but may easily be restored. I was asked to document the steps in irc. Does anyone have an idea, why the procedure failed on LVM?

Best regards,

Bequimão

If you plan to install snapper, that would not work. The subvolume .snapshots must not exist. Snapper will create it and fail.

No questions, no comments?

Bequimão