System fails to boot on kernel update it hangs on "running cleanup hook"
System:-
CPU: Quad Core Intel Xeon E3-1285L v4 (-MT MCP-) speed/min/max: 2032/800/3800 MHz Current Kernel: 5.6.15-1-next x86_64
Graphics: Device-1: Intel Iris Pro Graphics P6300 driver: i915 v: kernel
Command line: options systemd.show_status=0 root=PARTUUID=6be92431-1135-5b4d-9489-380c78427f96 resume=UUID=ec443000-2e95-44b4-9ff9-c29281884600 rw

Post the output of the update log. Did the mkinitcpio command run without errors? How can system show Current Kernel: 5.6.15-1-next x86_64 if you have 5.7.0 installed? I presume you kept the regular kernel, and booted from there to get the above output?

Current kernel has been downgraded to 5.6.15-1- next, until this is sorted.
No errors on on mkinitcpio, also reinstalled the new kernel, no difference.
On the same PC but on another disk and another Kaos install it gives same problem.
extract from todays pacman log.
[2020-06-10T08:30:08+0100] [PACMAN] Running 'pacman -Syu'
[2020-06-10T08:30:08+0100] [PACMAN] synchronizing package lists
[2020-06-10T08:30:09+0100] [PACMAN] starting full system upgrade
[2020-06-10T08:31:14+0100] [ALPM] transaction started
[2020-06-10T08:31:14+0100] [ALPM] upgraded bison (3.6.2-1 -> 3.6.3-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded iana-etc (20200511-1 -> 20200602-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded dbus (1.12.16-1 -> 1.12.18-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded iptables (1.8.4-1 -> 1.8.5-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded libgpg-error (1.37-1 -> 1.38-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded harfbuzz (2.6.6-1 -> 2.6.7-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded xkeyboard-config (2.29-1 -> 2.30-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded gnutls (3.6.13-3 -> 3.6.14-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded lcms2 (2.9-2 -> 2.10-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded librsvg (2.48.5-1 -> 2.48.6-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded libksba (1.3.5-2 -> 1.4.0-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded nss (3.52.1-1 -> 3.53-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded libyaml (0.2.2-1 -> 0.2.5-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded bluedevil (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:14+0100] [ALPM] upgraded kdecoration (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded breeze (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded breeze-gtk (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded drkonqi (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded git (2.26.2-1 -> 2.27.0-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded glib-networking (2.64.2-1 -> 2.64.3-2)
[2020-06-10T08:31:15+0100] [ALPM] upgraded widevine (85.0.4158.4-1 -> 85.0.4164.2-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded pciutils (3.6.4-1 -> 3.7.0-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded google-chrome (85.0.4158.4-1 -> 85.0.4164.2-1)
[2020-06-10T08:31:15+0100] [ALPM-SCRIPTLET] [0;32m To have better integration with KaOS
[2020-06-10T08:31:15+0100] [ALPM-SCRIPTLET] [0;32m Select Use GTK+ theme, this will follow your KDE settings
[2020-06-10T08:31:15+0100] [ALPM] upgraded imagemagick (6.9.11.15-1 -> 6.9.11.16-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded iproute2 (5.6.0-1 -> 5.7.0-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded kactivitymanagerd (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:15+0100] [ALPM] installed plasma-wayland-protocols (1.0-1)
[2020-06-10T08:31:15+0100] [ALPM] installed kwayland-server (5.19.0-1)
[2020-06-10T08:31:15+0100] [ALPM] upgraded kscreenlocker (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:16+0100] [ALPM] upgraded kwin (5.18.5-2 -> 5.19.0-3)
[2020-06-10T08:31:16+0100] [ALPM] upgraded libksysguard (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:16+0100] [ALPM] upgraded milou (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:16+0100] [ALPM] upgraded libkscreen (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:16+0100] [ALPM] upgraded qt5-tools (5.15.0-1 -> 5.15.0-2)
[2020-06-10T08:31:16+0100] [ALPM] upgraded sddm-kcm (1:5.18.5-1 -> 1:5.19.0-1)
[2020-06-10T08:31:16+0100] [ALPM] upgraded plasma-workspace (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded kde-cli-tools (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded kde-gtk-config (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded kdeplasma-addons (5.18.5-2 -> 5.19.0-2)
[2020-06-10T08:31:17+0100] [ALPM] upgraded khotkeys (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded kinfocenter (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded kmenuedit (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded kscreen (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded ksshaskpass (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded ksysguard (5.18.5-1 -> 5.19.0-2)
[2020-06-10T08:31:17+0100] [ALPM] upgraded kwallet-pam (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:17+0100] [ALPM] upgraded libktorrent (2.1.1-1 -> 2.2.0-1)
[2020-06-10T08:31:17+0100] [ALPM] warning: directory permissions differ on /boot/
filesystem: 700 package: 755
[2020-06-10T08:31:18+0100] [ALPM] upgraded linux-next (5.6.15-1 -> 5.7.0-1)
[2020-06-10T08:31:18+0100] [ALPM-SCRIPTLET] >>> Updating module dependencies. Please wait ...
[2020-06-10T08:31:22+0100] [ALPM] upgraded linux-next-headers (5.6.15-1 -> 5.7.0-1)
[2020-06-10T08:31:22+0100] [ALPM] upgraded man-db (2.9.1-1 -> 2.9.2-1)
[2020-06-10T08:31:22+0100] [ALPM] upgraded messagelib (20.04.1-1 -> 20.04.1-2)
[2020-06-10T08:31:22+0100] [ALPM] upgraded pepper-flash (32.0.0.371-1 -> 32.0.0.387-1)
[2020-06-10T08:31:22+0100] [ALPM-SCRIPTLET] [0;32m Firefox users make sure to add freshplayerplugin
[2020-06-10T08:31:22+0100] [ALPM-SCRIPTLET] [0;32m sudo pacman -S freshplayerplugin
[2020-06-10T08:31:22+0100] [ALPM-SCRIPTLET] [0;32m This way it is no longer needed to use adobe flash [0m
[2020-06-10T08:31:22+0100] [ALPM] upgraded polkit-kde-agent-1 (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:22+0100] [ALPM] upgraded systemsettings (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:22+0100] [ALPM] upgraded powerdevil (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:22+0100] [ALPM] upgraded plasma-integration (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded plasma-desktop (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded plasma-nm (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded plasma-pa (5.18.5-1 -> 5.19.0-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded plasma-workspace-wallpapers (5.18.0-1 -> 5.19.0-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded psutils (1.93-1 -> 2.03-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded seafile (7.0.7-1 -> 7.0.8-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded seafile-client (7.0.7-1 -> 7.0.8-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded strawberry (0.6.11-2 -> 0.6.12-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded telegram-desktop (2.1.7-1 -> 2.1.11-1)
[2020-06-10T08:31:23+0100] [ALPM] upgraded virtualbox-modules-next (6.1.8-3 -> 6.1.10-1)
[2020-06-10T08:31:25+0100] [ALPM] upgraded virtualbox-qt5 (6.1.8-1 -> 6.1.10-1)
[2020-06-10T08:31:25+0100] [ALPM-SCRIPTLET] 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
[2020-06-10T08:31:25+0100] [ALPM] upgraded virtualbox-ext-oracle (6.1.8-1 -> 6.1.10-1)
[2020-06-10T08:31:26+0100] [ALPM-SCRIPTLET] 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
[2020-06-10T08:31:26+0100] [ALPM] transaction completed
[2020-06-10T08:31:27+0100] [ALPM] running '99-linux-next.hook'...
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-next.preset: 'default'
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux-next -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-next.img
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] ==> Starting build: 5.7.0-1-next
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2020-06-10T08:31:27+0100] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2020-06-10T08:31:28+0100] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2020-06-10T08:31:28+0100] [ALPM-SCRIPTLET] -> Running build hook: [resume]
[2020-06-10T08:31:28+0100] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2020-06-10T08:31:28+0100] [ALPM-SCRIPTLET] -> Running build hook: [fsck]
[2020-06-10T08:31:28+0100] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2020-06-10T08:31:28+0100] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-next.img
[2020-06-10T08:31:29+0100] [ALPM-SCRIPTLET] ==> Image generation successful
[2020-06-10T08:31:29+0100] [ALPM] running 'gdk-pixbuf-query-loaders.hook'...
[2020-06-10T08:31:29+0100] [ALPM] running 'gio-querymodules.hook'...
[2020-06-10T08:31:29+0100] [ALPM] running 'gtk-update-icon-cache.hook'...
[2020-06-10T08:31:29+0100] [ALPM] running 'systemd-tmpfiles.hook'...
[2020-06-10T08:31:29+0100] [ALPM] running 'texinfo-install.hook'...
[2020-06-10T08:31:29+0100] [ALPM] running 'update-desktop-database.hook'...
[2020-06-10T08:31:29+0100] [ALPM] running 'update-mime-database.hook'...
[2020-06-10T13:47:09+0100] [PACMAN] Running '/usr/bin/pacman -S core/linux-next core/linux-next-headers'
[2020-06-10T13:47:12+0100] [ALPM] transaction started
[2020-06-10T13:47:12+0100] [ALPM] warning: directory permissions differ on /boot/
filesystem: 700 package: 755
[2020-06-10T13:47:12+0100] [ALPM] reinstalled linux-next (5.7.0-1)
[2020-06-10T13:47:12+0100] [ALPM-SCRIPTLET] >>> Updating module dependencies. Please wait ...
[2020-06-10T13:47:15+0100] [ALPM] reinstalled linux-next-headers (5.7.0-1)
[2020-06-10T13:47:15+0100] [ALPM] transaction completed
[2020-06-10T13:47:15+0100] [ALPM] running '99-linux-next.hook'...
[2020-06-10T13:47:15+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-next.preset: 'default'
[2020-06-10T13:47:15+0100] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux-next -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-next.img
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] ==> Starting build: 5.7.0-1-next
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [resume]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2020-06-10T13:47:16+0100] [ALPM-SCRIPTLET] -> Running build hook: [fsck]
[2020-06-10T13:47:17+0100] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2020-06-10T13:47:17+0100] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-next.img
[2020-06-10T13:47:18+0100] [ALPM-SCRIPTLET] ==> Image generation successful
[2020-06-10T14:22:21+0100] [PACMAN] Running 'pacman -U linux-next-5.6.15-1-x86_64.pkg.tar.zst linux-next-headers-5.6.15-1-x86_64.pkg.tar.zst'
[2020-06-10T14:23:47+0100] [PACMAN] Running 'pacman -U linux-next-5.6.15-1-x86_64.pkg.tar.zst linux-next-headers-5.6.15-1-x86_64.pkg.tar.zst virtualbox-modules-next-6.1.8-2-x86_64.pkg.tar.zst'
[2020-06-10T14:23:52+0100] [ALPM] transaction started
[2020-06-10T14:23:52+0100] [ALPM] warning: directory permissions differ on /boot/
filesystem: 700 package: 755
[2020-06-10T14:23:53+0100] [ALPM] downgraded linux-next (5.7.0-1 -> 5.6.15-1)
[2020-06-10T14:23:53+0100] [ALPM-SCRIPTLET] >>> Updating module dependencies. Please wait ...
[2020-06-10T14:23:57+0100] [ALPM] downgraded linux-next-headers (5.7.0-1 -> 5.6.15-1)
[2020-06-10T14:23:57+0100] [ALPM] downgraded virtualbox-modules-next (6.1.10-1 -> 6.1.8-2)
[2020-06-10T14:23:58+0100] [ALPM] transaction completed
[2020-06-10T14:23:59+0100] [ALPM] running '99-linux-next.hook'...
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-next.preset: 'default'
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux-next -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-next.img
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] ==> Starting build: 5.6.15-1-next
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [resume]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2020-06-10T14:23:59+0100] [ALPM-SCRIPTLET] -> Running build hook: [fsck]
[2020-06-10T14:24:00+0100] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2020-06-10T14:24:00+0100] [ALPM-SCRIPTLET] ==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-next.img
[2020-06-10T14:24:01+0100] [ALPM-SCRIPTLET] ==> Image generation successful

Can you get the journal log from the failed boot? Will need to see the exact errors to see why your hardware fails to boot with 5.7 (amd & intel based hardware here, no issue).
Also try to get more info from where it fails by editing the kernel line from Command line: options systemd.show_status=0 to systemd.show_status=1

Thanks Demm. I will do shortly.
5.7 is working OK on an older Amd machine and I think it was on this machine in a vbox system.

Seems the boot is going past the kernel loading, starting graphical interface and login manager. What exactly is the issue? You do not see SDDM? Boot into a black screen with mouse pointer/ command prompt?
Best to also post Octopi > tools > SysInfo from a successful boot (either linux or linux-next), so it is clear what hardware and graphics is affected.

System just locks up on reaching "running cleanup hook"
No usage possible, even after a long wait. Only option is to power off.
More info tomorrow.
Thanks.

pacman -Qi ethtool
and
pacman -Qi nvme-cli
You have a ton of packages not from the repo (quite a few used to be in, but had been removed, some well over a year ago), are any of those kernel modules? Make sure to rebuild those.

OK Demm,
I have deleted/removed a load of the extraneous packages, stopped all systemd processes (where possible) at boot, tried a clean desktop, ran disk checks, etc,etc. Tried many things and MANY reboots to 5.7.0-1. All hang. Removing "quiet" from command line shows boot stopping at "update UTMP about system Boot/Shutdown" otherwise it shows "reached cleanup hook".
I have rerun mkinitcpio producing no difference.
It may well be video related as the boot text changes to small immediately on starting 5.6 but stays at large on 5.7 where it hangs.
I will find time to install the new ISO on another disk and try that in due course and update you.
Many thanks for your input.

No need to try the ISO, if the extra packages are not kernel modules (and now removed), it clearly is just the new kernel and your hardware (test ISO still has 5.6 on it).
So far no idea yet were the bug is, nor have I seen any related bug reports (similar to what you report).
Linux-next 5.7.1 is about to move to core, 5.7.2 is in build, maybe the fix is already in.....

SUCCESS Demm.
Changed the "/boot/loader/entries/XXX" by removing the " / "in the linux and initrd entries as suggested in some "archlinux posts regarding 5.7.0 "and then rerunning "bootctl update".
Removing the " /" on its own made no difference.
Thanks for the help, happy now.:

EDIT.
It wasn't the "/" that made the difference it must have been "bootctl update". The boot screen text is very small now. I attached a new large screen 3 or 4 months ago so perhaps that what it was hanging on.

EDIT-2.
Scrub all that nonsense. IT WAS "disable CSM Support" in the BIOS that prevents it from loading.

  • demm replied to this.

    I see a lot of issues being reported about linux 5.7 the last 2 days, wonder why so few are hit with it in KaOS so far (yes, active KaOS users do use linux-next 🙂, they report in IRC very often).
    Good to know what fixed your issue.

    5 days later

    nezzie Scrub all that nonsense. IT WAS "disable CSM Support" in the BIOS that prevents it from loading.

    Linux-next moved to 5.7.3, with new intel-ucode, if you have a change, please test again sometime with CSM support disabled and report back if boot still fails then.

      Well I've now tried and there is no difference in the booting.
      If "CSM Support=Yes" it locks up, if nomodeset is on the commandline then it boots completely to a black screen and then the monitor produces a message that it fails to receive to a signal.
      If "CSM Support=No" It boots and operates with no trouble at all, apart from minuscule text on the systemd menu being almost unreadable.

      I hope you don't mind to keep trying with each linux-next update, since this is a blocker for 5.7 to ever be used as stable kernel.

      No trouble Demm. Just wish one could specify a screen resolution in systemd boot.
      I don't know if the efi stub takes its screen size from the one attached so my next effort is
      going to try connecting a smaller monitor and then reinstalling the efi and then swapping monitors back.

      • demm replied to this.