RHEL clone and UEFI booting, aka the dreaded unable to mount initrd error

[zix]

Hello Forum,

I had trouble installing Oracle Enterprise Linux (OEL) 7.2 using ipxe.efi the other day. OEL is a Redhat Enterprise Linux (RHEL) clone similar to CentOS and Scientific Linux. I noticed several users of this forum encountered similar to difficulties as myself during installation, namely that initrd wasn't being mounted correctly.

The failure occurred on an UEFI enabled HP Proliant DL360 Gen 9. I was also able to get the installation to fail on an UEFI enabled virtual machine in VMware Fusion 8.5. At first, both the HP physical server and the virtual machine on my laptop were failing in a similar manner with the same kernel panic.

Quote:Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I found the suggestion to reverse the kernel and initrd lines in the iPXE configuration on this forum. I modified my ipxe script and, interestingly, this allowed the virtual machine to install! The HP still kernel panicked, though.

A close inspection of the kernel logs turned up this error:

Quote:Initramfs unpacking failed: XZ-compressed data is corrupt

By default, RHEL based systems use XZ to compress their initial ram disks since the XZ algorithm provides excellent compression compared to other algorithms. As far as I could tell, the kernel's de-compression function was detecting errors in the data loaded into RAM by iPXE.

initrd.img is a cpio archive compressed with xz (or another compression algorithm supported by the kernel) and is easy to decompress and re-compress. I next tried a gzipped compressed initrd.img and got:

Quote:Unpacking initramfs...
Initramfs unpacking failed: junk in compressed archive

How odd. I then tried a raw initrd.img cpio archive with no compression:

Quote:Unpacking initramfs...
Freeing initrd memory: 134472k freed

An uncompressed image file worked! I was able to install OEL 7.2 without trouble using the uncompressed image. I haven't tested other operating systems or servers other than the HP and VM. My wild speculation since I don't know anything about the iPXE source is that the image binary is being mangled somehow while it's being pulled into RAM by iPXE.

Note that I did not run into any of this diffculty using BIOS iPXE.

It's easy to remove the xz compression from the default initrd.img.

Code:

mv initrd.img initrd.img.xz
xz -d initrd.img.xz

My iPXE script file:

Code:

#!ipxe

set base http://192.168.1.20/oracle/7.2

initrd --name initramfs ${base}/images/initrd.img
kernel ${base}/images/pxeboot/vmlinuz initrd=initramfs repo=${base} ip=dhcp console=tty0

[NiKiZe]

Great that you got a workaround for this particular case on the specific hardware. But I do believe that this is due to bug in old kernels

see this post and the linked replies from there: http://forum.ipxe.org/showthread.php?tid...6#pid11866

and if you can please post the output of uname -r so we can verify if that is a old kernel or not.

[zix]

(2016-10-02 22:32)NiKiZe Wrote:  Great that you got a workaround for this particular case on the specific hardware. But I do believe that this is due to bug in old kernels

see this post and the linked replies from there: http://forum.ipxe.org/showthread.php?tid...6#pid11866

and if you can please post the output of uname -r so we can verify if that is a old kernel or not.

The kernel version is a bit behind. This is the default kernel that comes with the 7.2 installation media:

Code:

$ file vmlinuz
vmlinuz: Linux kernel x86 boot executable bzImage, version 3.10.0-327.el7.x86_64 (mockbuild@x86-ol7-builder-01) #1 SMP Fri, RO-rootFS, swap_dev 0x4, Normal VGA

[NiKiZe]

(2016-10-04 15:49)zix Wrote:  

Code:

version 3.10.0-327.el7.x86_64

Ok so yes, that i probably a buggy kernel and should not be used for efi boot with initrd. I'd say it's pure luck if it works.

Quite sad to see this come up again and again just because RH refuses to release something decent (IMHO) /rant

[pxe_curious]

Just wanted to ask what exactly happens when iPXE loads a compressed image file:
- Does iPXE decompress the file or does the kernel that gets loaded and uses that file do that?
- Does the next command (lets say, for example, loading the initrd image file and then the kernel) wait until the previous command has completed? I would think so.

Just trying to understand if the bug is inside the kernel where it does not decompress the initrd.img properly or if iPXE is slow loading and expanding the imagefile and when the kernel gets executed, it does not find the file and therefore panics.

[NiKiZe]

iPXE does nothing, however, if a http server responds with a gzip stream that stream is decompressed, some http servers does send such headers for .gz which means you will have a uncompressed image in memory.

Yes everything is done one thing at a time.

And kernels pre 3.16 is buggy in regards to handling initrd in efi mode - it is corruption of the image itself - you could always try to decompress initrd server side and load it uncompressed in case that triggers the bug.

I would look for the original patch that was added in 3.16 to fix this issue to see what exactly is going on, but I don't think there is no way to avoid it - you are just lucky if it doesn't happen.