HP X2 210 tablet (UEFI) hang

[SebastianRoth]

I am with the FOG team and try to help getting to properly boot iPXE on the HP X2 210 tablet - UEFI enabled, secure boot off.

Loading bzImage (tried 4.3.3 or 4.4.0, EFI_STUB=yes) and initrd seams fine (DEBUG=image_cmd,image,efi_image). File sizes match and imgstat looks good as well. See picture here: https://forums.fogproject.org/uploads/fi...59_pro.jpg

But booting it just sits there (15 minutes...) and nothing happens.

Edit: Thanks to Michael I now added DEBUG=efi_wrap and will report back as soon as we have new information. Posting this here as this might help others as well.

[SebastianRoth]

With added debug for efi_wrap we see this: https://forums.fogproject.org/uploads/fi...30_pro.jpg (The lines with green/yellow font color come from debug statements I added to the function efi_image_exec in src/image/efi_image.c to see where it actually starts hanging. As we see the call to console_reset() is fine but StartImage never returns which is fine I guess because the linux kernel should take over but we never see anything on the screen (added earlyprintk to the kernel as well without any change).

We also tried an older iPXE version just to make sure this is not a but being introduced at some point. Used https://git.ipxe.org/ipxe.git/snapshot/0...0f1.tar.gz (02.02.2015) but saw the exact same behavior. Please let me know if we should try another version.

Hints on more debugging steps we can try or anything that seams odd with those EFI warp handles are highly appreciated. Thanks.

[mcb30]

(2016-02-03 10:54)SebastianRoth Wrote:  Hints on more debugging steps we can try or anything that seams odd with those EFI warp handles are highly appreciated. Thanks.

Does that system stop at the point shown in that screenshot?

If so, then from looking at the kernel source this must be in arch/x86/boot/compressed/eboot.c in setup_efi_pci() which is the only place I can find that performs two calls to LocateHandle(PciIo) followed by a loop of calls to HandleProtocol(PciIo) (in setup_efi_pci64(), assuming this is a 64-bit UEFI).

The calls made to the PciIo protocol in setup_efi_pci64() won't show up in the DEBUG=efi_wrap output, since that intercepts only calls to the EFI_BOOT_SERVICES methods. You could potentially amend efi_wrap.c to also log calls to AllocatePool() and FreePool(), which would give you some visibility into progress through setup_efi_pci64(). Alternatively, you could build a kernel with extra debug code in setup_efi_pci64() to find out where it's crashing.

Good luck!

Michael

[SebastianRoth]

Wow, thanks a lot for the information. I'll look into this!

I forgot to say that we can boot this exact same kernel (an even older ones) from USB stick using GRUB2 bootloader in UEFI mode. I guess there is quite a difference between loading the kernel via GRUB2/disk and iPXE/network. Do you think it would be worth trying to pxe boot on this device using pxelinux or GRUB (over PXE) just to see if this works?

[mcb30]

(2016-02-03 11:58)SebastianRoth Wrote:  I forgot to say that we can boot this exact same kernel (an even older ones) from USB stick using GRUB2 bootloader in UEFI mode. I guess there is quite a difference between loading the kernel via GRUB2/disk and iPXE/network. Do you think it would be worth trying to pxe boot on this device using pxelinux or GRUB (over PXE) just to see if this works?

GRUB has the ability (predating CONFIG_EFI_STUB) to treat the kernel as a bzImage rather than an EFI executable. This enters the kernel via a different path, which probably bypasses the code we're looking at in eboot.c.

To test the kernel's EFI stub path without using iPXE, you could try to start the kernel directly from the UEFI shell.

Michael

[SebastianRoth]

Adding debugging statements to arch/x86/boot/compressed/eboot.c I am certain that it hangs exactly here: http://lxr.free-electrons.com/source/arc...oot.c#L473 when trying to setup the second PCI device 'PciRoot(0x0)/Pci(0x2(0x0)' - as see in the efi_wrap screenshot.

The call to allocate_pool in line 439 returns fine so I'd expect the memory allocation to be fine?! What can go wrong with memcpy then?

I've searched through mailing lists but couldn't find anyone else seeing an issue at that particular point. Does anyone have an idea what is wrong there?

Will try to boot the kernel as bootx64.efi from a USB stick to see if it works this way.

[SebastianRoth]

Update: Booting this kernel straight from a USB stick (FAT32, EFI\BOOT\bootx64.efi) ends up hanging/crashing on the same call! I am very sorry for all the rumor as it seams to not be an iPXE issue at all. We tested booting the kernel from USB before but using GRUB (as I didn't know this is bypassing efi_init).

Thank you very much for all your help!! Informing linux-efi mailing list now - wil keep you posted here as well.

[mcb30]

(2016-02-04 10:07)SebastianRoth Wrote:  The call to allocate_pool in line 439 returns fine so I'd expect the memory allocation to be fine?! What can go wrong with memcpy then?

What are the actual destination address, source address, and length for the memcpy() that fails?

Michael

[SebastianRoth]

(2016-02-04 14:14)mcb30 Wrote:  What are the actual destination address, source address, and length for the memcpy() that fails?

I tried to print things like addresses but failed because efi_printk can only print strings and I was unable to format integers to print with efi_char16_printk. snprintf does not seam to be usable within that early kernel code ("undefined reference to `sprintf'"). I am still quite new to the world of kernel debugging. How would I do this?

[mcb30]

(2016-02-04 15:47)SebastianRoth Wrote:  I tried to print things like addresses but failed because efi_printk can only print strings and I was unable to format integers to print with efi_char16_printk. snprintf does not seam to be usable within that early kernel code ("undefined reference to `sprintf'"). I am still quite new to the world of kernel debugging. How would I do this?

In the worst case, you can always brute-force it. Create a hideous hack of a function doing something like:

Code:

hideous_hack ( uint64_t value ) {
  static const char digits[16] = "0123456789abcdef";
  char buf[17];
  unsigned int i;

  for ( i = 0 ; i < 16 ; i++ )
    buf[i] = digits[ ( value >> ( 4 * ( 15 - i ) ) ) & 15 ];
  buf[16] = '\0';
  efi_printk ( sys_table, buf );
}

This untested code almost certainly has bugs, but it should give you the general idea.

Michael

[SebastianRoth]

(2016-02-04 16:02)mcb30 Wrote:  In the worst case, you can always brute-force it. Create a hideous hack of a function doing something like:

Code:

...

Thanks a lot for the quick hack. Here we have the output: https://forums.fogproject.org/uploads/fi...90-usb.jpg

We tested booting it from USB and via iPXE and the numbers are identical in both cases.

[mcb30]

(2016-02-05 12:58)SebastianRoth Wrote:  Thanks a lot for the quick hack. Here we have the output: https://forums.fogproject.org/uploads/fi...90-usb.jpg

We tested booting it from USB and via iPXE and the numbers are identical in both cases.

That length is several orders of magnitude too large to be plausible.

Michael

[SebastianRoth]

(2016-02-05 13:37)mcb30 Wrote:  That length is several orders of magnitude too large to be plausible.

Could that be a conversion/overflow issue caused by the conversion function (used pretty much what you posted)? Or do you think this is really what the pci handle returned?

Booting this kernel in qemu/OVMF (where it does not freeze) I see much smaller numbers for size/length: 9200 and 2d000. Does that sound plausible?

[mcb30]

(2016-02-05 17:27)SebastianRoth Wrote:  

(2016-02-05 13:37)mcb30 Wrote:  That length is several orders of magnitude too large to be plausible.

Could that be a conversion/overflow issue caused by the conversion function (used pretty much what you posted)? Or do you think this is really what the pci handle returned?

Booting this kernel in qemu/OVMF (where it does not freeze) I see much smaller numbers for size/length: 9200 and 2d000. Does that sound plausible?

I suspect that the EFI_PCI_IO_PROTOCOL provided by the firmware contains garbage values for RomSize and RomImage. Possible the firmware is failing to zero these fields for a device that does not actually have an expansion ROM BAR.

You could add code in iPXE's interface/efi/efi_wrap.c at the end of HandleProtocol() to dump out the contents of the returned EFI_PCI_IO_PROTOCOL structure. For example (again, untested):

Code:

if ( ( efirc == 0 ) &&
     ( memcmp ( protocol, &efi_pci_io_protocol_guid, sizeof ( *protocol ) ) == 0 ) ) {
  DBGC_HD ( colour, *interface, sizeof ( EFI_PCI_IO_PROTOCOL ) );
}

Michael

[SebastianRoth]

(2016-02-05 18:08)mcb30 Wrote:  I suspect that the EFI_PCI_IO_PROTOCOL provided by the firmware contains garbage values for RomSize and RomImage. Possible the firmware is failing to zero these fields for a device that does not actually have an expansion ROM BAR.

Reading about ROM BAR here and there I somehow stumbled upon a check on EFI_PCI_IO_ATTRIBUTE_EMBEDDED_ROM that was removed from the kernel only two month after the whole PCI setup code was added in the first place. From the commit message it sounds like there was a good reason to remove this check but I wonder if this would prevent the issue we see. What do you think?

As well I will add the code you suggested to dump the returned EFI_PCI_IO_PROTOCOL structure.

[mcb30]

(2016-02-06 01:16)SebastianRoth Wrote:  Reading about ROM BAR here and there I somehow stumbled upon a check on EFI_PCI_IO_ATTRIBUTE_EMBEDDED_ROM that was removed from the kernel only two month after the whole PCI setup code was added in the first place. From the commit message it sounds like there was a good reason to remove this check but I wonder if this would prevent the issue we see. What do you think?

It's possible that the check may have masked the firmware bug, by causing the kernel to choose not to bother taking a copy of that particular ROM.

I've e-mailed a contact at HP to see if I can get in touch with the people responsible for that particular UEFI firmware build. In the meantime, it would be really helpful if you could get a dump of the contents of the EFI_PCI_IO_PROTOCOL structure.

Michael

[SebastianRoth]

(2016-02-06 09:56)mcb30 Wrote:  I've e-mailed a contact at HP to see if I can get in touch with the people responsible for that particular UEFI firmware build. In the meantime, it would be really helpful if you could get a dump of the contents of the EFI_PCI_IO_PROTOCOL structure.

Thank you so much for all the effort you are taking although this is not caused by iPXE in any way!! I am very grateful for all the help and support on debugging this.

Looking forward to see what we hear back from the people you e-mailed. I really hope that this is not any kind of issue with this particular tablet we are testing on. I asked the device owner to test on other tablets (I think he has more than one of those HP X2 210) as well just to make sure. I also added the code to dump the structure. But we will need to wait for his answer till he's back in the company to be able to test and report back.

[SebastianRoth]

Dumping the EFI_PCI_IO_PROTOCOL structure we see this: https://forums.fogproject.org/uploads/fi...pro_li.jpg

The tester reported that he's done most of the tests with two of his devices seeing the same behavior.

Sebastian

[mcb30]

(2016-02-08 09:51)SebastianRoth Wrote:  Dumping the EFI_PCI_IO_PROTOCOL structure we see this: https://forums.fogproject.org/uploads/fi...pro_li.jpg

The tester reported that he's done most of the tests with two of his devices seeing the same behavior.

Thanks! Let's see what HP/Insyde make of it.

Michael

[schnecki]

Hi,

have you heard anything from HP. My experiences with the HP x2 210 were as follows:

I couldn't get ArchLinux, nor Fedora Live USB flashdrives to boot. However, the alpha/beta release of Ubuntu 16.04 booted without a problem, and ran stable. From there I installed ArchLinux (https://wiki.archlinux.org/index.php/Ins...ng_Linux). Then booting into the new system rans most of the times. However, after some random time it seems like the system freezes. This seems to be an error in the i2c designware platform module of the kernel. See this bug report: https://bugzilla.kernel.org/show_bug.cgi?id=113291

So I guess Ubuntu 16.04 has a different setup of the i2c kernel module which prevents the error seen on other systems. Most likely these problems are related. What do you think?