Very slow kernel/initrd download (RPI + U-boot)

[ljfranklin]

Setup:
- Raspberry Pi 3 B+
- RPI Bootloader -> U-boot 64-bit -> iPXE 64-bit snp.efi

So far I have the Raspberry Pi successfully booting into u-boot (a popular ARM bootloader), which then chainloads iPXE snp.efi. U-boot steps shown here, although I'm trying to use HTTP rather than ISCSI. iPXE with snp.efi target successfully detects the network devices, gets an IP from DHCP, and starts the kernel download.

The issue is the kernel download is incredibly slow, taking 10+ minutes to download a 6MB kernel file. The Raspberry Pi only supports a 100-Mbps max speed, but this is down at 50-kbps or so. Appreciate any suggestions on how to debug this.

Other stuff I've tried:
- 32-bit U-boot + 32-bit iPXE has the same issue
- Tried building the recently added `rpi.efi` target using command shown here, but this build failed to detect the network devices

[NiKiZe]

make sure that you build an image that includes the nic driver instead of relying on nii/snp interface.
But other than that, a tcpdump is probably in order to see where the issue is, if it is re-transmits or similar.

[ljfranklin]

(2019-09-05 00:35)NiKiZe Wrote:  make sure that you build an image that includes the nic driver instead of relying on nii/snp interface.

Sorry for the delay. I tried building with `bin-arm32-efi/smsc95xx--lan78xx--snp.efi` which I thought included the two possible Raspberry Pi NIC drivers. But this showed the same slow download behavior as `snp.efi`. Building without `snp` (`smsc95xx--lan78xx.efi`) fails to find any network devices. Looks like Michael added a shortcut build rule for Raspberry Pis to the Makefile here, not sure how he was able to get it working.

(2019-09-05 00:35)NiKiZe Wrote:  But other than that, a tcpdump is probably in order to see where the issue is, if it is re-transmits or similar.

Good call, tcpdump file is here. I started the capture once iPXE booted and killed it after ~10 seconds. The kernel download had only reached ~2% during that time. After ~5 seconds the capture shows a large block of retransmits. The network is healthy and the Raspberry Pi when running u-boot is able to download files using TFTP at expected speeds. Any suggestions for debugging this further?

Thanks for the help!

[NiKiZe]

you need to include usb support as well, ecm--ncm I think it is, it always fallbacks to snp if that is included and no other matching driver is available.

You might want to check switch config, and verify that correct speed, and duplex are negotiated. also just for completeness, provide full usbid for the nic, just to be sure which it is there.
current git master also has some fixes mentioning the PI that is just a few days old.

[ljfranklin]

(2019-09-18 13:46)NiKiZe Wrote:  you need to include usb support as well, ecm--ncm I think it is, it always fallbacks to snp if that is included and no other matching driver is available.

Unfortunately building `smsc95xx--lan78xx--ecm--ncm.efi` also fails to find any network devices. Any other drivers I might need?

(2019-09-18 13:46)NiKiZe Wrote:  You might want to check switch config, and verify that correct speed, and duplex are negotiated.

Didn't get a chance to check that yet. Although I did see that with SNP, running iPXE's ifstat command after the download shows ""Error: No buffer space available (Error code 2a0860)" hundreds of times.

(2019-09-18 13:46)NiKiZe Wrote:  also just for completeness, provide full usbid for the nic, just to be sure which it is there.

Temporarily added Linux to the SD card to run `lsusb`. It shows: "ID 0424:7800 Standard Microsystems Corp." and dmesg has some lines that mention the lan78xx driver.

(2019-09-18 13:46)NiKiZe Wrote:  current git master also has some fixes mentioning the PI that is just a few days old.

I pulled the latest commits from master, same result.

[NiKiZe]

(2019-09-19 23:51)ljfranklin Wrote:  Temporarily added Linux to the SD card to run `lsusb`. It shows: "ID 0424:7800 Standard Microsystems Corp." and dmesg has some lines that mention the lan78xx driver.

Only the ID is interesting to verify that there is a driver, and it is: https://git.ipxe.org/ipxe.git/blob/41a9a...8xx.c#l384

Ah ... there is a named config in iPXE for the Pi: https://git.ipxe.org/ipxe.git/tree/HEAD:/src/config/rpi

Build using CONFIG=rpi as described in https://ipxe.org/appnote/named_config
That should make sure that the correct usb drivers is (not) available.

And with that I think that you will only want lan78xx.efi (sorry for all the other noise in the thread)

[ljfranklin]

(2019-09-20 03:51)NiKiZe Wrote:  Ah ... there is a named config in iPXE for the Pi: https://git.ipxe.org/ipxe.git/tree/HEAD:/src/config/rpi

Build using CONFIG=rpi as described in https://ipxe.org/appnote/named_config
That should make sure that the correct usb drivers is (not) available.

And with that I think that you will only want lan78xx.efi (sorry for all the other noise in the thread)

The named config does look promising, but I still can't get it to work. Building with `CONFIG=rpi lan78xx.efi` or `CONFIG=rpi lan78xx--ecm--ncm.efi` or CONFIG=rpi rpi.efi` all fail to detect any network devices.

Although looking at the `rpi` configs, I do see that it uses EFI_USB_IO_PROTOCOL which has this concerning warning about super slow downloads. So maybe the slow downloads are a current limitation of iPXE on the Raspberry Pi?

I'll try to ping Michael as he's been pushing commits related to Raspberry Pi drivers to see what his setup looks like.

[mcb30]

(2019-09-21 13:58)ljfranklin Wrote:  I'll try to ping Michael as he's been pushing commits related to Raspberry Pi drivers to see what his setup looks like.

I've been using the EDK2 UEFI firmware (https://github.com/tianocore/edk2-platfo...rryPi/RPi3), rather than U-Boot. It's still using the intrinsically slow EFI_USB_IO_PROTOCOL, but the download speed is acceptable: on the 3B (100Mbps NIC) it comfortably sustains 33Mbps using the iPXE smsc95xx driver.

There is a bug on the 3B+ that prevents the lan78xx driver from working properly; I'm hoping to fix this in the next week since I need it myself.

Michael

[mcb30]

(2019-09-27 09:55)mcb30 Wrote:  There is a bug on the 3B+ that prevents the lan78xx driver from working properly; I'm hoping to fix this in the next week since I need it myself.

This is now fixed: http://git.ipxe.org/ipxe.git/commitdiff/3fe683eba.

Michael

[ljfranklin]

I'm looking into the EDK2 firmware now. I was able to boot into EDK2, but it looks like the firmware expects the iPXE .efi file to be present on the SD card. I was hoping to have it netboot end-to-end starting with an empty SD card.

Michael, are you writing the iPXE files to an SD card first or do you have some netboot setup? I was searching for a way to embed iPXE as an EFI application into the EDK2 firmware or similar but it wasn't obvious whether this was supported. Thanks for the pointers!

[mcb30]

(2019-10-02 19:39)ljfranklin Wrote:  I'm looking into the EDK2 firmware now. I was able to boot into EDK2, but it looks like the firmware expects the iPXE .efi file to be present on the SD card. I was hoping to have it netboot end-to-end starting with an empty SD card.

Michael, are you writing the iPXE files to an SD card first or do you have some netboot setup? I was searching for a way to embed iPXE as an EFI application into the EDK2 firmware or similar but it wasn't obvious whether this was supported. Thanks for the pointers!

I am writing bin-arm64-efi/rpi.efi to the SD card as /EFI/BOOT/BOOTAA64.EFI. It then gets booted automatically by the UEFI firmware. You could potentially use the EDK2 network boot stack instead to load iPXE via the network, but there's no real point in doing so since you have to write the UEFI firmware to the SD card anyway. (As another option: the Pi's built-in network boot mechanism using the VC4 firmware would allow you to boot without any SD card, but it is too unreliable for production use.)

I will soon get around to automating and publishing the build for this, so there will be a ready-built download available.

Michael

[ljfranklin]

Initially I was using the Pi's built-in firmware to netboot into EDK2. But putting EDK2+iPXE directly on disk makes sense, less flaky and lets you add certs for security.

I've now got iPXE almost booting. It correctly downloads the kernel and initrd at an acceptable speed after pulling your latest commit to the lan78xx driver (thanks!). But iPXE hangs at "EFI stub: Exiting boot services and installing virtual address map...". I've tried messing with the kernel args, adding "net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1" didn't help. It also could be that I'm using a USB drive rather than an SD card. Any suggestions on debugging further?

[mcb30]

(2019-10-04 12:20)ljfranklin Wrote:  Initially I was using the Pi's built-in firmware to netboot into EDK2. But putting EDK2+iPXE directly on disk makes sense, less flaky and lets you add certs for security.

I've now got iPXE almost booting. It correctly downloads the kernel and initrd at an acceptable speed after pulling your latest commit to the lan78xx driver (thanks!). But iPXE hangs at "EFI stub: Exiting boot services and installing virtual address map...". I've tried messing with the kernel args, adding "net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1" didn't help. It also could be that I'm using a USB drive rather than an SD card. Any suggestions on debugging further?

I hit exactly that problem myself, along with several others (all now fixed).

Check that you are using the latest stable RPi firmware (bootcode.bin), latest edk2 master, and latest edk-platforms master commits. There was one bug relating to an incorrect internal device tree containing a spurious "interrupt-parent" property, which prevented the serial port from working. There is also a very recent kernel fix (https://github.com/raspberrypi/linux/com...6a403990ec) required to make the framebuffer work; this may not yet be present in the prebuilt kernel packages.

Michael

[mcb30]

(2019-10-02 21:18)mcb30 Wrote:  I will soon get around to automating and publishing the build for this, so there will be a ready-built download available.

This is now online at https://github.com/ipxe/pipxe.

Michael