iPXE slow to download vmlinuz and initrd

[leonard]

Hello,


I have a bios based 32-bit desktop from the previous decade, with a Fast Ethernet PXE NIC embedded in the mobo.
The desktop is multiboot with GRUB2 entries for my local OSes which I need from time to time. But it is primarily an LTSP thin client (the LTSP server is running on Ubuntu 16.04.3).
PXE booting via the embedded Fast Ethernet NIC was working well and rather fast, but once logged in, the desktop was a bit slow for certain applications (youtube videos would often freeze, etc.).
So I installed a 10 times faster Gigabit PCI NIC (TP-Link TG-3269, based on Realtek RTL8169SC), which doesn't have a PXE ROM.
To compensate for the lack of PXE ROM, I added a GRUB2 entry pointing to ipxe.lkrn, which I downloaded from ipxe.org.
This "works".


The good thing is: once I am done booting with iPXE, the old desktop behaves like a fast 2018 desktop (thanks to the LTSP server being powerful enough and to the Gigabit NIC of the desktop). This is really impressive.


Problem:

To boot the desktop, despite the Gigabit NIC, iPXE takes ~2 minutes to load vmlinuz (~6Mb) and ~8 minutes to load initrd (~36Mb).
The total boot time is over 10min, which is unacceptable in practice.
With the Gigabit NIC, iPXE could arguably take around 0.34s to download vmlinuz and initrd (i.e. almost 2000 times faster).

This is a real pain and this is also very surprising since PXE booting with the old Fast Ethernet NIC is an order of magnitude faster than iPXE booting with the Gigabit NIC.


What I can do?


My investigations:

iPXE prints (while booting) that it is using "RTL8169" (not "RTL8169SC"). Not sure how relevant this is.

I noticed, in the source code of iPXE, that the function:
static int realtek_phy_reset ( struct realtek_nic * )
which is implemented in src/drivers/net/realtek.c, comprises the following comment:
/* Some cards (e.g. RTL8169SC) do not advertise Gigabit by
* default. Try to enable advertisement of Gigabit speeds.
*/
However, since the issue seems to be addressed in iPXE (based on my understanding of the code), I don't know what more to do...
And I assume this would not explain the factor 2000 issue I face: I can't imagine iPXE configuring the NIC at a speed below 10Mbit/s (which would take about 34s, not 10min, to download vmlinuz and initrd).

I read that multi-NIC clients may pose problems to iPXE. But I already deactivated the Fast Ethernet NIC at bios level, it is no longer visible (the system appears to only have the Gigabit NIC) and this does not improve the situation.

I read on forums about real-mode issues in iPXE (which would employ old and buggy memory management techniques?) but this seemed to be in a different context and I am not sure how relevant/correct this is.

If I could easily preload vmlinuz and initrd locally from the HDD of the desktop without compromizing (i.e. without having to completely reconfigure) the rest of the iPXE boot, this could be an option.
I thought of a possible dirty hack (that I would need to investigate further) into:
int imgdownload_string ( const char *uri_string, unsigned long timeout, struct image **image )
so that when I recognize that the uri_string parameter points to my vmlinuz or to my initrd, instead of calling imgdownload I create an image structure and copy the relevant data into it from local copies.
Not sure how easy it is to implement in view of analyzing the image structure, understanding how to fill it so that it looks (to the rest of the program) like it was just downloaded, etc.

But I assume there must be another way.

At that stage I cannot afford to spend days on this while I may be doomed to fail (not in the near future at least), hence my present request for help.

The options I currently exclude:
1 - dropping iPXE and using a PXE gigabit NIC (I don't have one and I now think I need iPXE for other reasons anyway).
2 - dropping any form of PXE and booting the LTSP client locally by copying the LTSP ISO on the old desktop (despite this being advertized as a trivial possibility, it appears to be very tricky to configure, at least for me, and I gave up after a few days of unsuccessful attempts).


Thanks!

[NiKiZe]

Check the link speed at the switch port when iPXE is booted.
but even with 10Mbit/s you should have around 1MiB/s which means your kernel should be loaded in under 10 sec.
It might be that interupts is not available on that machine, and that would explain the slow speed if iPXE have to poll the nic for each packet
Realtek have quite a few buggy chipsets, try to grab some other nic instead, such as an intel gigabit nic instead.

I would also be interested to know what the embedded nic is, and what speeds you got with that, one solution might be to still booting over the fast ethernet nic, and then disable that and only use the gigabit nic when Linux boots.

If you want to have kernel and initrd on disk instead, just use grub, copy over the cmdline, no need for iPXE

[leonard]

Hi, I found a partial workaround.
My problem is a software issue (or at least can be addressed by software means), because I have a partial workaround with the same hardware.

1 - answers to NiKiZe questions
2 - a comment on a workaround I was initially contemplating
3 - the partial workaround that "worked" for me

1-
I connected the LTSP client straight to the server, without any switch in between: the speed is completely unacceptable.
Normally it takes 10+ minutes to boot but it once took between 30 minutes and an hour (this was unusual, I have no idea why it was so different this time).
The embedded NIC (Fast Ethernet only) is detected by iPXE as: "dlink-530tx".
When I iPXE boot from the embedded NIC, the boot speed is normal, and vmlinuz + initrd are downloaded in just a few seconds.
So one could suspect the Gigabit NIC. But it seems to depend on how you interface with the NIC.

2-
I was initially thinking that it was iPXE that had to download vmlinuz and initrd, but this does not seem to be accurate, at least this is not done via the imgdownload function.
I added a printf of the URI each time imgdownload is called, and imgdownload just downloads pxelinux.0, not vmlinuz and initrd.
I guess that it is pxelinux.0 that somehow triggers the download of vmlinuz and initrd.
So in fact I don't even seem to have an easy way to locally load vmlinuz and initrd (I would need to dig further but don't need this anymore - see below).

3-
I noticed that in my LTSP server (version 5.5.7-1) I have both pxelinux.0 and gpxelinux.0.
What I did was simply to rename pxelinux.0 to pxelinux.0.bak and gpxelinux.0 to pxelinux.0.
Now the "pxelinux.0" that gets downloaded by iPXE is in fact gpxelinux.0.
It works like a charm.
iPXE is still terribly slow (typically well below 100 kbit/s, i.e. well below 0.01% of the NIC's capacity).
But since gpxelinux.0 is a very small file, it only takes about a minute to get it (sometimes less, but this is not predictable) and is more or less "OK" (although it's a pain when it gets close to or exceeds the minute, you can live with it).
Then the gpxelinux.0 magic leads to downloading vmlinuz and initrd at a "normal" speed (although rather Fast Ethernet than Gigabit/s speed), i.e. a few seconds instead of more than 10 minutes (sometimes a lot more).
Once booted, my LTSP client enjoys a true Gigabit link - it can view and navigate through youtube videos without any freezing (which is not the case with the embedded Fast Ethernet NIC), etc.