boot command hangs

[bacaw]

Hi,

I'm running ipxe commit d5c88c0 (Tue Feb 19 2019)

I have the following iPXE script:

Code:

#!ipxe
kernel -n img  http://<webserver>/fedora/28/x86_64/boot/vmlinuz ks=https://<webserver>:8443/kickstart ramdisk_size=8216 pcie_aspm=off ipv6.disable=1 cmdline inst.sshd noverifyssl biosdevname=0 net.ifnames=0 selinux=0
initrd http://<webserver>/fedora/28/x86_64/boot/initrd.img
boot img

I am PXE booting a Solarflare X2522 network card in a Dell R740 server

Code:

d8:00.0 Ethernet controller: Solarflare Communications SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
...
Product Name: Solarflare XtremeScale X2522 (10G) Adapter

When the card is in PCI Slot 1, SF OptionRom -> PXE -> iPXE -> boot, no problems.
When the card is in PCI Slot 8, SF OptionRom -> PXE -> iPXE -> hangs on 'boot' command!

Code:

iPXE> boot
_
<flashing cursor indefinitely>

Does anyone have any DEBUG= flags I could try (tcp doesn't show anything interesting)
Or knows of any iPXE related problems to this?
Or thinks it could be a Dell BIOS/iDRAC problem? (I'm on BIOS 1.6.12, iDRAC 3.21.26.22)

Any advice welcome

Thanks!
Ross

[NiKiZe]

Just as a basic test and minimize issue of kernel you could instead try to chain to http://boot.ipxe.org/demo/

what does ifstat show and what if you enable DEBUG for the nic driver that is used?

We are also missing information on which iPXE you are running, undionly.kpxe, ipxe.efi are examples.
Have you tried any other binary? (for example switching from undionly.kpxe to ipxe.pxe, or the other way around)

[bacaw]

Sorry, extra information:

Binary: undionly.kpxe
ifstat:

Code:

iPXE> ifstat
net0: 00:0f:53:4c:d5:90 using undionly on 0000:d8:00.0 (open)
  [Link:up, TX:10 TXE:1 RX:17 RXE:5]
  [TXE: 1 x "Network unreachable (http://ipxe.org/28086011)"]
  [RXE: 5 x "The socket is not connected (http://ipxe.org/380a6001)"]
iPXE> ping <gateway>
64 bytes from <gateway>: seq=1
..etc

Chaining boot.ipxe.org/demo had the same result (flashing cursor indefinitely, after the '..ok' messages for vmlinuz and initrd loading)

ipxe.pxe just plain failed to load..

Code:

Booting from Solarflare 000F534CD590 (PCI D8:00.0)
Solarflare 000F534CD590 (PCI D8:00.0) starting execution...ok
00-0f-53-4c-d5-90 Features: HTTP DNS TFTP bzImage COMBOOT ELF PXE PXEXT
net2 link up
Waiting a further 5 seconds for net2 post link up... ok
DHCP (net2 00:0f:53:4c:d5:90).... ok
DHCP server: <dhcp server>
route net2: <ip>/255.255.255.192 gw <gw>
Booting from filename "ipxe.pxe"
tftp://<tftp>/ipxe.pxe.. ok
ipxe.pxe : 349834 bytes
PXE->EB: !PXE at 9900:0B20, entry point at 9900:05F6
         UNDI code segment 9900:0BC4, data segment 99BD:255C (612-625kB)
         UNDI device is PCI D8:00.0, type DIX+802.3

From looking at https://ipxe.org/appnote/hardware_drivers, my driver is sfc_hunt, so I've used DEBUG=sfc_hunt:3 while compiling my ipxe images, no extra output seen when booting any legacy bios images though..

Interestingly if I try ipxe.efi (64bit) it works, and my original fedora28 kernel boots at least, my build isn't setup for UEFI so it hangs later on due to other problems. I just had to add initrd=initrd.img to the ipxe script kernel line to get it to boot. I see some coloured sfc_hunt messages when ipxe.efi is attempting to DHCP on all the interfaces.

So, seems its a UEFI vs legacy problem? Still unknown if it's the BIOS at fault or iPXE, or even the Solarflare firmware.

Thanks for your reply!

[bacaw]

I had another solarflare card installed in the server, which when I removed, fixed the problem!

The card I removed also works perfectly fine when installed on its own.

So... 2 cards doesn't work! I'm going to open a ticket with Solarflare to see what they say as well (I notice they have contributed the SFC driver in the source code)

[NiKiZe]

undionly only contains the undionly drivers which uses the initial PXE rom.
so buiding undionly with DEBUG enabled for some other driver will never output anything since that driver isn't included

when you use ipxe.efi or ipxe.pxe that includes the native driver and you get debug output.

I would expect boot to work fine but weird bugs could happen.
Do you have a clean config? if not could you try building with clean?
Also do try to see if ipxe.pxe works differently from undionly.kpxe

[bacaw]

Ah ok, thanks for elaborating.

I've compiled vanilla undionly.kpxe, and ipxe.pxe just with syslog logging all messages, using commit 960d1e,

Code:

make bin/undionly.kpxe
make bin/ipxe.pxe

undionly.kpxe:

Code:

Feb 26 09:59:19  ipxe: iPXE> ifstat
Feb 26 09:59:19  ipxe: net0: 00:0f:53:4c:d5:90 using undionly on 0000:d8:00.0 (open)
Feb 26 09:59:19  ipxe:   [Link:up, TX:4 TXE:1 RX:6 RXE:0]
Feb 26 09:59:19  ipxe:   [TXE: 1 x "Network unreachable (http://ipxe.org/28086011)"]
Feb 26 09:59:32  ipxe: iPXE> dhcp
Feb 26 09:59:35  ipxe: Configuring (net0 00:0f:53:4c:d5:90)...... ok
Feb 26 09:59:45  ipxe: iPXE> chain http://ipxe.build.:8440/ipxe
Feb 26 09:59:45  ipxe: http://ipxe.build.:8440/ipxe...Downloaded "ipxe"
Feb 26 09:59:45  ipxe:  ok
Feb 26 09:59:45  ipxe: Executing "ipxe"
Feb 26 09:59:45  ipxe: http://yum.anycast./fedora/28/x86_64/boot/vmlinuz...Downloaded "vmlinuz"
Feb 26 09:59:45  ipxe:  ok
Feb 26 09:59:46  ipxe: http://yum.anycast./fedora/28/x86_64/boot/initrd.img...Downloaded "initrd.img"
Feb 26 09:59:46  ipxe:  ok
Feb 26 09:59:46  ipxe: Executing "img"

ipxe.pxe (doesn't boot as per before):

Code:

Booting from Solarflare 000F534CD590 (PCI D8:00.0)
Solarflare 000F534CD590 (PCI D8:00.0) starting execution...ok
00-0f-53-4c-d5-90 Features: HTTP DNS TFTP bzImage COMBOOT ELF PXE PXEXT
net2 link up
Waiting a further 5 seconds for net2 post link up... ok
DHCP (net2 00:0f:53:4c:d5:90).... ok
DHCP server: <dhcp server>
route net2: <ip>/255.255.255.192 gw <gw>
Booting from filename "ipxe.pxe"
tftp://<tftp>/ipxe.pxe.. ok
ipxe.pxe : 328339 bytes
PXE->EB: !PXE at 9900:0B20, entry point at 9900:05F6
         UNDI code segment 9900:0BC4, data segment 99BD:255C (612-625kB)
         UNDI device is PCI D8:00.0, type DIX+802.3

[MultimediaMan]

(2019-02-25 16:08)bacaw Wrote:  I had another solarflare card installed in the server, which when I removed, fixed the problem!

The card I removed also works perfectly fine when installed on its own.

So... 2 cards doesn't work! I'm going to open a ticket with Solarflare to see what they say as well (I notice they have contributed the SFC driver in the source code)

In PCBIOS mode, this doesn't surprise me: If the same Firmware shows up twice on the same IRQ, the firmware will often fail. I've had to shuffle cards in servers to get working configuration many, many times. The reason it works in EFI is the EFI environment enumerates devices (very) differently and handles multiples of devices on the same IRQ much more seamlessly.

[bacaw]

Thanks all!
FWIW Solarflare have confirmed they're working on a bug regarding this and X2522 cards.