Connection timeout 4c0a6035

[abhiniveshjain]

I have following kind of setup.

PXE server config

CentOS 7.6
DellEMC R640

PXE client is VM running on top of QEMU/KVM on same server. I trying to do ipxe boot with rhcos (Red Hat Core OS). It is able to reach DHCP server and get the IP and next server details.

After this, it calls matchbox URL of my installer host (i.e. PXE server) by passing uuid and mac address. here is the log that I see on console
-------------------------------------------
Booting from ROM.
iPXE (PCI 00:3.0) starting execution...OK
iPXE initializing devices...OK

iPXE 1.0.0+ (4e85b27) --OPen source network boot firmware --
Features:

net0: mac address using rt18139 on 0000:00.03.0 (open)
Link: up, TX:0, TXE:0, RX:0, RXE:0
Configuring (net0 macaddress) .........ok
net0:10.x.x.214/255.255.255.0
next server: 10.x.x.200
filename: http://10.x.x.200:8080/boot.ipxe
http://10.x.x.200:8080/boot.ipxe..OK
boot.ipxe :104 bytes (scripts)
ipxe?uuid=xxxx&mac=macaddress&domain=&hostname=&serial=...................................Connection timed out (http://ipxe.org/4c0a6035)
Could not boot image: Connection timed out (http://ipxe.org/4c0a6035)

No more network devices
--------------------------------------------

while calling this URL, it fails with connection timeout but when I run chain command manually on iPXE prompt, it works just fine.

iPXE> chain http://10.x.x.200:8080/boot.ipxe

system boots just fine from here....

Please help in fixing this issue as I am struggling on this from several days now.

Thanks,
Abhi

[NiKiZe]

Since it works when running manually, it seems that the suggestions on the error page would not help - but always good to read and double-check anyway.

I would check the servers http logs, and see if something pops up there.
tcpdump or wireshark to see which packets go thru and not.

... Oh wait... checking that version ... https://git.ipxe.org/ipxe.git/commit/4e85b27
Before anything else, Please do as it says on the error page! Try using the latest version of iPXE. Your problem may have already been fixed.

[abhiniveshjain]

Before running the iPXE boot, I have downloaded below mentioned files and placed those in /var/lib/tftpboot.

http://boot.ipxe.org/ipxe.efi
http://boot.ipxe.org/undionly.kpxe

I believe above link will give me latest files so not sure how it is showing 2 yrs old version? Am I missing something?

[root@localhost tftpboot]# ls -ltr
total 1028
-rw-r--r--. 1 root root 67629 Sep 17 02:44 undionly.kpxe
-rw-r--r--. 1 root root 981728 Sep 17 02:50 ipxe.efi

My DNSMASQ file is similar to one mentioned in below link.
https://coreos.com/matchbox/docs/latest/...setup.html (please refer to example /etc/dnsmasq.conf)

[NiKiZe]

(2019-09-17 08:13)abhiniveshjain Wrote:  iPXE 1.0.0+ (4e85b27) --OPen source network boot firmware --

What do you get there now? - so that you can check that you are running latest version?

[abhiniveshjain]

Since I am running it inside QEMU, it is picking up ipxe from below location.

[root@localhost cluster]# rpm -qa|grep ipxe
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch

I tried upgrading it but it says you are already on latest version.

[root@localhost cluster]# yum upgrade ipxe-roms-qemu
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.hbcse.tifr.res.in
* epel: repos.del.extreme-ix.org
* extras: centos.hbcse.tifr.res.in
* updates: centos.hbcse.tifr.res.in
No packages marked for update

[root@localhost cluster]# yum install ipxe-roms-qemu
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.hbcse.tifr.res.in
* epel: sg.fedora.ipserverone.com
* extras: centos.hbcse.tifr.res.in
* updates: centos.hbcse.tifr.res.in
Package ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch already installed and latest version
Nothing to do

Please let me know how to upgrade ipxe and why iPXE is not picked up from tftpboot?

[NiKiZe]

qemu comes with an iPXE rom.
you can chain to a newer version of iPXE, but your issue is that your distro don't update packages.
I will try to be clearer - do build a new ipxe ROM for qemu from the latest sources.
When you start qemu, use the options to provide the path to your new ipxe rom - which is an option that you can provide on the cmd line, or as option in libvirt if that is what you are using.

You should also file a bug report, and tell your distro to upgrade to recent iPXE - and make sure it works as you described in your usecase (it's probably not tested much with 2 nics)

[abhiniveshjain]

(2019-09-19 04:40)NiKiZe Wrote:  qemu comes with an iPXE rom.
you can chain to a newer version of iPXE, but your issue is that your distro don't update packages.
I will try to be clearer - do build a new ipxe ROM for qemu from the latest sources.
When you start qemu, use the options to provide the path to your new ipxe rom - which is an option that you can provide on the cmd line, or as option in libvirt if that is what you are using.

You should also file a bug report, and tell your distro to upgrade to recent iPXE - and make sure it works as you described in your usecase (it's probably not tested much with 2 nics)

Thanks for your response. I can see several ROMs inside qemu-kvm so the question is which one I need to build?

[root@localhost qemu-kvm]# pwd
/usr/share/qemu-kvm

[root@localhost qemu-kvm]# ls -ltr pxe*
lrwxrwxrwx. 1 root root 20 Sep 18 21:45 pxe-pcnet.rom -> ../ipxe/10222000.rom
lrwxrwxrwx. 1 root root 20 Sep 18 21:45 pxe-ne2k_pci.rom -> ../ipxe/10ec8029.rom
lrwxrwxrwx. 1 root root 20 Sep 18 21:45 pxe-e1000.rom -> ../ipxe/8086100e.rom
lrwxrwxrwx. 1 root root 20 Sep 18 21:45 pxe-virtio.rom -> ../ipxe/1af41000.rom
lrwxrwxrwx. 1 root root 20 Sep 18 21:45 pxe-rtl8139.rom -> ../ipxe/10ec8139.rom

[root@localhost ipxe]# ls -ltr
total 1536
-rw-r--r--. 1 root root 262144 Aug 9 05:41 10222000.rom
-rw-r--r--. 1 root root 262144 Aug 9 05:41 10ec8139.rom
-rw-r--r--. 1 root root 262144 Aug 9 05:41 10ec8029.rom
-rw-r--r--. 1 root root 262144 Aug 9 05:41 8086100e.rom
-rw-r--r--. 1 root root 262144 Aug 9 05:41 1af41000.rom
-rw-r--r--. 1 root root 262144 Aug 9 05:41 808610d3.rom
[root@localhost ipxe]# pwd
/usr/share/ipxe

Since each of these ROMs are used for different interface model type, I tried to find what is being used in my case? my VM is getting created using virsh create input.xml command and in this input.xml file, following setting is used.
Please let me know if this setting is OK or so should it be replaced with virtio?

<model type='rtl8139'/>

Above setting means, I have to build pxe-rtl8139.rom and replace it at /usr/share/qemu-kvm location by removing the symbolic link and placing the file. However this will be overwritten if new ipxe-qemu-kvm package is applied so better option would be to give rom file location in domain xml file that is used by "virsh create" command.

<rom bar='on' file='/tmp/pxe-rtl8139.rom'/>

For building rom file with latest ipxe, I am going to use below steps, please confirm if this is correct.

cd /tmp
git clone git://git.ipxe.org/ipxe.git
cd ipxe/src
make bin/pxe-rtl8139.rom

[abhiniveshjain]

make bin/pxe-rtl8139.rom doesn't seem to be correct. Please let me know the right value for it?

I referred to http://ipxe.org/appnote/buildtargets and http://ipxe.org/appnote/hardware_drivers but could not find out the driver name for

Broadcom 5720 Quad Port 1GbE BASE-T, RNDC card that are present on my Dell EMC R640. Closest one is "NetXtreme BCM5720 2-port Gigabit Ethernet PCIe". Should I use this?

14e4-165f as driver id

bin-x86_64-pcbios as platform and run below command.

make bin-x86_64-pcbios/14e4-165f.rom

later I will create a symbolic link of this file with pxe-rtl8139.rom

[NiKiZe]

I would not replace any of them.
You should build a rom for your nic, and then refer to that.
Assuming you are using legacy pcbios and not efi ...
If you are using rtl8139, then build with:
make bin/10ec8139.rom

Why are you trying to build for the broadcom nic?
guessing here, correct me if I'm wrong... you want this on real hw for chainbooting?
For real hardware, use eighter undionly.kpxe or ipxe.pxe (assuming pcbios again here)
The reason for referring to the alternate rom is only to have your 2 nics to work inside of qemu - not relevant to other hardware.

With that said - to detect iPXE roms and if it needs to chain into a "fuller" iPXE you can use something like: https://gist.github.com/robinsmidsrod/4008017

[abhiniveshjain]

I have built the new rom using following command and used that. Now iPXE is showing iPXE 1.0.0+ 41a9a but end result is same timeout error. Manual chain loading is working as before. How can I see more details of this timeout by doing some sort of debug ON and how can I increase this time out value to see if that works?

I have one Baremetal machine to be booted using iPXE hence I asked for driver details. But as you said, there we will be using UNDIONLY.kpxe (in bios mode)
=======

Booting from ROM.
iPXE (PCI 00:3.0) starting execution...OK
iPXE initializing devices...OK

iPXE 1.0.0+ (41a9a) --OPen source network boot firmware --
Features:

net0: mac address using rt18139 on 0000:00.03.0 (open)
Link: up, TX:0, TXE:0, RX:0, RXE:0
Configuring (net0 macaddress) .........ok
net0:10.x.x.214/255.255.255.0
next server: 10.x.x.200
filename: http://10.x.x.200:8080/boot.ipxe
http://10.x.x.200:8080/boot.ipxe..ok
boot.ipxe :104 bytes (scripts)
ipxe?uuid=xxxx&mac=macaddress&domain=&hostname=&serial=...................................Connection timed out (http://ipxe.org/4c0a6035)
Could not boot image: Connection timed out (http://ipxe.org/4c0a6035)

No more network devices
--------------

dnsmasq logs are showing below set of lines repeated.

Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 available DHCP range: 10.x.x.211 -- 10.x.x.230
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 vendor class: PXEClient:Arch:00000:UNDI:002001
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 user class: iPXE
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 DHCPDISCOVER(provisioning) 52:54:00:82:68:3f
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 tags: ipxe, bios, provisioning
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 DHCPOFFER(provisioning) 10.209.174.214 52:54:00:82:68:3f
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 requested options: 1:netmask, 3:router, 6:dns-server, 7:log-server,
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 requested options: 12:hostname, 15:domain-name, 17:root-path,
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 requested options: 26:mtu, 43:vendor-encap, 60:vendor-class,
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 requested options: 66:tftp-server, 67:bootfile-name, 119:domain-search,
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 requested options: 128, 129, 130, 131, 132, 133, 134, 135, 175,
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 requested options: 203
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 next server: 10.x.x.200
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 sent size: 1 option: 53 message-type 2
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 sent size: 4 option: 54 server-identifier 10.xx.xx.200
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 sent size: 4 option: 51 lease-time 30m
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 sent size: 37 option: 67 bootfile-name http://10.xx.xx.200:8080/boot.ipxe
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 sent size: 4 option: 58 T1 15m
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 sent size: 4 option: 59 T2 26m15s
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 sent size: 4 option: 1 netmask 255.x.x.0
Sep 20 10:22:48 dnsmasq-dhcp[1]: 631002211 sent size: 4 option: 28 broadcast 10.xx.xx.255