Threaded Mode | Linear Mode

cjs · 2017-02-03, 09:58

On a CentOS 7 system I'm using libvirt with the --pxe option to start a KVM/QEMU VM that will do a network boot. The iPXE banner indicates that it's version 1.0.0+ (6366fa7a).

Libvirt sets up a DHCP server using dnsmasq in a way that makes it rather painful to change its configuration, so I reckoned I'd just set up a second DHCP server (again using dnsmasq) as a "proxy" server listening on port 4011 to provide the boot information that the libvirt DHCP server can't provide, as well as doing the TFTP stuff.

I've got most everything working fine; if I manually interrupt the boot with Ctrl-B and type dhcp followed by boot pxelinux.0 I get into the boot menu I've configured and can load and run stuff.

However, if I let the boot go on in automatic mode it doesn't know to try to load `pxelinux.0` and says "Nothing to boot: No such file or directory" and gives up. I had thought it should try to send a DHCP request to port 4011 to get further boot information, but looking at what's going on with tcpdump this doesn't happen; the only requests I see are DHCP to port 67.

Is there some way, without modifying the configuration of the DHCP server running on port 67, that I can get iPXE to boot without human intervention?

***NiKiZe*** · 2017-02-03, 23:50

Proxy dhcp can be disabled by the main dhcp server via a option.
Example of this from https://gist.github.com/robinsmidsrod/4008017

Code:

# Disable ProxyDHCP, we're in control of the primary DHCP server

option ipxe.no-pxedhcp 1;

I know this is not dnsmasq, but the same can be done there as well.

But first of: https://git.ipxe.org/ipxe.git/commit/6366fa7a that version is over a year old, you should try and do dhcp && chain http://boot.ipxe.org/ipxe.pxe from the qemu provided ipxe since there have been lots of improvements (and we do not like to try and debug old versions)

regarding proxy dhcp there is a note at http://ipxe.org/cmd/ifconf#notes about something that is very similar to your issue

Instead of chaining into pxelinux, you might want to concider using the more advanced features that the ipxe menu system can provide via ipxe scripting.

cjs · 2017-02-06, 05:52

First, thanks for your help with this issue!

(2017-02-03 23:50)NiKiZe Wrote: Proxy dhcp can be disabled by the main dhcp server via a option.

Ah. I've looked into this a bit (the Gist you mentioned was very helpful) and, if I'm understanding this right, the primary DHCP server can send back a reply with DHCP option 175 set to say "don't attempt to contact a proxy server," right? (I'm not sure what the ipxe-green.conf file is in that Gist, but it looks to be a configuration file for some sort of DHCP server?)

That doesn't seem to be happening in my case. Here's a summary of all IPv4 UDP messages during the boot process:

Code:

52:05.223313 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:4d:12:82, length 396

52:08.227723 IP 192.168.122.1.67 > 192.168.122.141.68: BOOTP/DHCP, Reply, length 300

52:13.132583 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:4d:12:82, length 408

52:13.132728 IP 192.168.122.1.67 > 192.168.122.141.68: BOOTP/DHCP, Reply, length 300

Here's the decoded detail of of the DHCP reply:

Code:

12:52:08.227723 IP (tos 0xc0, ttl 64, id 30016, offset 0, flags [none], proto UDP (17), length 328)

    192.168.122.1.67 > 192.168.122.141.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0xc841183d, secs 8, Flags [none] (0x0000)

      Your-IP 192.168.122.141

      Server-IP 192.168.122.1

      Client-Ethernet-Address 52:54:00:4d:12:82

      Vendor-rfc1048 Extensions

        Magic Cookie 0x63825363

        DHCP-Message Option 53, length 1: Offer

        Server-ID Option 54, length 4: 192.168.122.1

        Lease-Time Option 51, length 4: 3600

        RN Option 58, length 4: 1800

        RB Option 59, length 4: 3150

        Subnet-Mask Option 1, length 4: 255.255.255.0

        BR Option 28, length 4: 192.168.122.255

        Default-Gateway Option 3, length 4: 192.168.122.1

        Domain-Name-Server Option 6, length 4: 192.168.122.1

So no option appears to be telling iPXE not to try sending a DHCP request to port 4011.

Quote:regarding proxy dhcp there is a note at http://ipxe.org/cmd/ifconf#notes about something that is very similar to your issue

Yes, I had already looked at this page. I suspect the advice is a bit old; it looks as if the 1.x versions of iPXE already build in the appropriate delay for a proxy server to reply. You'll note in the dump above that there's a second DHCP request made five seconds after the first DHCP reply is received; there's also a several-second delay after that before iPXE returns to the command line. I'm guessing that this would be iPXE waiting for a separate DHCP response from a proxy DHCP server. Unfortunately, since it never sent out a response on port 4011, no response is generated.

(One solution I'd tried to check this out, BTW, was running the proxy DHCP server on port 67 on a different IP address, but unfortunately the primary DHCP server is configured to listen on port 67 on all addresses on the host, and so it grabs port 67 on the new address the instant I add that address to the system.)

Quote:But first of: https://git.ipxe.org/ipxe.git/commit/6366fa7a that version is over a year old, you should try and do dhcp && chain http://boot.ipxe.org/ipxe.pxe m the qemu provided ipxe since there have been lots of improvements (and we do not like to try and debug old versions)

Unfortunately I can't do that automatically unless I can get the darn thing to send a query to my proxy DHCP server, since changing the configuration of the DHCP server running on port 67 isn't practical in my situation. (In which case my problem would be solved, anyway, since the rest of the boot process works fine once I can get a filename to iPXE.)

So at this point it looks as if I'm stuck.

I did, however chainload the above link, which gave me a build from commit 4a4da (which is HEAD of the master branch from the Git repo) and examined its boot behaviour; it has the same problem as the 6366fa7a that came with QEMU: it never sends a request to port 4011.

I'm wondering, do I not properly understand the behaviour that's supposed to be happening here? Is a PXE client not supposed to be sending DHCP requests to port 4011 as well as port 63, by default?

***NiKiZe*** · 2017-02-06, 08:10

(2017-02-06 05:52)cjs Wrote:
Quote:But first of: https://git.ipxe.org/ipxe.git/commit/6366fa7a that version is over a year old, you should try and do dhcp && chain http://boot.ipxe.org/ipxe.pxe m the qemu provided ipxe since there have been lots of improvements (and we do not like to try and debug old versions)

Unfortunately I can't do that automatically unless I can get the darn thing to send a query to my proxy DHCP server, since changing the configuration of the DHCP server running on port 67 isn't practical in my situation. (In which case my problem would be solved, anyway, since the rest of the boot process works fine once I can get a filename to iPXE.)

So at this point it looks as if I'm stuck.

I did, however chainload the above link, which gave me a build from commit 4a4da (which is HEAD of the master branch from the Git repo) and examined its boot behaviour; it has the same problem as the 6366fa7a that came with QEMU: it never sends a request to port 4011.

I'm wondering, do I not properly understand the behaviour that's supposed to be happening here? Is a PXE client not supposed to be sending DHCP requests to port 4011 as well as port 63, by default?

Pressing ctrl-b so you can run the above commands is what you tested right? (note we are talking about tests to identify the issue, a fix will have to come after that)

I'm a bit fuzzy on ProxyDHCP ... maybe this post will be helpfull:
http://forum.ipxe.org/showthread.php?tid=6865 I'm thinking that proxyDHCP requests are only done when there next-server or filename is missing in the orignal reply, which it does seem to be in your case but might be worth double checking both the DHCP Offer and the DHCP Reply.

***robinsmidsrod*** · 2017-02-06, 09:55

cjs: I wonder if the iPXE version you use to start KVM/QEMU with has had ProxyDHCP disabled with a compilation config switch. Either that or the dnsmasq server running on port 67 is somehow telling iPXE not to send a request to port 4011. Have you tried booting ipxe from e.g. the floppy image (ipxe.dsk) instead of via --pxe, as this could change some behavior. It is also possible that you have a network issue not allowing packets properly out of the VM.

Could you please elaborate on how your VM interacts with the rest of your network, especially routers, firewalls and topology?

BitBub · (This post was last modified: 2017-02-07 04:25 by BitBub.)

Not sure it matters here, but as far as I know, dnsmasq does not use port 4011 for anything when responding to legacy bios clients. Everything is done using Option 43 on port 67/68. The port 4011 stuff was added for EFI clients. http://lists.thekelleys.org.uk/pipermail...10551.html

What does your dnsmasq config look like?

cjs · (This post was last modified: 2017-02-08 04:41 by cjs.)

(2017-02-06 09:55)robinsmidsrod Wrote: cjs: I wonder if the iPXE version you use to start KVM/QEMU with has had ProxyDHCP disabled with a compilation config switch. Either that or the dnsmasq server running on port 67 is somehow telling iPXE not to send a request to port 4011.

I think I covered all this in my posts above; I chainloaded the most recent version from http://boot.ipxe.org/ipxe.pxe and got the same behaviour, so this "don't query port 4011" thing is compiled into that one, too, if that's what's doing it. And, as per my post of the DHCP reply from port 67, I don't see any options that would indicate to iPXE not to query the proxy DHCP server.

Quote:Have you tried booting ipxe from e.g. the floppy image (ipxe.dsk) instead of via --pxe, as this could change some behavior.

I've not, but I can go set that up if you think it's really worthwhile. Is that likely to be different from the behaviour when doing the chain load?

Quote:It is also possible that you have a network issue not allowing packets properly out of the VM.
Could you please elaborate on how your VM interacts with the rest of your network, especially routers, firewalls and topology?

There are no routers involved during the DHCP or TFTP portions of the boot. I am using the standard libvirt configuration:
* The host has an (internal) bridge, virbr0, which is the layer 2 network being used.
* The host uses interface virbr0-nic on this bridge and is configured for address 192.168.122.1. libvirt's DHCP server is listening on *:67, which listens on this interface as well as all others.
* The guest is also set up with an interface on this bridge, which is what it's using to send the DHCP requests, etc.
* I run tcpdump on the host on this interface to see what's going on.

I don't have any special firewall tools installed (e.g., CentOS firewalld), and the iptables rules in place on the INPUT chain allow everything through. When I monitor everything on this network I see all sorts of traffic, including DHCP, TFTP and HTTP from the VM, all of which works fine from the VM's point of view. (NAT on the host is forwarding requests from the VM to off-network hosts, e.g., HTTP to chain load from ipxe.org and pull down packages from a CentOS mirror, and that's all working.) The only place I can see where a firewall rule would be blocking port 4011 such that I wouldn't see it on the bridge (using tcpdump as above) would be on the VM, but that would have to be done by QEMU/KVM itself since the DHCP I'm looking at is happening before a Linux kernel is even loaded, and QEMU/KVM doesn't do that.

Has anybody else here actually seen iPXE send DHCP requests to port 4011? That really does appear to be the problem here, and I'm thinking the best thing to do might be to walk through the code and figure out how and where it's supposed to be doing this and doing enough tweaking on a test version of iPXE that we can see if this code path is actually being followed, if indeed it even exists at all.

The relevant code appears to be #define PXE_PORT 4011 (src/include/ipxe/dhcp.h rev 3f814f08e from Michael Brown <mcb30@etherboot.org> 2009-01-23) and the dhcp_proxy_tx function (src/net/udp/dhcp.c, various commits from Michael Brown). A quick look says that this may have something to do with iPXE's pxebs command? Or the end of the dhcp_request_rx seems to have some code that sends a proxy request under certain circumstances, but I've not looked at it too closely to find out what's going on there with that.

Michael Brown still seems to be active (he's had several commits in the last few weeks); is he around on these forums? Maybe I should contact him directly by e-mail to see what's up with all this?

(2017-02-07 03:13)BitBub Wrote: Not sure it matters here, but as far as I know, dnsmasq does not use port 4011 for anything when responding to legacy bios clients.

dnsmasq listens on port 4011 when given the dhcp-alternate-port option, though there are some bugs related to that that I have worked around. I've confirmed with lsof that it's actually listening on that port when run with my configuration.

Here's my configuration if you're curious. Note again that this is a second DHCP server that's not running on port 67 that's supposed to supplement (for PXE boots) the other DHCP server on port 67 which is configured only to hand out addresses and whose configuration I cannot change.

Code:

enable-tftp    

tftp-secure

#tftp-root=             # Set from command line

############################################################        

# DHCP Configuration                                                 

# This is mainly to document how we can set this up as a "DHCP proxy"

# server that can supply other information to a PXE-booting host even

# though a different DHCP server running on the standard port supplies

# the address.            

#                                                         

# Unfortunately, this doesn't actually work since the iPXE boot code

# used by the VM doesn't send a request on the proxy port (4011). I

# am not clear why it does not do this.

# The address below is ignored since as a proxy we don't give out addresses.

dhcp-range=127.0.0.1,proxy

# This option causes the server to listen on the given port (default           

# 1067) _and_ on the PXE port 4011. Specifying the PXE port directly 

# (with `=4011` actually causes the server to fail because it appears

# to try to bind to the port twice. this behaviour is not documented

# in the manpage.

#                       

dhcp-alternate-port

dhcp-boot=/pxelinux.0

pxe-prompt="Press F8 for boot menu", 8

pxe-service=X86PC, "Boot from network", /pxelinux

pxe-service=X86PC, "Boot from local hard disk", 0

Quote:Everything is done using Option 43 on port 67/68. The port 4011 stuff was added for EFI clients. http://lists.thekelleys.org.uk/pipermail...10551.html

This I don't understand. Maybe reading through the PXE specification would help, but that link (from the Wikipedia PXE entry) wants a user name and password to let me download it. Is there an alternate source for this?

BitBub · 2017-02-08, 06:33

(2017-02-08 04:29)cjs Wrote: Has anybody else here actually seen iPXE send DHCP requests to port 4011? That really does appear to be the problem here, and I'm thinking the best thing to do might be to walk through the code and figure out how and where it's supposed to be doing this and doing enough tweaking on a test version of iPXE that we can see if this code path is actually being followed, if indeed it even exists at all.

Yes, but ipxe skips the 4011 stuff when boot info has already been received. In your case, as option 43 on port 68.

https://git.ipxe.org/ipxe.git/commit/c34...c1321fa583
https://git.ipxe.org/ipxe.git/commit/831...e54b6ad43c

Quote:Everything is done using Option 43 on port 67/68. The port 4011 stuff was added for EFI clients. http://lists.thekelleys.org.uk/pipermail...10551.html

(2017-02-08 04:29)cjs Wrote: This I don't understand. Maybe reading through the PXE specification would help, but that link (from the Wikipedia PXE entry) wants a user name and password to let me download it. Is there an alternate source for this?

http://www.intel.com/design/archives/wfm...xespec.htm

Basically, some proxy DHCP servers use option 43, others use port 4011. A few do both.

***NiKiZe*** · 2017-02-08, 21:10

(2017-02-08 04:29)cjs Wrote:
Quote:Everything is done using Option 43 on port 67/68. The port 4011 stuff was added for EFI clients. http://lists.thekelleys.org.uk/pipermail...10551.html

This I don't understand. Maybe reading through the PXE specification would help, but that link (from the Wikipedia PXE entry) wants a user name and password to let me download it. Is there an alternate source for this?

For reference ProxyDHCP for PXE booting has been used by WDS for a long time, can't say for which reasons dnsmasq supports it, but it is quite standard in PXE booting.

Will try to figure out more around this when time permits, but one idea that I'm to pursue is to add "PXEClient" as userclass in the standard DHCP response. But not yet sure how this all is used, but all the magic seem to happen around: https://git.ipxe.org/ipxe.git/blob/HEAD:...hcp.c#l409