Infoblox and iPXE

[jov0]

Has anyone experienced DHCP issues using iPXE with Infoblox? I have an Infoblox appliance where DHCP works perfectly for Etherboot and gPXE ROMs, but works only sporadically with iPXE ROMs. When using iPXE the DHCP configuration times out. A packet capture reveals the DHCP server is sending a response. Additionally, using a different DHCP service such as AIX's dhcpsd there are no issues with iPXE+DHCP. I have observed the behavior on Intel Pro 1000 (e1000e) and Intel i210 (igb) NICs. Any insight would be appreciated.

Thanks,

[mcb30]

You could check the output from ifstat to see if any packets are being received and, if so, what errors are being recorded for these packets.

For more detail, you could try building iPXE with DEBUG=dhcp to see if the DHCP response is being received by iPXE but discarded for some reason.

Michael

[jov0]

Michael, Thank you for the tip on building with the DEBUG option. I built it with the dhcp and intel options. We were easily able to find the problem with the output. We have a local infoblox and a centralized fail over: both of which intentionally respond to the DHCP requests. The problem is both request go out and both DHCP servers respond, but our Cisco firewall is chopping off the second responder as it thinks the connection should be closed down. The dhcp servers are not in the same VLAN as the end points using iPXE. We can see that iPXE will ignore a DHCP response from an IP that differs from offer where gPXE was not as strict. As a work around I will have to dive in and see if I can find that logic in the source, but if anyone can reduce my search time I would appreciate it. We are also opening up a ticket with Cisco to fix the real problem, but there turn around time is typically sub optimal.

[mcb30]

(2014-11-10 04:27)jov0 Wrote:  Michael, Thank you for the tip on building with the DEBUG option. I built it with the dhcp and intel options. We were easily able to find the problem with the output. We have a local infoblox and a centralized fail over: both of which intentionally respond to the DHCP requests. The problem is both request go out and both DHCP servers respond, but our Cisco firewall is chopping off the second responder as it thinks the connection should be closed down. The dhcp servers are not in the same VLAN as the end points using iPXE. We can see that iPXE will ignore a DHCP response from an IP that differs from offer where gPXE was not as strict. As a work around I will have to dive in and see if I can find that logic in the source, but if anyone can reduce my search time I would appreciate it. We are also opening up a ticket with Cisco to fix the real problem, but there turn around time is typically sub optimal.

What's the exact message you get from DEBUG=dhcp which indicates the nature of the problem?

Michael

[isi_spike]

Every time I run into DHCP issues with PXE it ends up being an STP issue. Make sure that you have PortFast turned on or that you have disabled STP for your network port. I think this should clear up your issue.

[jov0]

(2014-11-10 15:30)mcb30 Wrote:  What's the exact message you get from DEBUG=dhcp which indicates the nature of the problem?

Michael

Michael here is an example output for when the Firewall is enabled and cuts off the dhcp acknowledgement from 172.23.223.141. Because 10.64.38.20 is physically closer, 172.23.223.141 acks almost always come in second and consequently are dropped by the firewall. This is a combination of design issues between infoblox's fail over setup and Cisco's firewall abilities for this variety of traffic. We do not see the same behavior using Etherboot or gPXE in that it either they do not care who is sending the ack or are making the request against first offer versus the last. I would test this for sure, but the last gpxe src from 2011 does not compile.

[mcb30]

(2014-11-11 05:35)jov0 Wrote:  Michael here is an example output for when the Firewall is enabled and cuts off the dhcp acknowledgement from 172.23.223.141. Because 10.64.38.20 is physically closer, 172.23.223.141 acks almost always come in second and consequently are dropped by the firewall. This is a combination of design issues between infoblox's fail over setup and Cisco's firewall abilities for this variety of traffic. We do not see the same behavior using Etherboot or gPXE in that it either they do not care who is sending the ack or are making the request against first offer versus the last. I would test this for sure, but the last gpxe src from 2011 does not compile.

Thanks. Are you able to disable either of the DHCP servers from responding to the initial DHCPDISCOVER?

If not, then as a workaround you could delete the check in dhcp_request_rx() in net/udp/dhcp.c:

Code:

if ( server_id.s_addr != dhcp->server.s_addr )
  return;

Michael

[jov0]

(2014-11-12 15:12)mcb30 Wrote:  Thanks. Are you able to disable either of the DHCP servers from responding to the initial DHCPDISCOVER?

If not, then as a workaround you could delete the check in dhcp_request_rx() in net/udp/dhcp.c:

Code:

if ( server_id.s_addr != dhcp->server.s_addr )
  return;

Unfortunately there exists a requirement for immediate fail-over and Infoblox's implementation options are limited.

I really appreciate your help. Also thank you for highlighting where the IP address comparison is located. I think you guys are doing great dev work with this project and can't imagine going back to something as primitive as a basic PXE ROM.

[jov0]

Does anyone by chance know if there exists an RFC describing the verification that the DHCP Offer and Ack IP's match? I am trying to bolster my argument with Cisco in correcting their firewall behavior.

[robinsmidsrod]

I didn't read too closely, but DHCP client behavior is defined in RFC 2131 paragraph 4.4, and net/udp/dhcp.c:523 seems to be the DHCP ACK behavior, so you should be able to find something about it in the ACK-related documentation.

Please do report back what you find and if you think iPXE's behavior might be questionable. You might also want to closely look at what the Intel PXE specification says about the topic.