Failed to restart TFTP issue

[richard_chen]

I am to an issue when iPXE load undionly.kpxe, it will show "Failed to restart TFTP",
1. bellow is my detail environment:
----------------------------------
1) have about 1000 NoteBook in my network
2) have three DHCP&iPXE server
3) These NB all with USB LAN card

2. issue like bellow info(I don't know how to upload picture as attachment, so...)
--------------------------------------------------------------------------
Realtek USB Ethernet controller (xHCI) v2.10AD(11/24/16)
CLIENT MAC ADDR:00 E0 4C 62 C6 26 GUID:44454C4C-3800-1056-8053-C4C04
CLIENT IP:100.110.153.105 MASK:255.255.0.0 DHCP IP:100.110.3.4
Downloaded WDSNBP from 100.110.3.4 WIN-RCLVTQ9LUFK
Press any key to cancel network boot service
Architecture:x64
Contacting Server: 100.110.3.4.............................................
Failed to restart TFTP.
TFTP download failed.
PXE-M0F:Exiting PXE ROM.
No Boot Device Found. Press any key to reboot the machine

3. Condition for issue happend
---------------------------------------------------------------------------
1) there lots of NB in network, if there have 100 NB, it can boot normal
2) more than DHCP&iPXE servers, if only have one DHCP&iPXE, it can boot normal
3) only happend with USB Ethernet, if NB with onborad Ethernet will boot normal
4) only happend with iPXE boot, WDS boot normal

I use Wireshark get the Packets, I found NB can get boot\x86\wdsnbp.com by TFTP, but after NB show "Contacting Server: 100.110.3.4............", NB show error like above info, and during NB show "Contacting Server: 100.110.3.4............", there have only two type packets, one is "DHCP request", the other is "DHCP ACK", who can hep me with this issue??

Many thanks,
Richard_Chen

[NiKiZe]

(2017-07-18 09:30)richard_chen Wrote:  I am to an issue when iPXE load undionly.kpxe, it will show "Failed to restart TFTP",
....
Failed to restart TFTP.
TFTP download failed.
PXE-M0F:Exiting PXE ROM.

This has nothing to do with iPXE or undionly.kpxe, The above error is from your PXE ROM which don't get any TFTP response.

Every error with iPXE will show a http://ipxe.org/xxxx url which will give more information (always post such an url if you get any issues with iPXE)

(2017-07-18 09:30)richard_chen Wrote:  I use Wireshark get the Packets, I found NB can get boot\x86\wdsnbp.com by TFTP, but after NB show "Contacting Server: 100.110.3.4............", NB show error like above info, and during NB show "Contacting Server: 100.110.3.4............", there have only two type packets, one is "DHCP request", the other is "DHCP ACK"

Check that your dhcp exchange are working correctly, and that your dhcp packets have the TFTP server IP and a filname that it can load.

And then check your TFTP servers logs to see what is happening (wireshark should also show enough information if there is any TFTP packets)


If it works on some nics but not others, comparing dhcp responses to see if they have different information is one way.

Also make sure that the machine responds properly in the full DHCP negotiation.

[richard_chen]

NiKiZe, thanks for your reply,
1.we use the server2012 for DHCP&iPXE, so we haven't found the logs for TFTP service
2.I checked wireshark packets of onboard LAN and USB LAN, no exception information was found.
3.we checked the packets of wireshark for normal boot iPXE NB, we found after NB show "Contacting server 100.110.3.4 .", it will start to get boot\undionly.kpxe by TFTP, there have only two types packets(DHCP request,ACK), and only need about 1 second it will start to get undiony.kpxe. But for fail boot iPXE NB, it always show dot with "Contacting server 100.110.3.4", and after about one minutes, it will show TFTP error, i checked the packets of wireshark during show "Contacting server 100.110.3.4.....................", there only have DHCP request and ACK, no any others packets, also have no packets for starting TFTP boot\undionly.kpxe. I am very puzzled that why it can get boot\x86\wdsnbp.com by TFTP at first, and it show TFTP error after "Contacting server 100.110.3.4...............", what is it doing during "Contacting server 100.110.3.4........"

[SebastianRoth]

(2017-07-18 09:30)richard_chen Wrote:  3. Condition for issue happend
---------------------------------------------------------------------------
1) there lots of NB in network, if there have 100 NB, it can boot normal

To me this sounds as if your TFTP server cannot handle all the requests if there are "too many". As already mentioned this has nothing to do with iPXE.

(2017-07-18 09:30)richard_chen Wrote:  2) more than DHCP&iPXE servers, if only have one DHCP&iPXE, it can boot normal

What do you mean by iPXE server? TFTP server hosting the iPXE binary or HTTP server hosting the iPXE script? Possibly you have your DHCP servers configured differently. In this case you'd see client boot properly sometimes but fail when clients get an answer from a DHCP server giving false information.

(2017-07-18 09:30)richard_chen Wrote:  ... we checked the packets of wireshark for normal boot iPXE NB, ...

How and where did you capture the packets? Next to the client (using a hub or mirror port on the switch) or on the server - which server exactly? As you seem to have several servers for DHCP/TFTP you better capture on the client side to make sure you don't miss any packets...

[richard_chen]

SebastianRoth, thanks for your reply,
I'm sorry to say nothing about the quantity of DHCP&IPXE, we found if there only have one DHCP server, it can boot normal. if there have two and more DHCP, it will happend this issue.
I get these packets with wireshark from mirror port on the switch, and packets like bellow,
1. DHCP Discover
2. DHCP offer --- DHCP 1
2. DHCP offer --- DHCP 2
2. DHCP offer --- DHCP 3
3. DHCP request ---source:0.0.0.0 Des:255.255.255.255
4. DHCP ACK----Source:XX.XX.XX.XX(DHCP serverip) 回覆your client ip:xx.xx.xx.xx
5. DHCP request
6. DHCP ACK ---- boot file name: boot\x86\wdsnbp.com
7. TFTP ---read request, File: boot\x86\wdsnbp.com
8. lots of packets for wdsnbp.com by TFTP
9. DHCP request
10. DHCP ACK ---- boot file name: boot\undionly.kpxe

After step 10, it always repeat setepping 9&10 for each 5 seconds.
Any idea for it debugging?

[NiKiZe]

You should never have more then one DHCP server per network.
Some DHCP servers has redundancy options so that they can work together, but you should never have multiple offers like that.

You can have one DHCP server and one ProxyDHCP but that's different.

For information about how the Microsoft DHCP can be configured see the first part of http://ipxe.org/appnote/chainload_wds and http://ipxe.org/howto/msdhcp

[richard_chen]

(2017-07-24 09:06)NiKiZe Wrote:  You should never have more then one DHCP server per network.
Some DHCP servers has redundancy options so that they can work together, but you should never have multiple offers like that.

You can have one DHCP server and one ProxyDHCP but that's different.

For information about how the Microsoft DHCP can be configured see the first part of http://ipxe.org/appnote/chainload_wds and http://ipxe.org/howto/msdhcp

Considering that if there is only one server, if a fault occurs, it will cause the production line to stop, so we use multiple DHCP servers to assign different IP scope. we use this solution for WDS and iPXE long time.
Current this issue only happend with USB LAN card, production with lots of units and more than one DHCP, If any of these three conditions is missing, this issue also can not been duplicated, these units have no onboard LAN, so we must use USB LAN card. We also cannot split the network, because have no lots of procution server. so current used one DHCP&iPXE server for short time solution. However, this solution poses a risk of stopping production.
The point of this issue is why it always send DHCP request when DHCP reply ACK? and why it always repeat this stepping and not begin to go next stepping for TFTP undionly.kpxe?

[SebastianRoth]

(2017-07-24 03:29)richard_chen Wrote:  SebastianRoth, thanks for your reply,
I'm sorry to say nothing about the quantity of DHCP&IPXE, we found if there only have one DHCP server, it can boot normal. if there have two and more DHCP, it will happend this issue.
I get these packets with wireshark from mirror port on the switch, and packets like bellow,
1. DHCP Discover
2. DHCP offer --- DHCP 1
2. DHCP offer --- DHCP 2
2. DHCP offer --- DHCP 3
3. DHCP request ---source:0.0.0.0 Des:255.255.255.255
4. DHCP ACK----Source:XX.XX.XX.XX(DHCP serverip) your client ip:xx.xx.xx.xx
5. DHCP request
6. DHCP ACK ---- boot file name: boot\x86\wdsnbp.com
7. TFTP ---read request, File: boot\x86\wdsnbp.com
8. lots of packets for wdsnbp.com by TFTP
9. DHCP request
10. DHCP ACK ---- boot file name: boot\undionly.kpxe

Ahhhhh, now I see the picture a bit more clear:

1. You have several DHCP servers which are all handing out IPs in different scopes. While this is not a standard setup I suppose this could work when setup properly. That said I still wouldn't recommend doing it as some clients might get confused when getting several answers with different IPs and PXE information!
2. From the wireshark packet listing I get that you are chainloading iPXE from WDSNBP (boot\x86\wdsnbp.com). Why do you do this? Why not just telling the client to load and boot iPXE directly?
   

To be able to help we need all the details of which server is answering what in both cases. I am willing to help you debugging this further if you take more wireshark packet dumps of the full boot process that is working (one where you only have a single DHCP server and USB LAN and one where several DHCPs are running but a PC with a different network card is booting properly). Please upload The three packet dumps (non working with USB LAN, working with single DHCP server and working with non USB LAN) here or to a different location where I can download those. In case you don't want to upload the files publicly just send me a private message with the links!
As well you need to tell us which USB LAN adapter you exactly have!! We need the exact VEN(DOR) and DEV(ICE) ID. Check out this screenshot on where to find those in windows: http://www.ccboot.com/upload/nicprop.jpg

As already mentions this is not an issue caused by iPXE. But I highly doubt that you'll get help from Microsoft ™ on this I am willing to offer my help if you provide the packet dumps!