[SOLVED] Problems with Asus Z87-K MB: NIC Does not reset after boot

[sebastian]

I have a problem with a Z87-K motherboard:
I have flashed in iPXE successfully into the BIOS PCI-ROM.

It successfully boot into the iscsi-target (Windows 7) ONCE. After that, Windows 7 works wonderfully.

However, after one reboot, iPXE fail at DHCP, and it won't even communicate with internet after setting a static IP.

However, booting a Linux live CD, so it connect the network card to the internet once, and then rebooting, then iPXE successfully DHCP ONCE again. After next reboot, same problem.

It seems like Windows 7 network driver "does" something with the network card so iPXE cannot use it, and the Linux live CD "resets" this state so its useable by iPXE again.

What is it? How can it be fixed?

[mcb30]

(2014-02-27 13:17)sebastian Wrote:  It successfully boot into the iscsi-target (Windows 7) ONCE. After that, Windows 7 works wonderfully.

However, after one reboot, iPXE fail at DHCP, and it won't even communicate with internet after setting a static IP.

However, booting a Linux live CD, so it connect the network card to the internet once, and then rebooting, then iPXE successfully DHCP ONCE again. After next reboot, same problem.

It seems like Windows 7 network driver "does" something with the network card so iPXE cannot use it, and the Linux live CD "resets" this state so its useable by iPXE again.

Does a full cold reset (i.e. removing the power cable) also clear the problem for a single boot?

Which NIC is present on your motherboard? You can find out using the commands "ifstat" (or "show chip") from within the iPXE shell.

If you know which iPXE driver is being used, then try adding an appropriate "DEBUG=" option to the iPXE build. For example, if your onboard NIC is a Realtek, try building a USB-bootable debug version of iPXE using:

make bin/ipxe.usb DEBUG=realtek

That should produce some output relating to the NIC initialisation, which you can post here.

Michael

[sebastian]

VEN is 10ec, DEV is 8168, so its a realtek.
Should try the DEBUG.

COLD BOOT solves the problem for one single boot.

[sebastian]

Here is the DEBUG output of FAILED boot:


Here is the DEBUG output of SUCCESS boot:


Only difference is the fourth line, which is "PHY status = 90" when failed and "PHY status = 88" when successful.

[mcb30]

(2014-02-27 15:25)sebastian Wrote:  Only difference is the fourth line, which is "PHY status = 90" when failed and "PHY status = 88" when successful.

That shouldn't matter, since the PHY gets reset after that anyway, and the final PHY status in both screenshots is the same (0x93 = 1000Mbps full-duplex, as expected).

Could you try building with DEBUG=realtek:3, and observe network traffic in real-time using tcpdump or wireshark? The extra debug will show you messages for each transmitted and received packet, and tcpdump/wireshark will allow you to see what's actually happening on the wire. It would be interesting to know if, for example, the failure case involves iPXE being able to transmit packets which show up on the wire, but not being able to receive anything.

It would also be good to know if the link status lights on the switch agree with the link status as reported by iPXE via the debug messages. I've just pushed an extra commit which adds an explanation of the PHY status register value to the debug messages.

Michael

[sebastian]

Link status on switch do agree with the reported link status.

Unable to do packet dump on the network, too much traffic and since its DHCP traffic, I cannot filter based on IP. Im tried to do the packet dump from the firewall machine since its the only machine (except for the iPXE machine) that can see the DHCP reply traffic. But the software cannot filter by MAC.

However, the debug messages show that the network card IS able to receive packets, but fail at DHCP anyays. So it MIGHT be some bug in the dhcp module or something like that, which isnt correctly reset after a warm reboot.

Cannot do any pics on successful boot, it boot pretty Quick so it start spamming debug messages when iSCSI negotiation and data transfer start.

All Three Pictures are FAILING boot:








However, is it possible to force the card to 100MBit durin iPXE boot, and then "upgrading" the link when I have got a DHCP lease?
Could so that with a embedded script. It might be some HW bug in the card that causes it.

[mcb30]

(2014-03-06 11:02)sebastian Wrote:  However, the debug messages show that the network card IS able to receive packets, but fail at DHCP anyays. So it MIGHT be some bug in the dhcp module or something like that, which isnt correctly reset after a warm reboot.

The messages show that you are receiving packets, but there is clearly something very wrong with what the hardware is reporting as received, since some of the packets apparently have a length of zero.

It would be very useful to know whether the problem affects both transmitted and received packets. If you are unable for whatever reason to capture the DHCP traffic (e.g. by filtering on "port 67 or port 68"), then you could try transmitting broadcast pings using e.g.

Code:

set net0/ip 10.0.0.123
set net0/netmask 255.255.255.0
ifopen net0
ping 10.0.0.255

If the transmit datapath is working but the receive datapath isn't, then you should be able to observe the broadcast pings using a packet capture with a filter for a source IP of 10.0.0.123.

Michael

[sebastian]

Did try that now.
Nope the transmit path is broken.
the only thing I find in the packet capture is ARP packets of other computers on the network who ask where the computer with iPXE on it is.

I also tried pinging one computer that is able to run Wireshark directly from iPXE. That computer does not see the pings either, so the transmit path are really broken.


I also observed this:


Network card returns "Operation not supported" http://ipxe.org/3c086003 on some RX packets

[mcb30]

(2014-03-06 13:20)sebastian Wrote:  Did try that now.
Nope the transmit path is broken.
the only thing I find in the packet capture is ARP packets of other computers on the network who ask where the computer with iPXE on it is.

I also tried pinging one computer that is able to run Wireshark directly from iPXE. That computer does not see the pings either, so the transmit path are really broken.

You need to use a broadcast ping, as per my example. Unicast pings won't work unless the receive datapath is functional, since they require the ability to receive ARP replies. A broadcast ping can still be sent even if the receive datapath is broken.

Could you retry using a broadcast ping ("ping 192.168.1.255") instead?

Thanks,

Michael

[sebastian]

I did try that too.
I did try BOTH a broadcast ping (192.168.1.255), and listening from the server 192.168.1.2, AND a unicast ping (192.168.1.2) to the server where wireshark is running.
Both did not result in any packets except a lot of ARP packets.

The screenshot was from the second try (eg with unicast ping).


Only packets I saw, when sending out the broadcast ping, was a lot of ARP packets originating from the other computers behind the same firewall: "Who has 192.168.1.65? Tell 192.168.1.101", "Who has 192.168.1.65? Tell 192.168.1.121" , "Who has 192.168.1.65? Tell 192.168.1.111", "Who has 192.168.1.65? Tell 192.168.1.105"
etc..
where 192.168.1.xxx is other computers on the network.

But the actual pings were not visible in the Wireshark.

[sebastian]

Now I was able to wireshark while there was no traffic on the network, so I could run the packet capture with no filters at all.

Turned out that a IP filter set on 192.168.1.65 also filters out ICMP traffic for some reason (or it ignores broadcast traffic).


Can now confirm that ICMP traffic are sent out when making broadcast pings from the iPXE computer when its in a "broken state" (wont get DHCP), so the transmit path is fine but not receive path.
Sorry for the misleading information, it was due to Wireshark filters because when I tested last, there was multiple users on the network downloading gigabytes of updates and such, so I could not run wireshark without filters because of all garbage traffic.

Now today the network was completely clear and vacant of traffic so I could run wireshark without filters.



Note that the bug only appears if the computer have completed a boot into Windows 7, for some reason, regardless of if this windows 7 is locally installed or installed on a iscsi-target. If the computer do not complete the boot into Windows 7 due to a premature Power loss or boot error, the iPXE will successfully DHCP on a subsuquent boot. Same when the computer is Cold rebooted (eg by turning off the switch on the Power supply)

Its only when a complete boot in Windows 7, that the network card is put into a state where it wont be able to receive traffic, but the card works fine while inside Windows 7.
Have tried poking around in the advanced settings and changing things that can prevent the NIC from receiving packets, like "green Ethernet", "Power saving mode" and such nonsense, but does not seem to have any effect.

Here is the official driver.
See if you can analyze the driver and see if it sets some state that cause iPXE to be unable to receive traffic:
http://support.asus.com/download.aspx?SL...Z87-K&os=8

---

FOUND THE SOLUTION!

Poked around in the Network card settings.

Found out that there was a setting "automatic link-down to 10MBIT on poweroff" that was set to "enabled".
Disabling this default setting, made the network card fully functional.

Maybe iPXE should look into this and add a patch in the network driver that saves all NVRAM/temporary network card settings somewhere, this could also be accomplished by a HTTP upload, like this:
"savesettings http://www.server.se/script.php"
that will do a HTTP upload to scrpt.php with all settings that you know work.
And then this uploaded file on server will contain a #!ipxe script that contains all settings read from card.
Basically, iPXE could pretend like you took a #!ipxe file and put into a <input type=file> box, so a php script can easly be used to receive this file.
This file is then added as a embedded script, and the network card settings will be preserved each boot, regardless what windows does with the settings.

And then you can add all these settings to a embedded script that restores the settings on each bootup, preventing Windows from garbaging the settings inside network card.

the culprit setting:


Default setting was to linkdown to 10mbit. Changed it to no speed down.

[mcb30]

(2014-03-10 10:18)sebastian Wrote:  FOUND THE SOLUTION!

Poked around in the Network card settings.

Found out that there was a setting "automatic link-down to 10MBIT on poweroff" that was set to "enabled".
Disabling this default setting, made the network card fully functional.

Fantastic find; well done!

I have pushed an update to the realtek driver which will dump out the MII (PHY) register contents as debug information (when building with DEBUG=realtek). Could you try reproducing the failure with the latest iPXE code, and post screenshots showing the MII register contents for success (warm reboot with "Not Speed Down") and failure (warm boot with "Automatic link-down to 10MBIT")?

If we can identify which PHY register is persisting across a PHY reset, then we can explicitly reset that register within the iPXE driver, which should make this problem go away permanently.

Thanks,

Michael

[sebastian]

Here it is: (ON = linkdown, OFF = no linkdown)

COLD BOOT with setting ON: (success):
1040 79c9 001c c800 0de1 0000 0064 2001 0000 0200 0000 0000 0000 0000 0000 2000

WARM BOOT with setting ON: (failure):
1040 79c9 001c c800 01e1 0000 0064 2001 0000 0200 0000 0000 0000 0000 0000 2000

WARM BOOT with setting OFF: (success):
1040 79c9 001c c800 0de1 0000 0064 2001 0000 0200 0000 0000 0000 0000 0000 2000


seems to be the 0de1/01e1 difference that is the culprit.
0d (working) -> BINARY ->
00001101
01 (nonworking) -> BINARY ->
00000001

Seems the 3rd and 4th bit from right of the first byte of the fifth register, needs to be SET for the card to work.


However, it was not only that setting after a Little bit of research. It was Another setting too, that in combination of this setting, sets the "incorrect" MII PHY register.

[Christian Bovey]

Your workaround solve also my link speed problem with Intel 82579LM/I217LM and AMT enabled.
- With standard configuration (Reduce Speed On Power Down => Enabled), link speed go down to 10MBit/s during boot.
- With custom configuration (Reduce Speed On Power Down => Disabled), link speed is correct (100MBit/s or 1GBit/s depending on switch).

[sebastian]

Also the other setting causing problem was "receive-side scaling". Seems like both setting caused problems in combination.

so try turning that off if you experience similiar problems.


But a good idea could be if a couple of other who has a GIGABIT REALTEK card with a VENDEV of "10ec:8168" could report in using the new DEBUG=realtek option, could report in their MII PHY registers here.

If they all are identical to:
1040 79c9 001c c800 0de1 0000 0064 2001 0000 0200 0000 0000 0000 0000 0000 2000

then the realtek driver could be coded to always set the MII PHY register to "1040 79c9 001c c800 0de1 0000 0064 2001 0000 0200 0000 0000 0000 0000 0000 2000" on each bootup, on all 10ec:8168 cards, thus guranteeing work even if some stubborn windows driver pokes there and it last through a reboot.

[mcb30]

(2014-03-10 14:26)sebastian Wrote:  COLD BOOT with setting ON: (success):
1040 79c9 001c c800 0de1 0000 0064 2001 0000 0200 0000 0000 0000 0000 0000 2000

WARM BOOT with setting ON: (failure):
1040 79c9 001c c800 01e1 0000 0064 2001 0000 0200 0000 0000 0000 0000 0000 2000

WARM BOOT with setting OFF: (success):
1040 79c9 001c c800 0de1 0000 0064 2001 0000 0200 0000 0000 0000 0000 0000 2000

seems to be the 0de1/01e1 difference that is the culprit.

Seems the 3rd and 4th bit from right of the first byte of the fifth register, needs to be SET for the card to work.

Strange; those bits are ADVERTISE_PAUSE_CAP and ADVERTISE_PAUSE_ASYM: they control what flow control capabilities the NIC advertises. In the failure case, the NIC is therefore advertising that it does not support sending or receiving pause frames. This should not be causing the symptoms you are seeing.

I've pushed an experimental quick patch to the "asusz87k" branch at https://git.ipxe.org/people/mcb30/ipxe.g...s/asusz87k. You can get the patch on its own from https://git.ipxe.org/people/mcb30/ipxe.g...01db671138 if you prefer. Could you try building with this patch, which should force that register back to 0x0de1, and see what effect it has?

Thanks,

Michael

[sebastian]

Nope. Patch did not fix the issue but the registers are now completely identical regardless of failing boot or good boot.

Do you have any idea of what RSS is? (Receive Side Scaling)?
http://technet.microsoft.com/en-us/libra...97036.aspx
http://msdn.microsoft.com/library/window...s.85).aspx

Seems that are the real culprit. See if you can make sure RSS is turned off on the network card when it boots.
It seems the autolinkdown affects the RSS setting too. There is a setting for RSS that causes the same issue.

[mcb30]

(2014-03-13 10:50)sebastian Wrote:  Nope. Patch did not fix the issue but the registers are now completely identical regardless of failing boot or good boot.

Do you have any idea of what RSS is? (Receive Side Scaling)?
http://technet.microsoft.com/en-us/libra...97036.aspx
http://msdn.microsoft.com/library/window...s.85).aspx

Seems that are the real culprit. See if you can make sure RSS is turned off on the network card when it boots.
It seems the autolinkdown affects the RSS setting too. There is a setting for RSS that causes the same issue.

As far as I can tell from the datasheet, the RTL8168 lacks the hardware required to support RSS: it has no support for multiple RX queues and no ability to direct interrupts to specific CPUs using MSI-X.

I've pushed another testing commit to the asusm87k branch, which will dump out all of the MAC registers immediately after initialisation.

Michael

[sebastian]

Here it come. I now post 4 variants.

COLD boot with SETTING on (SUCCESS):


WARM boot with SETTING on (FAILURE):


COLD boot with SETTING off (SUCCESS):


WARM boot with SETTING off (SUCCESS):


Hope this helps pinpointing the HW issue.

[mcb30]

(2014-03-17 12:08)sebastian Wrote:  Here it come. I now post 4 variants.

Thanks for that. There are many differences between the cold-boot and warm-boot cases, which is puzzling since it suggests that the chip reset functionality isn't actually doing much.

The warm-boot success and failure cases have the following differences:

- the Receive Configuration Register (offsets 0x44-0x47) has the value 0x00024f0e in the success case but 0x01024f0e in the failure case

- the Power Management Wakeup Frame0 register has a different value

- an unknown register at offset 0xd2 has the value 0x00 in the success case but 0x04 in the failure case.

Since we know the problem is with the receive datapath, the Receive Configuration Register looks to be the most likely culprit. The difference is in bit 24, which is listed in the 8168B datasheet as "reserved", and isn't defined in any other Realtek datasheet I can find. (I am unable to find datasheets for the later 8168 variants, which apparently do support RSS.)

I have pushed an extra commit to the "asusz87k" branch, which should clear bit 24 of the Receive Configuration Register. Could you test this and see if it makes any difference?

Thanks,

Michael