Intel i350 dualport, mac mismatch

[HHML]

Hi, because of my network topology (lacp channelbonding) I want to use iPXE (from usb) to pxe boot on one of the links in the ether-channel.

To get the (cisco) hardware to forward packets, it apparently needs lacp responses on one of the configured interfaces and iPXE kindly supports this.

This works, but the i350 dual-port gigabit interface behaves odd.

It now boots from net0, but this interface has the mac address of the linux equivalent eth1. After booting the linux kernel and bonding, the mac adress of eth0 is used.

Any change iPXE is going to fix this weird behaviour of the intel chipset?

[HHML]

Hi, because of my network topology (802.3ad channel bonding) I need to sent LACP answers to my switch which iPXE is gladly doing for me.

This enables me to pxe-boot servers connected to port-channels.

However the hardware I'm using is based on i350 gigabit interfaces and iPXE is having some problems detecting the correct 'first' interface.
When running with intel debug, I do see the interfaces get enumerated ok, but iPXE selects the second interface, introducing extra administrative load (got to configure both mac addresses, adding an extra layer of potential human failure).

Is there any work on this issue? Or will this always be an issue with multi-port interfaces?

[robinsmidsrod]

Are you sure that your Linux distribution isn't renaming the ethernet devices during bootup. I think udev can be configured to do this, and e.g. Ubuntu does this by default, as it detects new cards in the machine.

As far as I know, iPXE does not support LACP in any way, and you getting linkup on one of the channels is just the switch working in some standardized fallback mode when connected to a non-LACP system.

Also be aware that the way iPXE numbers ethernet devices is not guaranteed to be the same as your operating system does it. Different systems, different rules.

To be able to understand better what's going on, we'll need some sharp photos or log files which show what is going on.

[HHML]

(2013-10-27 10:34)robinsmidsrod Wrote:  Are you sure that your Linux distribution isn't renaming the ethernet devices during bootup. I think udev can be configured to do this, and e.g. Ubuntu does this by default, as it detects new cards in the machine.

It might be renaming devices, but the hardware has a one and two printed on the back of the server, Intel always uses the lowest numbered mac addresses for the '1' interface and the higher mac for the second interface. Running ipxe with intel debug on does show it is using the highest mac to sent/receive data. Ofcourse not a big deal but now I've got to add 2 mac addresses to dhcp/tftp config files.

(2013-10-27 10:34)robinsmidsrod Wrote:  As far as I know, iPXE does not support LACP in any way, and you getting linkup on one of the channels is just the switch working in some standardized fallback mode when connected to a non-LACP system.

Also be aware that the way iPXE numbers ethernet devices is not guaranteed to be the same as your operating system does it. Different systems, different rules.

To be able to understand better what's going on, we'll need some sharp photos or log files which show what is going on.

I'm pretty sure it is supporting the LACP negotiation, the switch refuses to forward packets when no lacp answers are received from the server. I'll try to get some debug logging from the switch, afaik I did see lacp packets from the ipxe client.

To make things more robust, I've embedded a script to use the net0 interface to boot and add both mac addresses to the dhcp and tftp services.

I'll try to take some pictures about the mac adresses and try to obtain a log from the switch.

btw: eth_slow.c states it implements a simple lacp entity. might ofcourse be dead code when using the intel driver, but I do think it is doing 'something' with lacp or the switch would keep refusing to forward packets.

Eric.

[HHML]

See http://pastebin.com/Bh1cNXfT for the cisco lacp debug.

At Oct 28 15:30:29 the switch is sending LACP-PDU (via Gi0/6) and it receives an answer. This does not happen with the original Intel PXE client. So it looks to me the iPXE is supporting port-channels (although on one interface but that isn't an issue).

Without the lacp replies, nothing is received by the dhcp server.


The screen shot at http://imagebin.org/275040 shows 2 network interfaces, port0 and port3.
Port3 is the internal (out of band) network interface and not suitable for booting over pxe.

I'ld expect 3 interfaces , 00:1e:67:85:06:fa and 00:1e:67:85:06:fb and the OOB interface.
iPXE doesn't select the internal/oob/port3 interface, but out of the two normal interfaces, it selects the one with the highest mac address, not quite what I would expect. A bit odd, but not a big issue.

It would be nice though, when it would use the lower numbered interface as this (the ..06:fa MAC) mac address will be used by the OS/Switch once the port-channel is up.

[robinsmidsrod]

Not experienced about LACP, so I'm not exactly sure how to read those logs. But in terms of understanding how it responds and why exactly the interface ordering is different, I think you'd need to talk with the developer of the Intel driver on the mailing-list. Send a message there. If you have any other 2-port NIC with another chipset (possibly a realtek), it would be interesting to see if it does the same thing (which seems counter-intuitive).

[HHML]

Unfortunately I've only got HP servers with dualport Intel interfaces.
The situation isn't a problem, but it is a bit weird to have to configure both interfaces (in DHCP/tftp) to get things working.

Eric.