iMAC with Broadcom 14e4-1686 BCM57766 : link status down

[marcolefo]

Hi

Working with FOG, we have an iPXE issue with iMac Late 2015 with a Broadcom ethernet adapter.

Here is the thread on FOG forum :
https://forums.fogproject.org/topic/9269...-ipxe-boot

To resume it, when we boot on ipxe :

Code:

IPXE initialising devices...ok
Waiting for link-up on net0... failed: Down (http://ipxe.org/38086193)

On ipxe shell, ifstat say :

Code:

net0: 38:c9:86:xx:xx:xx using 14e4-1686 on 0000:03:00.0 (closed)
  [Link: down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Down (http://ipxe.org/38086193)]

In the ipxe shell, we have tried to set up ip manually.

We have made our own ipxe.efi with https://rom-o-matic.eu with :
- all drivers
- only 14e4-1686

We have tried also all the efi image provided by FOG developers.

We have changed RJ 45 cable.
We have portfast activated on the switch port and tested with a non manageable switch. The link LED is always on.

Is there a problem with the driver ?
Thanks for help.

[NiKiZe]

The driver says the link is down.
To debug this you could try and build ipxe from current git master with

Code:

make bin-x86_64-efi/ipxe.efi DEBUG=tg3

this should give you some status information that will be a start for us to help you figure out what the issue is and hopefully fix it.

one other thing to test is to try and use snp.efi instead, which will default to the snp or nii driver provided by the firmware, this is in no way a fix but rather a possible debug step to ensure everything else is working as expected.

[marcolefo]

Hi ! Thanks for your help.

(2017-01-18 21:13)NiKiZe Wrote:  The driver says the link is down.
To debug this you could try and build ipxe from current git master with

Code:

make bin-x86_64-efi/ipxe.efi DEBUG=tg3

this should give you some status information that will be a start for us to help you figure out what the issue is and hopefully fix it.

I have tried three debug mode : tg3 tg3:3 and tg3:4
tg3 only says nothing more.
Here are the results for the two other mode:
https://youtu.be/rvjjC_ZHtw8
https://youtu.be/vPOcXwJUikU

Quote:one other thing to test is to try and use snp.efi instead, which will default to the snp or nii driver provided by the firmware, this is in no way a fix but rather a possible debug step to ensure everything else is working as expected.

With the snp.efi, provided by FOG :

Code:

Configuring (net0 38:c9:86:12:xx:xx)..............No configuration method succeeded (http://ipxe.org/040ee186)

[SebastianRoth]

Bumping this post as we seem to have another one here https://forums.fogproject.org/topic/9278...ac-netboot

iMac17,1 (Late 2015 model) as well as iMac14,1 with the same ethernet chip. Find more details in the FOG forum posts. I guess it's best to focus on that iMac17,1 model first. If we can get this to work the other one might be fixed as well.

We have of Macmini6,2 with the same ethernet chip (same PCI IDs) proof working:

Possibly the issue is with net1 interface (wireless adapter AFAIK) being initialized first and then net0 (ethernet)?!

@marcolefo You might want to add even more debugging with DEBUG=tg3,tg3_hw,tg3_phy

[NiKiZe]

please test with the official ipxe git master. (so not the fog sources)
go in to the shell, enter dhcp, and then ifstat and post the result of the whole process.

the :3 for each .c file in the DEBUG option is a bit mask, meaning :1 (default) gives first level debug output, :2 gives second level (and makes no sense to use) :3 gives both first and second level, :4 is AFAIK never used so should give absolutely nothing.

since this is a tg3 only issue test this with: make bin-x86_64-efi/tg3.efi DEBUG=tg3,tg3_hw,tg3_phy
that will also eliminate the possible issue of wireless.

It might also be interesting to test from full power of, as well as after a reset without power of, since it might be something in some register some where that gets set or cleared in different scenarios.

[SebastianRoth]

Hi NiKiZe,

thanks heaps for getting back so quickly. Hope we can figure this one out with your help.

(2017-02-02 22:43)NiKiZe Wrote:  please test with the official ipxe git master. (so not the fog sources)
go in to the shell, enter dhcp, and then ifstat and post the result of the whole process.

Definitely a good idea to try the very latest ipxe code although we are not much behind. Usually Tom is updating to the latest every two or so weeks.

(2017-02-02 22:43)NiKiZe Wrote:  since this is a tg3 only issue test this with: make bin-x86_64-efi/tg3.efi DEBUG=tg3,tg3_hw,tg3_phy
that will also eliminate the possible issue of wireless.

Also worth a try but I doubt that it will get around the wireless. See this video https://youtu.be/vPOcXwJUikU - from sec 5 to sec 10 - to me this looks like tg3 is also recognizing net1 (wireless adapter).
Here is a screenshot showing the same without debug - first net1 and then net0, weird: https://forums.fogproject.org/uploads/fi...1694-2.jpg

Just found this https://launchpadlibrarian.net/230900456/Lspci.txt (from https://bugs.launchpad.net/ubuntu/+sourc...ug/1525615) - not sure if this is of any help but definitely good to have the PCI IDs of all the devices in those iMac 17,1 machines. 14e4:43ba should definitely not be recognized by any of the iPXE drivers. But why do we see net1 and net0?!?

Quote:Broadcom Corporation BCM43602 802.11ac Wireless LAN SoC [14e4:43ba]

Take care,
Sebastian

[NiKiZe]

in your video it detects net1
in that image there is only net0 - but twice (no net1)

Again please give the output of dhcp followed by ifstat from official ipxe git sources, without any configuration modifications at all.

Just to limit this down to just the tg3 driver use make bin-x86_64-efi/tg3.efi

so steps I really want you to do so we have a common, minimal, reproducable, understanable testcase:
git clone git.ipxe.org/ipxe.git
cd ipxe/src
make bin-x86_64-efi/tg3.efi
(note that there should not be any configuration changes at all)
cp make bin-x86_64-efi/tg3.efi -> boot device

press ctrl-b when booting
type dhcp
type ifstat
take a picture and upload

so far there is no information provided at all with output from ifstat after running dhcp.

we might also want the mapping from net0/1 to the busid as shown by show net0/busid:hex and show net1/busid:hex


There is lots of features enabled in that video, maybe this is just an issue with VLAN support enabled, or maybe something else, but we need to know if the absolute minimal testcase works at all first, is it a total driver issue or not.

[marcolefo]

Hi.
Thanks again for your help.

(2017-02-04 00:34)NiKiZe Wrote:  in your video it detects net1
in that image there is only net0 - but twice (no net1)

Again please give the output of dhcp followed by ifstat from official ipxe git sources, without any configuration modifications at all.

Just to limit this down to just the tg3 driver use make bin-x86_64-efi/tg3.efi

so steps I really want you to do so we have a common, minimal, reproducable, understanable testcase:
git clone git.ipxe.org/ipxe.git
cd ipxe/src
make bin-x86_64-efi/tg3.efi
(note that there should not be any configuration changes at all)
cp make bin-x86_64-efi/tg3.efi -> boot device

Ok, I've done that :

Quote:press ctrl-b when booting
type dhcp
type ifstat
take a picture and upload

I can't do that. When asked, I have pressed ctrl-b and I didn't go to the shell but the iMac boot to OS.

[NiKiZe]

(2017-02-13 17:05)marcolefo Wrote:  

(2017-02-04 00:34)NiKiZe Wrote:  Again please give the output of dhcp followed by ifstat

Ok, I've done that :

Sorry but this image has ifstat followed by dhcp, we need dhcp followed by ifstat

(2017-02-13 17:05)marcolefo Wrote:  

(2017-02-04 00:34)NiKiZe Wrote:  press ctrl-b when booting
type dhcp
type ifstat
take a picture and upload

I can't do that. When asked, I have pressed ctrl-b and I didn't go to the shell but the iMac boot to OS.

Might be a notorious EFI bug, you can press ESC followed by b to emulate ctrl-b
Also note that you can't have a embeded script, since that disables the default prompt, if it still does not work you can add a embedded script with just shell to get into the shell.

So for this case lets go with: testscript.ipxe

Code:

#!ipxe
dhcp
ifstat
shell

build with

Code:

make bin-x86_64-efi/tg3.efi EMBED=testscript.ipxe

and test the resulting bin-x86_64-efi/tg3.efi

After uploading an image of running with the above embeded script without debug, please also build and do the same with

Code:

make bin-x86_64-efi/tg3.efi DEBUG=tg3,tg3_hw,tg3_phy EMBED=testscript.ipxe

It will probably be next to unreadable due to all output, so we might need to go thru this a couple of times.

If you can it will be easier to debug if you could join #ipxe at irc.freenode.net for example via: http://webchat.freenode.net/?channels=#ipxe if you don't have or want to install a client. Personally I'm mostly available from 19:00 CET

[speleo14]

Having the same problem, did anybody find a solution?

Thanks!

[marcolefo]

(2017-02-13 18:35)NiKiZe Wrote:  Might be a notorious EFI bug, you can press ESC followed by b to emulate ctrl-b

Yes, that's work, thanks .

So, dhcp followed by ifstat :

(2017-02-13 18:35)NiKiZe Wrote:  Also note that you can't have a embeded script, since that disables the default prompt, if it still does not work you can add a embedded script with just shell to get into the shell.

So for this case lets go with: testscript.ipxe

Code:

#!ipxe
dhcp
ifstat
shell

build with

Code:

make bin-x86_64-efi/tg3.efi EMBED=testscript.ipxe

and test the resulting bin-x86_64-efi/tg3.efi

Maybe it's a clue : I've tried that and I never go to the shell.

(2017-02-13 18:35)NiKiZe Wrote:  After uploading an image of running with the above embeded script without debug, please also build and do the same with

Code:

make bin-x86_64-efi/tg3.efi DEBUG=tg3,tg3_hw,tg3_phy EMBED=testscript.ipxe

Here is the video of the boot :
https://youtu.be/dZJCd0R7ZG8


Thanks for your help.

[NiKiZe]

(2017-03-07 11:45)marcolefo Wrote:  So, dhcp followed by ifstat :

Link down, no tx, no rx.

my bad, since dhcp fails it exits use the double pipe (||) at the end of dhcp:

Code:

#!ipxe
dhcp ||
ifstat
shell

(2017-03-07 11:45)marcolefo Wrote:  Here is the video of the boot :
https://youtu.be/dZJCd0R7ZG8

Interesting that the link goes up, can we identify exactly when and how that link up message is shown? Is it when ipxe exits?

I think we need to dig much further into the driver then this but maybe full debug enabled could give us something more:

Code:

make bin-x86_64-efi/tg3.efi DEBUG=tg3:7,tg3_hw:7,tg3_phy:7 EMBED=testscript.ipxe

[marcolefo]

(2017-03-07 22:41)NiKiZe Wrote:  my bad, since dhcp fails it exits use the double pipe (||) at the end of dhcp:

Code:

#!ipxe
dhcp ||
ifstat
shell

Great.

(2017-03-07 22:41)NiKiZe Wrote:  Interesting that the link goes up, can we identify exactly when and how that link up message is shown? Is it when ipxe exits?

I don't know exactly. It's when it initialize devices ;).
When it happens, my dhcp server give an IP to the workstation. I can see a complete dhcp sequence in dhcpd logs.

(2017-03-07 22:41)NiKiZe Wrote:  I think we need to dig much further into the driver then this but maybe full debug enabled could give us something more:

Code:

make bin-x86_64-efi/tg3.efi DEBUG=tg3:7,tg3_hw:7,tg3_phy:7 EMBED=testscript.ipxe

It's very difficult to see the logs. Is there any way to redirect to a file ?
https://youtu.be/hxddZjh0Hhs

In slow motion ;) :
https://youtu.be/NoGQuDImPMU

Thanks again.

[speleo14]

Sorry to mix in... as I mentioned, I have the same problem, and maybe this information can help:

We have several iMacs of that model; some we bought more than a year ago, and I installed Windows on some of them using iPXE and WDS - without any problems! It's only the ones that we bought recently that will not PXE-boot.
It seems as if the ones that came with Sierra (10.12) pre-installed will not work with iPXE while the ones that had El Capitan (10.11) are OK.

I checked with System Profiler to find any differences between these two, hmm, versions of that model and found the following:

iMac 27'' Retina 5K Late 2015, bought in May 2016 (iMac17,1):
Boot ROM: IM171.0105.B05
SMC Version: 2.33f10

Ethernet:
Broadcom 57766-A1:
Manufacturer ID: 0x14e4
Device ID: 0x1686
Versions-ID: 0x0001
Kext-Name: AppleBCM5701Ethernet.kext
Firmware-Version: 57766a-v1.15, 0xad0d59c9
Version: 10.1.12

OS: 10.11.3


iMac 27'' Retina 5K Late 2015, bought in March 2017 (iMac17,1):
Boot ROM: IM171.0105.B15
SMC Version: 2.34f2

Ethernet:
Broadcom 57766-A1:
Manufacturer ID: 0x14e4
Device ID: 0x1686
Versions-ID: 0x0001
Kext-Name: AppleBCM5701Ethernet.kext
Firmware-Version: 57766a-v1.15, 0xad0d59c9
Version: 10.2.7

OS: 10.12.3

Differences are in bold. Both versions use the very same ethernet NIC, the same kext and the same firmware on the NIC, just the "version" (whatever this is) is different. However, the Boot ROM and SMC versions are different on the two iMacs.
Thus, I suspect that nothing is wrong with the Broadcom NIC and its iPXE driver, but with the newer BootROM / firmware on the newer iMacs.

I have no idea if this helps... Unfortunately, I don't have one of the older "versions" of the iMac at hand right now, they're all out with my users, so I can't compare iPXE boots, but I could try to get one back for a day or two - then we could try to hunt down differences?

[NiKiZe]

(2017-03-09 17:22)speleo14 Wrote:  iMac 27'' Retina 5K Late 2015, bought in May 2016 (iMac17,1):
Boot ROM: IM171.0105.B05
SMC Version: 2.33f10

Boot ROM: IM171.0105.B15
SMC Version: 2.34f2

In that case I only think that boot ROM might be relevant.

Does any of these work with snponly.efi?

[speleo14]

(2017-03-09 18:01)NiKiZe Wrote:  In that case I only think that boot ROM might be relevant.

I agree. I have now tested 2 iMacs, both were bought at the same time (spring 2016). One was upgraded to Sierra (10.12), the other is still on Capitan (10.11).
The 10.12 iMac has Boot-ROM IM171.0105.B15, and doesn't PXE boot ("link down").
The 10.11 iMac has Boot-ROM IM171.0105.B05, and PXE booting does work.
So the Sierra upgrade does also upgrade the Boot-ROM, and this breaks booting with iPXE.
The SMC version does not seem to be relevant, both have 2.33f10.

Currently checking if the Boot-ROM can be downgraded... but not much hope.

(2017-03-09 18:01)NiKiZe Wrote:  Does any of these work with snponly.efi?

Will try and post back.

---

OK, built snponly.efi with "make bin-x86_64-efi/snponly.efi".

Trying to boot with that results in "No more network devices" on all machines, that is, no NICs were even recognized (neither internal nor external).

[mastacontrola]

(2017-03-10 09:12)speleo14 Wrote:  

(2017-03-09 18:01)NiKiZe Wrote:  In that case I only think that boot ROM might be relevant.

I agree. I have now tested 2 iMacs, both were bought at the same time (spring 2016). One was upgraded to Sierra (10.12), the other is still on Capitan (10.11).
The 10.12 iMac has Boot-ROM IM171.0105.B15, and doesn't PXE boot ("link down").
The 10.11 iMac has Boot-ROM IM171.0105.B05, and PXE booting does work.
So the Sierra upgrade does also upgrade the Boot-ROM, and this breaks booting with iPXE.
The SMC version does not seem to be relevant, both have 2.33f10.

Currently checking if the Boot-ROM can be downgraded... but not much hope.

(2017-03-09 18:01)NiKiZe Wrote:  Does any of these work with snponly.efi?

Will try and post back.

---

OK, built snponly.efi with "make bin-x86_64-efi/snponly.efi".

Trying to boot with that results in "No more network devices" on all machines, that is, no NICs were even recognized (neither internal nor external).

This, to me, (This is the "Tom" referred to above about the developer of FOG.) sounds exactly like a firmware issue.

Maybe we need to go to Apple and somehow file a Firmware issue? While I'm not sure how Apple feels about PXE Booting machines, the fact remains, from what I can tell here, that they pushed an update and that update broke something unexpectedly. These Mac's aren't going away anytime soon, and updating firmware to limit potential areas of boot seems like a rather large 'exploitation' of customer's own rights when purchasing machines.

I don't know who we would talk to, but I really doubt this is a problem from FOG or iPXE, but rather some "fix" Apple added to the ROM information on the NIC. Whether this was intentional or not I don't know.

One of the problems I've had with Apple is they tend not to care about the consumer and treat the customers as if they're temporary borrowers of their equipment. They're typically helpful if you do have an issue and are in their support terms, but the moment your warranty or "apple care" expires, they're really quick to try to make you go out and buy new equipment. I'm still not sure how to report a bug like this with Apple, though I'm certain somebody knows how somewhere.

Sorry about the vent there. Supporting Mac's should not be this difficult and it seems anytime a new update comes out for "Apple" devices, a new issue is broken and rarely does there seem to be a fix implemented.

[speleo14]

(2017-03-13 13:19)mastacontrola Wrote:  This, to me, (This is the "Tom" referred to above about the developer of FOG.) sounds exactly like a firmware issue.

Maybe we need to go to Apple and somehow file a Firmware issue? While I'm not sure how Apple feels about PXE Booting machines, the fact remains, from what I can tell here, that they pushed an update and that update broke something unexpectedly. These Mac's aren't going away anytime soon, and updating firmware to limit potential areas of boot seems like a rather large 'exploitation' of customer's own rights when purchasing machines.

I don't know who we would talk to, but I really doubt this is a problem from FOG or iPXE, but rather some "fix" Apple added to the ROM information on the NIC. Whether this was intentional or not I don't know.

One of the problems I've had with Apple is they tend not to care about the consumer and treat the customers as if they're temporary borrowers of their equipment. They're typically helpful if you do have an issue and are in their support terms, but the moment your warranty or "apple care" expires, they're really quick to try to make you go out and buy new equipment. I'm still not sure how to report a bug like this with Apple, though I'm certain somebody knows how somewhere.

Sorry about the vent there. Supporting Mac's should not be this difficult and it seems anytime a new update comes out for "Apple" devices, a new issue is broken and rarely does there seem to be a fix implemented.

I agree completely!

I have an Apple developer account, so I can file a bug report; in fact, I have already been thinking about doing so, just wanted to wait until we are sure that it is indeed the firmware.
So, since we all agree that this is an issue with Apple's firmware, I will do so - however, it's not unlikely that they will more or less ignore it.
Maybe it would be a good idea if somebody else also reports this to Apple.

[mastacontrola]

(2017-03-15 10:06)speleo14 Wrote:  

(2017-03-13 13:19)mastacontrola Wrote:  This, to me, (This is the "Tom" referred to above about the developer of FOG.) sounds exactly like a firmware issue.

Maybe we need to go to Apple and somehow file a Firmware issue? While I'm not sure how Apple feels about PXE Booting machines, the fact remains, from what I can tell here, that they pushed an update and that update broke something unexpectedly. These Mac's aren't going away anytime soon, and updating firmware to limit potential areas of boot seems like a rather large 'exploitation' of customer's own rights when purchasing machines.

I don't know who we would talk to, but I really doubt this is a problem from FOG or iPXE, but rather some "fix" Apple added to the ROM information on the NIC. Whether this was intentional or not I don't know.

One of the problems I've had with Apple is they tend not to care about the consumer and treat the customers as if they're temporary borrowers of their equipment. They're typically helpful if you do have an issue and are in their support terms, but the moment your warranty or "apple care" expires, they're really quick to try to make you go out and buy new equipment. I'm still not sure how to report a bug like this with Apple, though I'm certain somebody knows how somewhere.

Sorry about the vent there. Supporting Mac's should not be this difficult and it seems anytime a new update comes out for "Apple" devices, a new issue is broken and rarely does there seem to be a fix implemented.

I agree completely!

I have an Apple developer account, so I can file a bug report; in fact, I have already been thinking about doing so, just wanted to wait until we are sure that it is indeed the firmware.
So, since we all agree that this is an issue with Apple's firmware, I will do so - however, it's not unlikely that they will more or less ignore it.
Maybe it would be a good idea if somebody else also reports this to Apple.

I don't have an account, though creating one shouldn't be too problematic. That said, as this does appear to be a firmware issue, if we can't hold our breaths that a fix will come to place, might it be easier to find out a "change log" of what was changed between firmware B05 vs. B15? Even if it means something was added/adjusted that might be able to be corrected for in the iPXE BCM driver handling? Of course this would lead down a road that would essentially make the Mac's only usable on an 'ipxe' created driver system (Leaving out SNP/SNPONLY I suppse?)

Just thoughts in my head. I don't expect we'll see Apple pushing out a fix for this because of iPXE not working, but maybe we can get what was changed so the Opensource community might be able to provide a work around.

[speleo14]

(2017-03-15 14:43)mastacontrola Wrote:  I don't have an account, though creating one shouldn't be too problematic.

Yes it is, because it's not for free (Apple.... ).

(2017-03-15 14:43)mastacontrola Wrote:  That said, as this does appear to be a firmware issue, if we can't hold our breaths that a fix will come to place, might it be easier to find out a "change log" of what was changed between firmware B05 vs. B15? Even if it means something was added/adjusted that might be able to be corrected for in the iPXE BCM driver handling? Of course this would lead down a road that would essentially make the Mac's only usable on an 'ipxe' created driver system (Leaving out SNP/SNPONLY I suppse?)

Just thoughts in my head. I don't expect we'll see Apple pushing out a fix for this because of iPXE not working, but maybe we can get what was changed so the Opensource community might be able to provide a work around.

I agree that Apple will probably not fix this (or at least not in the near future), but I have no idea how and where to get any information on what was changed in the firmware...

I checked the Sierra installer and indeed there are 2 install packages there that refer to firmware (EmbeddedOsFirmware.pkg and FirmwareUpdate.pkg); unfortunately, both are .pkg files of the newer type (single binaries, not directories) so there is no possibility to look into them.
Additionally, I checked if the new firmware version for the Late 2015 iMac is available as a download, but it isn't (and neither is the older on that is working).

I have absolutely no experience with Boot ROMs and firmware, so I'm out of ideas what we could try. Anyone else? I have enough (different) Macs here that I can use for testing if anybody has any suggestions on WHAT to test.