HP Smart Array issues

[epoch]

Hello All,

First, thanks to all the contributors for making iPXE. My colleague and I have been really happy about all the stuff we've been able to accomplish with that in just a few days (replacing most of our aging pxelinux infrastructure with a really cool iPXE setup...)

BUT, we've hit a snag. we have a couple HP DL360 Gen 8 with a P420i controller in them. pxelinux works fine with that controller, but iPXE seems to write into some memory that the controller uses. once that memory is touched, the controller locks the machine up with the very helpful message:

1706-The Extended BIOS Data Area in Server Memory has been Overwritten - Smart Array Interrupt 13h BIOS Cannot Continue - System Halted

the setup uses pxelinux.0 to load ipxe.lkrn and that's pointing to a http server that feeds it a menu. I get to the menu, but the second i hit a key from the menu, the controller locks the machine up. If i setup ipxe to just chain load something else, i does the same.

I don't believe it's a controller problem per se. I think i need to "disable" something on there to go past that, but i've tried so many things on this box, i no longer know where i should start from here.

What would the best option be to get this debugged? I'm leary about changing too much

[epoch]

(2012-12-04 05:19)epoch Wrote:  1706-The Extended BIOS Data Area in Server Memory has been Overwritten - Smart Array Interrupt 13h BIOS Cannot Continue - System Halted

posting some progress here (not that i'm going anywhere).

I found that i can get a shell (if that's what i feed to my ipxe.lkrn), and type a couple of commands before it bombs. so i'm going to try to get a remote GDB console, and see what i can find out...
(first time debugger user - be gentle!)

[robinsmidsrod]

You should build your iPXE with syslog debugging support (see http://ipxe.org/console) and enable debugging for memory regions (or something like that). You should set the make parameter DEBUG= to something, but I'm honestly not sure what might help to give you more information.

This is obviously a bug/problem that is not just a user doing something wrong. I think you should try to post your question to the mailing-list where more developers hang around.

[epoch]

(2012-12-05 09:38)robinsmidsrod Wrote:  You should build your iPXE with syslog debugging support (see http://ipxe.org/console) and enable debugging for memory regions (or something like that). You should set the make parameter DEBUG= to something, but I'm honestly not sure what might help to give you more information.

I was looking at that earlier. Trying to simplify the problem, i was loading "undionly.kpxe" directly, and that worked flawlessly. So i'm going to be looking at building ipxe.lkrn with console support and loading that with our current setup. My colleague and I are talking about moving away from pxelinux.0 chainloading ipxe to just use ipxe, as it can do everything we want, and more

... and updating our WDS infrastructure to get this working is well documented.

(2012-12-05 09:38)robinsmidsrod Wrote:  This is obviously a bug/problem that is not just a user doing something wrong. I think you should try to post your question to the mailing-list where more developers hang around.

will do. I will also try to keep it in here, so that the next guy can get to it...

[MultimediaMan]

We're running out of BIOS memory to use...I think this problem goes all the way back to DOS/Real Mode; There is only about 64k-640k of low level BIOS RAM available. Each Bootable device with a BIOS to load takes a bite out of this pool of available low level memory, including iPXE.

Some time ago, it wasn't uncommon to see a machine with a number of an add-in cards with BIOS fail to load some add-in BIOS because of these limitations.

Dell and HP are kind of "worst offenders" in this area; they have very complex BIOS boot processes and a number of direct BIOS IO devices (iDRAC/iLO, IPMI, BMC/ Power Managers, Onboard RAID controllers, before even considering Network and Storage HBAs)

Intel-based Network Cards (Pro 1000%FF%, I340, I350, as well as X520 10GbE) have traditionally have had less problems; they also tend to have a much more developed UNDI PXE Stack. Broadcom 1GbE are also fairly trouble free.

Personally, I've run into this problem (chaining from undionly.kpxe to ipxe.pxe) with HP Virtual Fabric (HP 55x series; Emulex and QLogic based cards) and Broadcom 10GbE (HP 53x series) hardware and OEM Broadcom hardware.

I've also run into issues with iPXE retries eventually freezing a machine with the same error: I have compensated by simply putting a reboot in the retry script.

[epoch]

(2012-12-06 00:24)MultimediaMan Wrote:  Personally, I've run into this problem (chaining from undionly.kpxe to ipxe.pxe) with HP Virtual Fabric (HP 55x series; Emulex and QLogic based cards) and Broadcom 10GbE (HP 53x series) hardware and OEM Broadcom hardware.

Our regular infrastructure includes a WDS server that we use for PXE booting. It had been configured to use pxelinux.0 as the PXE payload. we were bootstrapping from that into ipxe.lkrn (roughly 350K). what we found is that in the lab, the same machine can boot from ISC's DHCPD and undionly.kpxe (only 64K). We expect that the difference in size is why it works. We will be converting our menu sets from pxelinux to ipxe menu (which i think looks better )

with that in mind, I'll be rebuilding ipxe.lkrn with some debug code and see if i can track down where the bug is.

[trueshanti]

(2012-12-06 23:43)epoch Wrote:  with that in mind, I'll be rebuilding ipxe.lkrn with some debug code and see if i can track down where the bug is.

Did you finally came to a conclusion/workaround/fix about that issue ?

i am stuck with my ipxe-boot-cd

syslinux4.x -> ipxe-latest -> vmlinuz -> 1706error

reading here about the same issue without using syslinux lets me hope this can be fixed in ipxe

thanks for any updates

best regards

-c-

[UPDATE 02.2014] due to this ugly bug i solved my issue by skipping the idea to boot directly from ipxe .. instead i dumped the ipxe.usb-image to disk and booted from there

[antonym]

I happened to run into this today. It's odd because I have a multi-layered menu. When I select an option to go one more menu deep, the server locks up with a single 1 on the screen. If chain directly to that menu and attempt to start an install, I get the:

1706-Smart Array Controller EBDA Memory Corrupted
INT 13h BIOS Cannot Continue - System Halted

So my guess is that I'm probably pushing the limits of memory one the iPXE image is loaded, I'm going to probably try and remove some options and see how minimal an image I can make.

I'm doing ISOLINUX -> ipxe -> menu -> vmlinuz -> 1706error as well.

[robinsmidsrod]

To be honest, this seems to be a bug somewhere with the memory map. It seems either the Smart Array Controller or iPXE is using memory they have not claimed for use, and one is corrupting the other. Try a BIOS upgrade (on the raid controller as well) and see if it makes any difference.

[antonym]

Yeah, I agree. I am on the latest BIOS revisions that are available and I've seen these issues happening on various HP models. I'm working on getting HP engaged to see if we can try and get this issue resolved once and for all.

Can you think of any debugging data that I could pull that might be helpful for the devs to diagnose on the iPXE side?

[robinsmidsrod]

antonym: Based on these commands

Code:

$ find . -type f -name \*.c| sort | grep mem
./arch/i386/core/basemem_packet.c
./arch/i386/firmware/pcbios/basemem.c
./arch/i386/firmware/pcbios/hidemem.c
./arch/i386/firmware/pcbios/memmap.c
./arch/i386/interface/pcbios/memtop_umalloc.c
./core/memblock.c
./core/memmap_settings.c
./libgcc/memcpy.c
$ find . -type f -name \*.c| sort | grep malloc
./arch/i386/interface/pcbios/memtop_umalloc.c
./core/malloc.c
./interface/efi/efi_umalloc.c

I'd try out maybe this:

For legacy BIOS:
DEBUG=basemem,hidemem,memmap,memtop_umalloc,memblock,memmap_settings,memcpy,memt​op_umalloc,malloc

For EFI builds this might yield some answers:
DEBUG=memblock,memmap_settings,memcpy,malloc,efi_umalloc