Post Reply 
 
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
No space left on device (http://ipxe.org/34182006)
2018-07-02, 12:05
Post: #1
No space left on device (http://ipxe.org/34182006)
Hey guys,

I'm new in the world of ipxe and was pushed into this topic by a colleague leaving, so please be gentle.

I'm dealing with an issue right now that only appears on on newer machines.
We are running many HPE G7 nodes and one HPE G9 node.

The G7 nodes boot up just fine, but the G9 node runs into an error as soon as it starts to load the initramfs.

Quote:nfs://... No space left on Device (http://ipxe.org/34182006)

My IPXE script loos the following:
Code:
#!ipxe

:retry_dhcp
echo Configuring network with DHCP...
dhcp || goto retry_dhcp
kernel nfs://10.131.33.101/srv/exports/boot-images/image-deployed/vmlinuz ramdisk_size=8388608 root=/dev/ram0 edd=off HOSTNAME=${hostname} intel_pstate=disable intel_iommu=on
initrd nfs://10.131.33.101/srv/exports/boot-images/image-deployed/initramfs.img

imgargs vmlinuz ramdisk_size=8388608 root=/dev/ram0 edd=off HOSTNAME=${hostname} intel_pstate=disable intel_iommu=on user_namespace.enable=1 namespace.unpriv_enable=1

boot || echo Booting via NFS failed!
echo In case of iPXE error 0x2a818006: Make sure the initramfs ramdisk_size is large enough to fit the boot image.

All nodes are equipped with a minimum of 96GB of RAM and boot in legacy boot mode, as the old G7 nodes don't support any UEFI boot.

I hope we can figure out whats wrong
Find all posts by this user
Quote this message in a reply
2018-07-02, 20:16
Post: #2
RE: No space left on device (http://ipxe.org/34182006)
Which of the nfs:// chain lines (kernel / initrd) is it that fails?
iPXE can be built in a few different ways, and for different platforms, so it would be relevant to include the filename of the iPXE binary, or how you built it.
I'm guessing that you are using ipxe.lkrn or undionly.kpxe ? (and that it is built for 32 bit and not 64 bit)

please note that imgargs will replace what was given on the kernel line, you might want to change that and only use arguments on kernel or imgargs

do you also have the filesize of the 2 files being downloaded?

Use GitHub Discussions
VRAM bin
Visit this user's website Find all posts by this user
Quote this message in a reply
2018-07-02, 20:47
Post: #3
RE: No space left on device (http://ipxe.org/34182006)
Thank you for your fast reply.
The file that breaks is the initramfs. The filename is undionly.kpxe and it is basically build with this command:
Code:
make bin/undionly.kpxe EMBED=../../boot-datev-node-centos-from-datev01-via-nfs.ipxe
As far as I can tell, there haven't been any changes applied to the Makefile.

I will remove the arguments from the kernel command line, but as all these commands are included in the imgargs line as well, I don't expect any changes.

The initramfs.img file is 1.7GB in size, the kernel is only 6MB big.

We are handing over the undionly.kpxe by out tftp server.
Find all posts by this user
Quote this message in a reply
2018-07-02, 20:57
Post: #4
RE: No space left on device (http://ipxe.org/34182006)
So it seams that you are simply running out of (available) RAM here,
I would say that what you are seeing is similar if not the same as described at http://lists.ipxe.org/pipermail/ipxe-dev...04793.html

To verify what is going on you could build your undionly.kpxe with DEBUG=memmap added to the make line
that should print out the memory map of the system and hopefully which segment it is trying to use.

There is also the option of building 64 bit images for pcbios, I would not recommend this unless you really need it, but might be worth a try:
Code:
make bin-x86_64-pcbios/undionly.kpxe

It would any case be interesting if you could post the full output when enabling the memmap debug, and maybe also if there is any difference between 32bit and 64bit builds of iPXE

Use GitHub Discussions
VRAM bin
Visit this user's website Find all posts by this user
Quote this message in a reply
2018-07-04, 21:17
Post: #5
RE: No space left on device (http://ipxe.org/34182006)
Hey,

sadly, the 64-bit version did not help.

Here is the debug output:
Code:
CLIENT MAC ADDR: 94 18 82 0A FC 1C  GUID: 30393137-3436-5A43-4A36-333836573946                                                                                                                              
CLIENT IP: 10.131.33.131  MASK: 255.255.255.0  DHCP IP: 10.131.33.101                                                                                                                                      
GATEWAY IP: 10.131.33.1                                                                                                                                                                                    
PXE->EB: !PXE at 9861:0040, entry point at 9861:00D6                                                                                                                                                        
         UNDI code segment 9861:6B60, data segment 94AA:3B70 (594-637kB)                                                                                                                                    
         UNDI device is PCI 02:00.0, type DIX+802.3                                                                                                                                                        
         594kB free base memory after PXE unload                                                                                                                                                            
Fetching system memory map                                                                                                                                                                                  
FBMS base memory size 579 kB [0,90c00)                                                                                                                                                                      
INT 15,88 extended memory size 15360 kB [100000, 1000000)                                                                                                                                                  
INT 15,e801 extended memory size 15360+64*32512=2096128 kB [100000,80000000)                                                                                                                                
INT 15,e820 region [0,94000) type 1 (enabled)                                                                                                                                                              
INT 15,e820 region [94000,a0000) type 2 (enabled)                                                                                                                                                          
INT 15,e820 region [e0000,100000) type 2 (enabled)                                                                                                                                                          
INT 15,e820 region [100000,5a661000) type 1 (enabled)                                                                                                                                                      
INT 15,e820 region [5a661000,5b4a1000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [5b4a1000,790ff000) type 1 (enabled)                                                                                                                                                    
INT 15,e820 region [790ff000,791ff000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [791ff000,7b5ff000) type 4 (enabled)                                                                                                                                                    
INT 15,e820 region [7b5ff000,7b7ff000) type 3 (enabled)                                                                                                                                                    
INT 15,e820 region [7b7ff000,7b800000) type 1 (enabled)                                                                                                                                                    
INT 15,e820 region [7b800000,7c000000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [7c000000,7fc00000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [7fc00000,80000000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [80000000,90000000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [ff800000,100000000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [100000000,2080000000) type 1 (enabled)                                                                                                                                                  
Obtained system memory map via INT 15,e820                                                                                                                                                                  
iPXE initialising devices...Fetching system memory map                                                                                                                                                      
FBMS base memory size 579 kB [0,90c00)                                                                                                                                                                      
INT 15,88 extended memory size 15360 kB [100000, 1000000)                                                                                                                                                  
INT 15,e801 extended memory size 15360+64*32512=2096128 kB [100000,80000000)                                                                                                                                
INT 15,e820 region [0,94000) type 1 (enabled)                                                                                                                                                              
INT 15,e820 region [94000,a0000) type 2 (enabled)                                                                                                                                                          
INT 15,e820 region [e0000,100000) type 2 (enabled)                                                                                                                                                          
INT 15,e820 region [100000,5a661000) type 1 (enabled)                                                                                                                                                      
INT 15,e820 region [5a661000,5b4a1000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [5b4a1000,790ff000) type 1 (enabled)                                                                                                                                                    
INT 15,e820 region [790ff000,791ff000) type 2 (enabled)                                                                                                                                                    
INT 15,e820 region [791ff000,7b5ff000) type 4 (enabled)                                                                                                                                                    
INT 15,e820 region [7b5ff000,7b7ff000) type 3 (enabled)                                                                                                                                                    
Attempting Boot FromwNIC Boot Firmware -- h 1 (enabled)
gg-- Open Source Network Boot Firmware -- hg execution...ok
Features: VLAN HTTP iSCSI DNS TFTP bzImage COMBOOT ELF MBOOT PXE PXEXT Menu
Initialising completed./exports/boot-images/image-deployed/vmlinuz...Fetching system memory map

INT 15,88 extended memory size 15360 kB [100000, 1000000)
Mellanox FlexBoot v3.4.460y size 15360+64*30725=1981760 kB [100000,79050000)
iNT 15,e820 region [0,90c00) type 1 (enabled)
INT 15,e820 region [e0000,100000) type 2 (enabled)
INT 15,e820 region [100000,5a661000) type 1 (enabled)
INT 15,e820 region [5a661000,5b4a1000) type 2 (enabled)0)
INT 15,e820 region [5b4a1000,7905f000) type 1 (enabled) kB [100000,79050000)
INT 15,e820 region [790ff000,791ff000) type 2 (enabled)
INT 15,e820 region [791ff000,7b5ff000) type 4 (enabled)
INT 15,e820 region [7b5ff000,7b7ff000) type 3 (enabled)
INT 15,e820 region [7b7ff000,7b800000) type 1 (enabled)
INT 15,e820 region [7b800000,7c000000) type 2 (enabled)
INT 15,e820 region [7c000000,7fc00000) type 2 (enabled)
INT 15,e820 region [7fc00000,80000000) type 2 (enabled)
INT 15,e820 region [80000000,90000000) type 2 (enabled)
INT 15,e820 region [ff800000,100000000) type 2 (enabled)
INT 15,e820 region [100000000,2080000000) type 1 (enabled)
Obtained system memory map via INT 15,e820e 2 (enabled)
ok 15,e820 region [7fc00000,80000000) type 2 (enabled)
nfs://10.131.33.101/srv/exports/boot-images/image-deployed/initramfs.img... No space left on device (http://ipxe.org/34182006)
Fetching system memory map00,100000000) type 2 (enabled)
FBMS base memory size 579 kB [0,90c00)00) type 1 (enabled)
INT 15,88 extended memory size 15360 kB [100000, 1000000)
INT 15,e801 extended memory size 15360+64*32512=2096128 kB [100000,80000000)
INT 15,e820 region [0,94000) type 1 (enabled)
INT 15,e820 region [94000,a0000) type 2 (enabled)1000000)
INT 15,e820 region [e0000,100000) type 2 (enabled)81760 kB [100000,79050000)
INT 15,e820 region [100000,5a661000) type 1 (enabled)
INT 15,e820 region [5a661000,5b4a1000) type 2 (enabled)
INT 15,e820 region [5b4a1000,790ff000) type 1 (enabled)
INT 15,e820 region [790ff000,791ff000) type 2 (enabled)
INT 15,e820 region [791ff000,7b5ff000) type 4 (enabled)
INT 15,e820 region [7b5ff000,7b7ff000) type 3 (enabled)
INT 15,e820 region [7b7ff000,7b800000) type 1 (enabled)
INT 15,e820 region [7b800000,7c000000) type 2 (enabled)
INT 15,e820 region [7c000000,7fc00000) type 2 (enabled)
INT 15,e820 region [7fc00000,80000000) type 2 (enabled)
INT 15,e820 region [80000000,90000000) type 2 (enabled)
INT 15,e820 region [ff800000,100000000) type 2 (enabled)
INT 15,e820 region [100000000,2080000000) type 1 (enabled)
Obtained system memory map via INT 15,e820pe 2 (enabled)
INT 15,e820 region [100000000,2080000000) type 1 (enabled)
Obtained system memory map via INT 15,e820
ok

Sadly this output is a little bit messed up, as the Mellanox Flexboot, which starts right after the normal netboot, fucks up the VSP output. I hope the important information is still visible.

I thought about testing UEFI boot, but I need some time to do the research on this, so this will probably happen over the weekend. The nodes failing on the ipxe boot are all UEFI capable.
Find all posts by this user
Quote this message in a reply
2018-07-05, 10:13
Post: #6
RE: No space left on device (http://ipxe.org/34182006)
So, just a short update:

I managed to build and deploy a UEFI version. This works from the ipxe perspective!

But sadly, it does not work for me. We are running a Centos 7 which provides a 3.10 Kernel. This kernel version seems to have a bug on UEFI Ramdisk booting:
https://forum.ipxe.org/printthread.php?tid=8195
So I still need to figure out what is wrong with my IPXE - node combination in bios mode.

But for everyone running into the same problem and using a newer kernel (>3.16) here is a solution which should work:

This is my IPXE script:
Code:
#!ipxe

:retry_dhcp
echo Configuring network with DHCP...
dhcp || goto retry_dhcp
kernel nfs://10.131.33.101/srv/exports/boot-images/image-deployed/vmlinuz  initrd=initram ramdisk_size=8388608 root=/dev/ram0 edd=off HOSTNAME=${hostname} intel_pstate=disable intel_iommu=on user_namespace.enable=1 namespace.unpriv_enable=1

initrd --name initram nfs://10.131.33.101/srv/exports/boot-images/image-deployed/initramfs.img

boot || echo Booting via NFS failed!
echo In case of iPXE error 0x2a818006: Make sure the initramfs ramdisk_size is large enough to fit the boot image.

Note the initrd=<name> option in the kernel line. This is required for a UEFI boot system.

building this image works with:
Code:
make bin-x86_64-efi/ipxe.efi EMBED=../../boot-datev-node-centos-from-datev01-via-nfs-efi.ipxe

I additionally configured my dnsmasq to send the uefi file only to uefi nodes:
Code:
# Test for the architecture of a netboot client. PXE clients are
# supposed to send their architecture as option 93. (See RFC 4578)
dhcp-match=x86PC, option:client-arch, 0 #BIOS x86
dhcp-match=BC_EFI, option:client-arch, 7 #EFI x86-64

# Load different PXE boot image depending on client architecture
pxe-service=tag:x86PC,X86PC, "Install Linux on x86 legacy BIOS", undionly.kpxe
pxe-service=tag:BC_EFI,BC_EFI, "Install Linux on x86-64 UEFI", ipxe.efi


# Set boot file name only when tag is "bios" or "uefi"
dhcp-boot=tag:x86PC,undionly.kpxe  # for Legacy BIOS detected by dhcp-match above
dhcp-boot=tag:BC_EFI,ipxe.efi # for UEFI arch detected by dhcp-match above
Find all posts by this user
Quote this message in a reply
Post Reply 




User(s) browsing this thread: 1 Guest(s)