Boot ESXi installer and install to iSCSI

[nry]

Pulling my hair out with this one a little. As previously I got this up and running in under an hour with little to no hassle.

Setup PXE on pfSense to chainload iPXE with the following command on a script on a webserver...

sanhook --drive 0x91 iscsi:10.0.0.30:::5:iqn.2013-04.home.nas:esxi4
sanboot --no-describe iscsi:10.0.0.30:::1:iqn.2013-04.home.nas:installer

Worked perfectly!

Skip forward a month I'm starting from fresh, slightly different this time though as I am now using VLANs and the network port to the host is a trunk port so ESXi has access to multiple VLANs (Tried to PXE chain load using native VLAN but just couldn't get DHCP to find an IP so gave up).
Built a bootable USB image with VLAN support enabled using the below script and seems to start the VLAN fine, accesses the web scripts no problem but I am having an issue with the booting of the ESXi 5.1 U1 image as shown in the image below.

What am I doing wrong? Must have missed something somewhere

Compiled script on the USB key taken from https://gist.github.com/robinsmidsrod/2234639 and slightly modified

Code:

#!ipxe

# Setup the VLANs for my network
vcreate --tag 40 net0
dhcp net0-40

set boot-url http://10.0.5.10/pxe
set bootstrap-url ${boot-url}/bootstrap
set menu-url ${boot-url}/menu.ipxe

# Boot <boot-url>/bootstrap/hostname/<hostname>
# if hostname DHCP variable is set and script is present
isset ${hostname} && chain --replace --autofree ${bootstrap-url}/hostname/${hostname} && exit ||

# Boot <boot-url>/bootstrap/uuid/<UUID>
# if uuid SMBIOS variable is set and script is present
isset ${uuid} && chain --replace --autofree ${bootstrap-url}/uuid/${uuid} && exit ||

# Boot <boot-url>/bootstrap/mac/01-02-03-04-05-06 if script is present
chain --replace --autofree ${bootstrap-url}/mac/${mac:hexhyp} && exit ||

# Boot <boot-url>/bootstrap/bus/01-80-86-10-0e if PCI Intel adapter
# is present and script is present
chain --replace --autofree ${bootstrap-url}/bus/${busid:hexhyp} && exit ||

# Boot <boot-url>/menu.ipxe script if all other options have been exhausted
chain --replace --autofree ${menu-url}

Code inside
pxe/bootstrap/mac/00-24-21-1d-35-e8

Code:

#!ipxe

sanboot --drive 0x90 iscsi:10.0.5.10:::2:iqn.2013-05.home.server:n6esxi
sanboot --drive 0x91 --no-describe iscsi:10.0.5.10:::3:iqn.2013-05.home.server:installer

I have also tried

Code:

#!ipxe

sanboot --drive 0x90 iscsi:10.0.5.10:::2:iqn.2013-05.home.server:n6esxi
sanboot --drive 0x91 --no-describe http://10.0.5.10/pxe/esxiinstaller.iso

Both which result in the below



I have also tried using memdisk using

Code:

#!ipxe

sanboot --drive 0x90 iscsi:10.0.5.10:::2:iqn.2013-05.home.server:n6esxi
initrd http://10.0.5.10/pxe/esxiinstaller.iso
chain http://10.0.5.10/pxe/memdisk iso

But this just resulted with it loading and placing many ... on the screen, gave up after leaving it for 10 minutes.

Any help/suggestions would be much appreciated.

[robinsmidsrod]

What version of ESXi are you using on what hardware? As you can see from the gist I wrote (the one you linked to) I was using 5.0, which did work. I believe I've tried 5.1, and I did also get into some trouble (can't recall exactly what it was).

By the way, why are you using 0x90 and 0x91? Normally the first boot drive is 0x80?

In the last code example (the one using memdisk), have you tried replacing sanboot with sanhook and not use the --drive parameter? That should hook the drive (put it in iBFT), but not try to boot it.

From what I can recall, ESXi doesn't actually use the iBFT during install, but I believe it does during regular boot. Have you tried just booting the ESXi installer, install to the iSCSI volume, and then just do a plain "sanboot uri-for-esxi-target"?

[nry]

Using ESXi 5.1 U1
Hardware is some generic old MSI AMD board, AMD 7750, 4GB DDR2 RAM, 512MB USB Boot disk.

I did start out with 0x80 and 0x81, but was just trying to change things which could have been causing an issue.

The following results in the below image...

Code:

#!ipxe

sanhook iscsi:10.0.5.10:::2:iqn.2013-05.home.server:n6esxi
initrd http://10.0.5.10/pxe/vmware/esxi.iso
chain http://10.0.5.10/pxe/memdisk iso

[mcb30]

Which build of iPXE are you using? If it's undionly.kpxe or similar, then you may be unable to use VLANs; it's possible that the underlying UNDI driver configures the NIC for a maximum frame size of 1514 bytes (rather than the 1518 bytes required for the VLAN tag).

Non-maximum-length packets (such as for DHCP and for the small HTTP script download) wouldn't be affected, but you would fail to receive as soon as you attempted a large download. This matches the behaviour that you are seeing.

If this is the problem, then your only option is to use a native iPXE driver instead of undionly.

Michael

[nry]

Ah, I made a new usb boot stick which was unaware of any VLANs and kept everything on one VLAN. All installed in under 5 minutes!

To build ipxe I simply did...
git clone git://git.ipxe.org/ipxe.git
cd ipxe/src
Added script.ipxe
make bin/ipxe.usb EMBED=script.ipxe

Not too sure what you mean by using native iPXE driver instead of undionly?

[nry]

Just thought I'd post an update, I seem to have it all working now besides a few issues with ESXi not saving configs.

I added a new PXE VLAN, set DHCP on this and setup network booting of iPXE with the following script

Code:

#!ipxe

set boot-url http://10.0.5.10/pxe
set bootstrap-url ${boot-url}/bootstrap
set menu-url ${boot-url}/menu.ipxe

# Boot <boot-url>/bootstrap/hostname/<hostname>
# if hostname DHCP variable is set and script is present
isset ${hostname} && chain --replace --autofree ${bootstrap-url}/hostname/${hostname} && exit ||

# Boot <boot-url>/bootstrap/uuid/<UUID>
# if uuid SMBIOS variable is set and script is present
isset ${uuid} && chain --replace --autofree ${bootstrap-url}/uuid/${uuid} && exit ||

# Boot <boot-url>/bootstrap/mac/01-02-03-04-05-06 if script is present
chain --replace --autofree ${bootstrap-url}/mac/${mac:hexhyp} && exit ||

# Boot <boot-url>/bootstrap/bus/01-80-86-10-0e if PCI Intel adapter
# is present and script is present
chain --replace --autofree ${bootstrap-url}/bus/${busid:hexhyp} && exit ||

# Boot <boot-url>/menu.ipxe script if all other options have been exhausted
chain --replace --autofree ${menu-url}

On the switch I set the normal VLANs to tagged on the network port, then my PXE VLAN as untagged.
DHCP picks up an IP no problem, then I can boot the iSCSI volume no issue

[trentster]

Hi there.

I see you mentioned you got ipxe chainloading to work with pfsense. I am battling with this and wondering if you could share your setup on the pfsense side. Can this be done natively with within the pfsense Ui or did you ahve to serve a script?

Could you paste your config and/or a screenshot of the setting you are using in the pfsense UI to get it to work. I am on latest version of pfsense 2.1

Thanks so much

Mark

(2013-05-20 23:42)nry Wrote:  Pulling my hair out with this one a little. As previously I got this up and running in under an hour with little to no hassle.

Setup PXE on pfSense to chainload iPXE with the following command on a script on a webserver...

sanhook --drive 0x91 iscsi:10.0.0.30:::5:iqn.2013-04.home.nas:esxi4
sanboot --no-describe iscsi:10.0.0.30:::1:iqn.2013-04.home.nas:installer

Worked perfectly!

[robinsmidsrod]

trentster: I would suggest you create a new thread on how to get iPXE working with pfSense. Asking another question far down in a thread rarely gives responses.