Chainloading linux in AWS EC2 HVM

[plachance]

(2016-01-27 14:31)mcb30 Wrote:  Alternatively, you could try reproducing the problem in a local XenServer instance. That would allow you to examine the state using xenstore-ls from dom0, without having to patch the guest kernel/initrd.

It works as expected. I could successfully run the following sequence:
1. create the domain using "builder='hvm'"
2. boot it using ipxe.usb + bootstrap embedded (after dd if=ipxe.usb of=<path_to_lvm_dev>
3. chainload an ipxe.pxe + boot script embedded image from an http server
4. chainload kernel + initramfs and trigger kickstart installation

Xen Platform info

Code:

xen_version            : 4.6.1-1.el7
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64

bootstrap.ipxe

Code:

#!ipxe
dhcp &&
route
chain -ar http://<bootserver>/boot.pxe

boot.ipxe

Code:

#!ipxe
set base http://<bootserver>
set repo mirrors/centos/ftp.centos.org
set ose centos
set ver 7
set arch x86_64

dhcp &&
route

set params ip=dhcp inst.repo=${base}/${repo}/${ver}/os/${arch} inst.ks=${base}/ks/${ose}${ver}  console=hvc0 console=ttyS0 inst.cmdline inst.lang=en_US net.ifnames=0 biosdevname=0 edd=off

kernel ${base}/${ver}/os/${arch}/isolinux/vmlinuz
initrd ${base}/${ver}/os/${arch}/isolinux/initrd.img
imgargs vmlinuz ${params}
boot -ar

Boot sequence output

Code:

# xl create -c <path_to_domu_cfg_file>
Parsing config from <path_to_domu_cfg_file>
iPXE initialising devices...ok



iPXE 1.0.0+ (e2cf) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP HTTPS iSCSI TFTP SRP VLAN AoE ELF MBOOT NBI PXE SDI bzImage COMBOOT Menu PXEXT
Configuring (net0 00:50:56:04:47:4c)...... ok
net0: <ip>/255.255.255.240 gw <gateway>
http://<bootserver>/boot.pxe... ok
iPXE initialising devices...ok



iPXE 1.0.0+ (e2cf) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP HTTPS iSCSI TFTP SRP VLAN AoE ELF MBOOT NBI PXE SDI bzImage COMBOOT Menu PXEXT
Configuring (net0 00:50:56:04:47:4c)...... ok
net0: <ip>/255.255.255.240 gw <gateway>
http://<bootserver>/mirrors/centos/ftp.centos.org/7/os/x86_64/isolinux/vmlinuz... ok
http://<bootserver>/mirrors/centos/ftp.centos.org/7/os/x86_64/isolinux/initrd.img... ok
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-327.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov 19 22:10:57 UTC 2015
:::::::
:::::::

As expected, ipxe stored on disk gets loaded, it can fetch and load a new ipxe.pxe from the bootserver and load the kernel and initramfs.

But using the same ipxe.usb and boot.pxe files on AWS EC2 (region: eu-west-1, instance type t2-medium) doesn't work. Here is the console output from AWS console, trimmed a little bit to remove control codes. The first line (ok) is certainly the output of ipxe.usb and the other lines, from the boot.pxe file downloaded from the bootserver.

Code:

ok
iPXE initialising devices...ok

iPXE 1.0.0+ (e2cf) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP HTTPS iSCSI TFTP SRP VLAN AoE ELF MBOOT NBI PXE SDI bzImage COMBOOT Menu PXEXT
Configuring (net0 06:4c:e3:f2:87:39).................. Error 0x040ee119 (http://ipxe.org/040ee119)

So it seems that ipxe can't chainload itself from the network on AWS.

Other test
A few weeks ago I made another test. I tried to create a chainload loop using the following ipxe script, expecting the sequence to loop infinitely. But it didn't!

Code:

#!ipxe
ifstat
dhcp
ifstat
sanboot --no-describe --drive 0x80 ||
sanboot --no-describe --drive 0x81 ||
exit

Console output

Code:

iPXE 1.0.0+ (3c26) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP SRP AoE ELF MBOOT PXE bzImage Menu PXEXT
Amazon EC2 - iPXE boot via user-data
net0: 0a:dd:a6:cb:d5:b3 using netfront on vif/0 (closed)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Down (http://ipxe.org/38086101)]
Configuring (net0 0a:dd:a6:cb:d5:b3)...... ok
net0: 172.31.47.39/255.255.240.0 gw 172.31.32.1
http://169.254.169.254/latest/user-data... ok
net0: 0a:dd:a6:cb:d5:b3 using netfront on vif/0 (open)
  [Link:up, TX:6 TXE:0 RX:10 RXE:3]
  [RXE: 3 x "Operation not supported (http://ipxe.org/3c086003)"]
Configuring (net0 0a:dd:a6:cb:d5:b3)...... ok
net0: 0a:dd:a6:cb:d5:b3 using netfront on vif/0 (open)
  [Link:up, TX:10 TXE:0 RX:14 RXE:4]
  [RXE: 4 x "Operation not supported (http://ipxe.org/3c086003)"]
Booting from SAN device 0x80
iPXE initialising devices...ok



iPXE 1.0.0+ (3c26) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP SRP AoE ELF MBOOT PXE bzImage Menu PXEXT
Amazon EC2 - iPXE boot via user-data
net0: 0a:dd:a6:cb:d5:b3 using netfront on vif/0 (closed)
  [Link:down, TX:0 TXE:0 RX:0 RXE:0]
  [Link status: Down (http://ipxe.org/38086101)]
Configuring (net0 0a:dd:a6:cb:d5:b3).................. Error 0x040ee119 (http://ipxe.org/040ee119)
http://169.254.169.254/latest/user-data... Network unreachable (http://ipxe.org/280a6011)
Boot from SAN device 0x80 failed: Operation canceled (http://ipxe.org/0b8080a0)
Booting from SAN device 0x81

ipxe load from disk at instance creation, can configure the network and reload itself from 0x80. But then it can't configure the network, fails to boot and (I presume), the first ipxe sequence continue and try to boot from 0x81 which fails because there is only one disk attached to the instance.

It doesn't seem related to kernel/initramfs but specific to AWS.
Can it be related to the Xen version (mine is 4.6.1 and AWS is 4.2) or AWS-specific Xen tuning preventing chainloading?
Do you have a working example of chainloading ipxe from ipxe on AWS?

Thanks again for your great work and your help on this issue!