Openstack Instance paused when booted from ipxe iso

[k_mouza]

Hello all,

I have the following problem when trying to boot an Openstack instance using the ipxe.iso. I downloaded the latest ipxe source from git and recompiled it. I use the ipxe.iso to create a new glance image within Openstack, but when I try to create a VM with it, its Power State changes to "Paused" almost immediately. From the console I see that it's stuck after "Loading ipxe.krn...ready".

This is what I have in the libvirt/qemu log file:

Code:

2017-04-11 14:45:54.490+0000: starting up libvirt version: 2.0.0, package: 10.el7_3.5 (CentOS BuildSystem <http://bugs.centos.org>, 2017-03-03-02:09:45, c1bm.rdu2.centos.org), qemu version: 1.5.3 (qemu-kvm-1.5.3-126.el7_3.5), hostname: compute19.cix.vscaler.cloud
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-000004c9 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu Penryn,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+est,+tm2,+​xtpr,+pdcm,+dca -m 14336 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid db9ab7be-f945-4ffa-a44d-807b64c54118 -smbios 'type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=12.0.4-1.el7,serial=659846d2-7392-4795-afd4-fe0e77f22b81,uuid=db9ab7be-f945-4ffa-a44d-807b64c54118,family=Virtual Machine' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-7-instance-000004c9/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/db9ab7be-f945-4ffa-a44d-807b64c54118/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:74:80:e4,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/db9ab7be-f945-4ffa-a44d-807b64c54118/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:1 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/0 (label charserial1)
KVM internal error. Suberror: 1
emulation failure
EAX=fc20dbc0 EBX=00000130 ECX=00002ef0 EDX=00170d18
ESI=0014f740 EDI=03e8f000 EBP=ffffffff ESP=00007b72
EIP=00000769 EFL=00000087 [--S--PC] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 ffffffff 00c09300
CS =9c2c 0009c2c0 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =9cbc 0009cbc0 ffffffff 00c09300
FS =1000 00010000 ffffffff 00c09300
GS =1000 00010000 ffffffff 00c09300
LDT=0000 00000000 ffffffff 00c00000
TR =0008 00000580 00000067 00008b00
GDT=     00001170 0000002f
IDT=     00000000 0000ffff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=00 16 66 9c 66 60 0f a8 0f a0 06 1e 16 0e fa 2e 8e 1e 4e 07 <0f> ae 06 30 1e 0f 01 0e 26 1e 0f 01 06 20 1e fc 66 b9 38 00 00 00 66 ba 10 02 00 00 66 68

And this is what I see in the nova log file:

Code:

2017-04-11 15:45:59.101 27188 INFO nova.compute.manager [req-f7636b18-c316-4ddb-a6ac-06cba6fc89cc - - - - -] [instance: db9ab7be-f945-4ffa-a44d-807b64c54118] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (3). Updating power_state in the DB to match the hypervisor.
2017-04-11 15:45:59.210 27188 WARNING nova.compute.manager [req-f7636b18-c316-4ddb-a6ac-06cba6fc89cc - - - - -] [instance: db9ab7be-f945-4ffa-a44d-807b64c54118] Instance is paused unexpectedly. Ignore.

Currently running Centos 7.2, kernel 3.10, Openstack Liberty, qemu-kvm 1.5.3-126.

Anyone seen this before? Any help will be much appreciated!

Thanks,
Mouza

[NiKiZe]

I have a box which last synced almost 2 years ago which has app-emulation/qemu-2.0.0-r1 installed... How old is qemu-kvm 1.5.3-126 ?

You might want to minimize that commandline so it is easier (for others) to reproduce.

Also if you know of a working iPXE commit use git bisect to identify when and how the issue was introduced.

There have been some updates to virtio a while back which might have regressions for really old versions of qemu-kvm (this is just a guess).

Development and testing is mostly done with Qemu so the basic stuff is known working for more recent versions