A disk read error occurred (under load)

[ahhmyface]

Hi. First post on this group. Thanks to everyone involved in this project!

I don't actually use iPXE, I'm using gPXE, but I expect the differences may be inconsequential for my purposes. (At this time I am restricted to gPXE).

I have a boot server with several iSCSI targets booting a number of VMs with embedded grub2/gPXE linux kernels. Most of the time everything works great.

My problem is that when the boot server is under heavy load (eg. all the VMs are booting at once), or starved for resources from the environment (eg. I limit its CPU cycles), several of the VMs fail to boot with the error message: "A disk read error occurred Press Ctrl+Alt+Del to restart"

I've tried the various "retry on failed boot" options in vmware, but they don't appear to detect this. I've also tried fallback options in grub, but the failure happens after grub has handed off control to gPXE. Restarting the VMs enough times will eventually lead to a successful boot.

I'm looking for mitigation strategies. Either a retry, or a increase to the timeout, or something else that can prevent the error entirely. Most of this stuff is automated, and its a major timesink to check each vm's console and restart it if it has failed. I'm comfortable editing source, if necessary, but I need someone to point me in the right direction.

Thanks in advance.