FTP protocol problem

[iosif_matei]

Hi everyone ,

I did buy last week a router from assus ( RT-N18U) . Router have gigabit lan and 2 USB ports ( one usb 3.0 port and one usb 2.0 port) . So I conected a usb 3.0 stick to router and configured FTP server so i can acces those files from stick via FTP . I took latest ipxe source from github and recompiled to include FTP suport .I did also embeded a script that contain 2 lines :

dhcp
chain http://ftp.user:password@192.168.2.1/sda/ipxe/menu.ipxe

Then I uploaded the undionly.kpxe to router and configured dnsmask.conf.
FTP server on the router is "" vsftpd - 2.3.4-1 "" .

Now when I started the client to boot this is what happend :

1 . Client did request a IP and boot file from dnsmask .
2. Dnsmask received the request and alocated a dynamic IP to client and give the boot file name .
3. client downloaded the undionly.kpxe file from TFTP and released control to undionly .kpxe
4. undionly.kpxe detected the network card , initialized and then launched the commands from the EMBEDed script .
5 .Client requested IP again from dhcp and then downloaded the menu.ipxe from the ftp server.
6.Menu launched succesfully and downloaded ipxe.png from same ftp server.
7.Then I launched tinycore linux that use next code in the menu.ipxe :

:tinycore
echo Booting Tinycore (With Gui)
set url http://ftp.user:password@192.168.2.1/sda...es/liveiso
initrd ${url}/tinycore/tinycore.iso
chain ${url}/tinycore/memdisk iso || goto failed
goto start

When the ISO image need to be downloaded from ftp server somthing very strange happend . The nr at the end of line that indicate how much % of the file is downloaded in real time go in about 0.001 seconds from 0% to 100% and then everything freez on screen .The file is never downloaded and other lines never executed as a result of this behavior . If i press CTRL+C to abort the download that work and i got an error that is no OS to load and i need to reboot . In order to figure out if ipxe is doing something or is completly frozen , I did started another ftp client and start download something from same ftp server .I noticed that download speed droped from 50 mb/sec to 25% mb/sec on the second download. Its like ipxe is practically using the remaining 25 mb/sec , trying desperatly to download that ISO file but never succeding. I did tested like 15 diff FTP clients with the ftp server and all 15 working fine except ipxe .
I did atached a print screen with the problem below .

[NiKiZe]

using memdisk to load a iso with linux on is less then suboptimal - instead you should extract the kernel and intrd etc and load it directly - one of the obvious reasons for this is that the iso contains several files and data that you don't need or want so it is just unnecessary download, an alternative method would be to use sanboot instead, but not sure if that works with ftp.

The reason that ctrl-c exit is that you are running a script, when that script compleats the next step is to exit ipxe - you might want to test without having any embeded script - and press ctrl-b on boot and then type in those commands manually instead to see exactly where it fails and also having the ability to press ctrl-c without it exiting ipxe

[iosif_matei]

(2016-12-25 19:47)NiKiZe Wrote:  using memdisk to load a iso with linux on is less then suboptimal - instead you should extract the kernel and intrd etc and load it directly - one of the obvious reasons for this is that the iso contains several files and data that you don't need or want so it is just unnecessary download, an alternative method would be to use sanboot instead, but not sure if that works with ftp.

The reason that ctrl-c exit is that you are running a script, when that script compleats the next step is to exit ipxe - you might want to test without having any embeded script - and press ctrl-b on boot and then type in those commands manually instead to see exactly where it fails and also having the ability to press ctrl-c without it exiting ipxe

I have an update . I did let client trying to boot and did go out in town to buys some groceries from the store .When I did come back ....15 min later to my surprize , client eventually booted in tinycore .That mean that the client eventually managed to download the iso file from the ftp server . So I did repeat the boot process in order to isolate the problem . So this is what happend .To download the tinycore ISo file it take arround 5 min . But that does not make any sense .And here is why . With all other FTP clients it takes around 20 seconds to download a 1 Gb iso file from ftp server . that mean that average read speed from ftp server is around 50 MB/sec . From the previous tests we know that ipxe actually use 50% of that speed (25 mb/sec ) while it seems to be frozen . The size of the tinycore iso file is only 24 MB . What happen is during those 5 min its downloading constantly from ftp server 25 mb of data every second . So in 5 min is actually downloading 300 seconds x 25 mb = 7500 Mb of data . The obvious questions here are :
1. Why ipxe need to download 7+ GB of data for a 24 mb iso file ???
2. Why the % counter indicating the progress of download in real time go from 0% to 100% almost instantly .....despite the fact that the iso file is not succesfully downloaded yet ?

[NiKiZe]

Sounds like a lot of assumptions (quite sure ipxe does not download that 24MB file multiple times unless there is a transfer error - there is something else going on), to actually know what happens you should probably do a tcpdump/capture as close to the ftp-server as possible


by building ipxe with debug flags enabled you could get some more info on client side as well
make bin/undionly.kpxe DEBUG=ftp

However you might need some more flags to get enough info about the transfer to be useful. However together with a TCP dump for the same sessions it should be enugh to figure out what is going on.

[iosif_matei]

(2016-12-27 15:24)NiKiZe Wrote:  Sounds like a lot of assumptions (quite sure ipxe does not download that 24MB file multiple times unless there is a transfer error - there is something else going on), to actually know what happens you should probably do a tcpdump/capture as close to the ftp-server as possible


by building ipxe with debug flags enabled you could get some more info on client side as well
make bin/undionly.kpxe DEBUG=ftp

However you might need some more flags to get enough info about the transfer to be useful. However together with a TCP dump for the same sessions it should be enugh to figure out what is going on.

I did recompiled ipxe with DEBUG=ftp flag and here is the result :

FTP 0x27794 fetching /sda/ipxe/modules/liveiso/tinycore/tinycore.iso
http://ftp.admin@***@192.168.2.1//sda/ip...nycore.iso
FTP 0x27794 received status 220
FTP 0x27794 sending USER admin
FTP 0x27794 received status 331
FTP 0x27794 sending PASS ***
FTP 0x27794 received status 230
FTP 0x27794 sending TYPE I
FTP 0x27794 received status 200
FTP 0x27794 sending SIZE /sda/ipxe/modules/liveiso/tinycore/tinycore.iso
FTP 0x27794 received status 550
FTP 0x27794 sending PASV
FTP 0x27794 received status 227
FTP 0x27794 sending RETR /sda/ipxe/modules/liveiso/tinycore/tinycore.iso
FTP 0x27794 received status 150
100%

After all this debug info is displayed i need to wait 5 min and tinycore boot .

[NiKiZe]

(2016-12-27 19:08)iosif_matei Wrote:  FTP 0x27794 received status 200
FTP 0x27794 sending SIZE /sda/ipxe/modules/liveiso/tinycore/tinycore.iso
FTP 0x27794 received status 550
FTP 0x27794 sending PASV
FTP 0x27794 received status 227
FTP 0x27794 sending RETR /sda/ipxe/modules/liveiso/tinycore/tinycore.iso
FTP 0x27794 received status 150
100%

After all this debug info is displayed i need to wait 5 min and tinycore boot .

I would have expected you to see a FTP close as well before seeing the boot.
Could you try the same thing with HTTP?

Looking on those FTP commands you have SIZE with a 550 response, no size means that ipxe can't pre-allocate a buffer to fit the file so it will need to remap the local memory for each packet received, this might very well be the issue causing the delay.

I would have tried with http or a different FTP server implementation that supports the size command to compare the speed.

[iosif_matei]

(2016-12-27 19:31)NiKiZe Wrote:  

(2016-12-27 19:08)iosif_matei Wrote:  FTP 0x27794 received status 200
FTP 0x27794 sending SIZE /sda/ipxe/modules/liveiso/tinycore/tinycore.iso
FTP 0x27794 received status 550
FTP 0x27794 sending PASV
FTP 0x27794 received status 227
FTP 0x27794 sending RETR /sda/ipxe/modules/liveiso/tinycore/tinycore.iso
FTP 0x27794 received status 150
100%

After all this debug info is displayed i need to wait 5 min and tinycore boot .

I would have expected you to see a FTP close as well before seeing the boot.
Could you try the same thing with HTTP?

Looking on those FTP commands you have SIZE with a 550 response, no size means that ipxe can't pre-allocate a buffer to fit the file so it will need to remap the local memory for each packet received, this might very well be the issue causing the delay.

I would have tried with http or a different FTP server implementation that supports the size command to compare the speed.

I did try boot with another router that have OpenWRT installed and I noticed that instead of """status 550"" the openwrt ftp replyed with ""status 213 "" and the file size . Maybe something is wrong in the vsftpd.conf file .
Here is the content of vsftpd.conf :

anonymous_enable=NO
nopriv_user=root
write_enable=YES
local_enable=YES
chroot_local_user=YES
local_umask=000
dirmessage_enable=NO
xferlog_enable=NO
syslog_enable=NO
connect_from_port_20=YES
use_localtime=YES
listen=YES
pasv_enable=YES
ssl_enable=NO
tcp_wrappers=NO
max_clients=9
ftp_username=anonymous
ftpd_banner=Welcome to ASUS RT-N18U FTP service.

[NiKiZe]

(2016-12-27 19:40)iosif_matei Wrote:  I did try boot with another router that have OpenWRT installed and I noticed that instead of """status 550"" the openwrt ftp replyed with ""status 213 "" and the file size .

I guess it worked just fine then with that other router?

I would guess that the SIZE command is not supported at all in your ftp server, searching for info about the issue and vsftpd the only issue i can find is in regards to a possible DDoS attack in ASCII mode which could be "disabled" by adding ascii_download_enable=YES however we already know that TYPE I is sent before SIZE meaning that binary mode is already enabled.

Are you absolutely sure your RT-N18U router uses vsftpd? and if so which version is it using?

[iosif_matei]

(2016-12-27 20:08)NiKiZe Wrote:  

(2016-12-27 19:40)iosif_matei Wrote:  I did try boot with another router that have OpenWRT installed and I noticed that instead of """status 550"" the openwrt ftp replyed with ""status 213 "" and the file size .

I guess it worked just fine then with that other router?

I would guess that the SIZE command is not supported at all in your ftp server, searching for info about the issue and vsftpd the only issue i can find is in regards to a possible DDoS attack in ASCII mode which could be "disabled" by adding ascii_download_enable=YES however we already know that TYPE I is sent before SIZE meaning that binary mode is already enabled.

Are you absolutely sure your RT-N18U router uses vsftpd? and if so which version is it using?

FTP server on the router is "" vsftpd - 2.3.4-1 "" .

[NiKiZe]

Please tell us (and anyone that might read this in the future due to possible the same issue)
Did it work as expected with a different router? (The one where you got a response to the SIZE FTP command?)

[NiKiZe]

So reading the source "and ignoring possible security issues with this version of vsftpd " http://vsftpd.sourcearchive.com/document...ource.html in the handle_size function might have some issue with reading the size from file on fs it detects it as "file not found" might be some kind of permission issue or just a bug in that old version.

[iosif_matei]

(2016-12-27 23:22)NiKiZe Wrote:  So reading the source "and ignoring possible security issues with this version of vsftpd " http://vsftpd.sourcearchive.com/document...ource.html in the handle_size function might have some issue with reading the size from file on fs it detects it as "file not found" might be some kind of permission issue or just a bug in that old version.

On the OpenWRT router there is same vsftp server but diff version : 3.0.2 and its working perfectly and here is the vsftpd.conf file :

background=YES
listen=YES
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
check_shell=NO
#dirmessage_enable=YES
#ftpd_banner=Welcome to blah FTP service.
session_support=NO
#syslog_enable=YES
#userlist_enable=YES
#userlist_deny=NO
#userlist_file=/etc/vsftpd.users
#xferlog_enable=YES
#xferlog_file=/var/log/vsftpd.log
#xferlog_std_format=YES
###
### TLS/SSL options
### example key generation: openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/config/vsftpd_privkey.pem -out /etc/config/vsftpd_cert.pem -subj /C="DE"/ST="Saxony"/L="Leipzig"/CN="OpenWrt"
#ssl_enable=YES
#allow_anon_ssl=NO
#force_local_data_ssl=NO
#force_local_logins_ssl=NO
#ssl_tlsv1=YES
#ssl_sslv2=NO
#ssl_sslv3=NO
#rsa_cert_file=/etc/config/vsftpd_cert.pem
#rsa_private_key_file=/etc/config/vsftpd_privkey.pem

I noticed the umask seting is diff .But I dont understand how that setting work .

[NiKiZe]

the vsftpd is broken in one way or other.
Maybe you can enter the path to size some other way and it might work, but don't bother just update vsftpd or switch to a http server instead.

You should probably take this issue to the openwrt/Asus forum instead.

[iosif_matei]

(2016-12-29 12:29)NiKiZe Wrote:  the vsftpd is broken in one way or other.
Maybe you can enter the path to size some other way and it might work, but don't bother just update vsftpd or switch to a http server instead.

You should probably take this issue to the openwrt/Asus forum instead.

I did installed a diff FTP server on router from the optware repository . Server is bftpd and is working fine but ....download speed droped from 50 mb/sec to 20 mb/sec .
I did put a request to assus suport to update vsftpd to latest version in next firmware . Let's hope they will do that .

Thanks for help NiKiZe .