[haiku-files.org] Slow download speed

[deejam]

Sometimes I experience extreme slow download speeds from haiku-files.org. At the moment I am downloading the latest r1a3-rc build at ~8kB/s. I've experienced slow speeds at 20-30kB/s several times recently. Normally, haiku-files.org download speeds are at least 200kB/s.

Therese nothing wrong with my connection (~130Mbit). Also tested with a 3G connection with the same low download speed (the 3G modem is an old 3.6Mbit modem, but it usually maxes out at 250-300kB/s) and with a 8Mbit ADSL.

I am located in Sweden.

[deejam]

Now (~2 hours later) the download speed is ~100kB/s.

[umccullough]

haiku-files.org is hosted for free on Dreamhost (it's a complimentary account for non-profits).

Performance issues are out of our hands, and we really have no leverage to complain. One might say we "get what we pay for" in that situation.

[deejam]

How about adding torrent support and creating a RSS feed of haiku-files.org? In that way, users can have the torrents downloaded automatically using their torrent client and then help seeding.

[umccullough]

Creating torrents for nightlies with an RSS feed was actually considered at one point... but it wasn't deemed important enough to implement.

We do create torrents for our official releases, of course.

FWIW, there are RSS feeds already on haiku-files.org...

I might look into the torrent creation again - using the haiku-files.org as a web seed would be convenient. It would also serve as an interesting archival strategy so we can eliminate older nightly images while people still seed them if they wish.

[jrabbit]

If we're running low on space we could remove some of the zips and keep the xzs. But yeah an intrepid poweruser could use the rss feed to generate torrents.

[deejam]

Replying to umccullough:

Creating torrents for nightlies with an RSS feed was actually considered at one point... but it wasn't deemed important enough to implement.

This needs to be raised again. Today the download speed is ~10kB/s.

FWIW, there are RSS feeds already on haiku-files.org...

Yes, but there are no torrents in those seeds.

I might look into the torrent creation again - using the haiku-files.org as a web seed would be convenient. It would also serve as an interesting archival strategy so we can eliminate older nightly images while people still seed them if they wish.

It would be great if you could look into it.

[RandomInsano]

I did build a torrent-based Haiku mirror about two years ago. My web server company suffered a RAID5 double-fault and I lost literally everything. I have no idea where the heck I put the code if I still have any of it.

The basic mechanism was this:

• bash script
  • curl + grep + sed to pull down a list of file names from haiku-files
  • Download first 10 files (head ten lines). Ensure no duplicate files were downloaded (usually a flag in the download command)
  • While used space was > 1GB delete oldest master file. Kept disk usage down, never deleted torrents
  • Run PHP script and output to index.html. Avoids repeat processing when no new data exists.
• PHP script
  • Use torrentrw (PHP script!) to generate torrent from downloaded file
    • Grab from here: ​http://www.phpclasses.org/browse/file/24765.html
    • Can set web seeds. Works perfectly for web servers since it's written in PHP
    • Used both myself and the original haiku-files site as web seeds.
  • Output a nice simple web page

Ran that in a cron job every night and dumped the PHP output to a static file. MUCH more efficient than doing on-demand file listing and torrent creation. I highly recommend that paradigm if you have RSS feeds or content that changes rarely.

I worked pretty darn hard on that stupid thing. Back up your files boys and girls. If anyone wants to actually put this thing into action on a web server, let me know and I'll try and recreate it some weekend.

[luroh]

Haven't heard any complaints lately, closing.