~1 million Creative Commons images on 500px.com are dissapearing tomorrow!
If you have the resources please install the Archiveteam's Warrior program and select the 500px project! https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior
irc is #500pieces on efnet
thank you (boosts very appreciated)
@eq
I just turned my 4 warriors with 4 concurrent items each to the 500px project. Hope we get it in time.
What a bunch of suckers.
@luricaun I believe right now doing 1 concurrent item per warrior might be more efficient because of rate limiting.
@eq
Thanks for the advice. I was wondering if there is a way to speed things up. Any suggestions? I will lower the concurrent items for now then.
@luricaun Well, the ratelimiting is around 1 request per second per IP, so you should run each warrior on its own IP if you aren't already. Basically at this point that's all you can do, I think.
@luricaun Unfortunately even if you did, apparently this site doesn't have an ipv6 address to use.
Thank you for setting the warrior up!
@eq Unfortunately I'm running those puppies at home and I don't have IPv6 yet. So there is no chance to serve from different IPs for me.
Anyhow thanks for the tip and let's safe a little piece of the web, shall we?