As regular readers of this blog will know, my repository has been having growing-pains issues, recently.
First, my hosting provider (DreamHost) informed me that my main site - the one that only served my repository’s XML file - was using too many server resources (specifically, too many connections per second), which was severely impacting the websites of other customers on the same shared hosting server. They complained to DreamHost, who then complained to me. They told me I had to reduce traffic to my site and added a “Deny from All” into my webserver configuration, which essentially prevented anyone from refreshing my site, in Installer.app. I couldn’t leave it like that, so I removed that from my configuration and began looking for ways to reduce the traffic. Nate True suggested the free, global caching system, CoralCDN. I changed my repository’s location URL to point to the caches and Nullriver updated the Community Sources package to include the change. If everyone had updated their CS package, traffic to the site would have dropped to virtually nothing. Vast numbers of people have not done so, however. In spite of that, traffic to the site was reduced by over 50% and DreamHost did not shut down my site again. Unfortunately, CoralCDN’s caching system, while nice, isn’t without it’s flaws and many users continue to have issues refreshing or, in some cases, even seeing, my site. More on that, in a bit.
Secondly, a few days ago, DreamHost informed me that my other account with them - the one that had been serving approximately 50% of the package zip files - was causing the same problems, on that server, as the other site was previously. They didn’t, however, give me any advance warning and they throttled the site in such a way that I can’t un-throttle it, effectively taking the site mostly down - only a few downloads were succeeding while the vast majority of the thousands and thousands of download attempts failed. Over the next few days, I gradually kept shifting most of the traffic to a server that Nate True had been letting me use. It had previously been handling around 25% of the traffic and as of last night it was handling 75% of it. The remaining 25% of the traffic was being handled by a server that the IRC user “extremis” (the guy who used to maintain all those PXL packages for everyone) let me use, and I had - at least for the time being - abandoned my DreamHost site. So, as of last evening, everyone should have been able to download zips from me again, assuming they didn’t run into the refreshing issue, mentioned above. I couldn’t, however, keep letting Nate’s server take a pounding (nor would his admin let me).
Various people came forward with offers of assistance (and I will be writing each of you a ‘thank you’ email, as soon as the dust settles - your offers meant a lot and were most appreciated). One company offered to pay for a dedicated server box - no shared hosting, no VPS (Virtual Private Server - a better form of shared hosting), but a box with no one else on it to complain that their site is being impacted by my repository. My site can use every resource the box has to offer, right up to its physical limitations. It’s not cheap, either. They ordered the server and the hosting company for it - FDCservers - had it ready for me late last night. There were some initial hiccups - the server went down while I was looking it over and so they replaced all the RAM. Later, it went down again, so they simply yanked the box and set up a new one for me. This box seems to be quite healthy. After beating my head against a wall for a bit, trying to figure out the cPanel configuration tool (I’ve decided I loathe it), I got a site created on the box, copied all the zips to it and at 3:38 am, Eastern, it started serving zip files all of you. As I write this, the server is now serving all of the zips to everyone now, even people who may not have refreshed and picked up the change, as I configured the sites that previously served the zips with permanent redirects to this server.
I have not moved the serving of the repository’s XML file to this server. The server is handling the load well (though it might need more RAM), but adding the XML to it will add an immediate 500,000+ connections per day to the box. I’m not convinced that this box can handle that and still leave enough system overhead for the inevitable increase in the number of refreshes and package downloads that is always occurring. Plus, should I decide to start using per-connection PHP scripts again, it would consume CPU and memory that I’m not sure I want to take away from this server. So, for the moment, the serving of the XML file is still mostly being handled by the imperfect CoralCDN system and my original site with DreamHost. Personally, I think the wisest move would be to take someone else up on their offer of help and get another dedicated box. That also has the advantage of my not having all of my eggs in one basket. So, I’m still looking for another sponsor, if anyone is feeling generous.
Since I don’t think it would be fair to announce my current sponsor at the bottom of this long post, where many might not see it, please see the announcement of it in my next post. :)
-ste
