Ste Packaging

It’s important that everyone switch to the repository’s new home. That means you must update your “Community Sources” package, so that you can access it.

To force everyone to do so, I have just deliberately broken every one of my old repo sites - only the new one works. If that doesn’t make people switch, nothing will.

-ste

UPDATE: Nullriver has updated the “Community Sources” package. Just install that and skip what’s below. 

I have emailed Nullriver and asked them to update the “Community Sources” package with the new URL for my repository, but they don’t seem to be around today.

If you’d like to switch to the new repository now, you can do so by:

1. Going to the “Sources” page in Installer.app and deleting the current entry for “Ste Packaging”.
2. Add the new repo URL of: http://repo.smxy.org/iphone-apps/ (Make sure you include that final slash! Also make sure that your iPhone doesn’t capitalize the “p” in “iphone”. Going to that URL with your browser won’t work BTW - you’ll just get redirected to this blog site.)
3. You will know you have the new repo, as the tagline for it is “New, and Improved!”

You may need to manually refresh it once, after Installer.app adds the repo and does it’s own refresh. The site will show up as “Uncategorized” (at least, it does for me), but that’s ok.

When Nullriver updates the “Community Sources” package, and you update to it, I don’t know if it will just move my repo to the “Community Sources” category, or if you will wind up with it listed twice. If it does get listed twice, and one is in the “Uncategorized” category, then just delete that one. If you wind up with two of them in the “Community Sources” category, then just pick one and delete it.

If you decide not to do this, please know that the site repo you currently point to will be down for 3-5 hours (or more), tomorrow morning, while the electric company repairs several transformers in the building that server is in.

-ste

The second server was handed over to me this morning. I’m working fast and furiously to get it set up to take over serving the entire repository - both XML and zip files - before the other server’s scheduled downtime of 3-5 hours, tomorrow morning. As soon as I finish that, I will switch all the traffic to this server, and then start getting those long-delayed package updates out.

I’m not happy with the setup of the first server (the one that will be down tomorrow morning), so after it comes back up I plan to have the hosting company re-OS it, whereupon I will set it back up and then split the traffic between it and the second server, with round-robin DNS.

At that point, I believe all of the recent issues with the repository should be totally resolved. Then I’ll finish getting out any remaining, delayed updates.

Only then, will I relax. :)

-ste

My new hosting provider, FDCServers, just sent out mail saying there will be 3-5 hours of downtime this Saturday, starting around 8:00 am CST. Read about it here.

Grr.

-ste

As regular readers of this blog will know, my repository has been having growing-pains issues, recently.

First, my hosting provider (DreamHost) informed me that my main site - the one that only served my repository’s XML file - was using too many server resources (specifically, too many connections per second), which was severely impacting the websites of other customers on the same shared hosting server. They complained to DreamHost, who then complained to me. They told me I had to reduce traffic to my site and added a “Deny from All” into my webserver configuration, which essentially prevented anyone from refreshing my site, in Installer.app. I couldn’t leave it like that, so I removed that from my configuration and began looking for ways to reduce the traffic. Nate True suggested the free, global caching system, CoralCDN. I changed my repository’s location URL to point to the caches and Nullriver updated the Community Sources package to include the change. If everyone had updated their CS package, traffic to the site would have dropped to virtually nothing. Vast numbers of people have not done so, however. In spite of that, traffic to the site was reduced by over 50% and DreamHost did not shut down my site again. Unfortunately, CoralCDN’s caching system, while nice, isn’t without it’s flaws and many users continue to have issues refreshing or, in some cases, even seeing, my site. More on that, in a bit.

Secondly, a few days ago, DreamHost informed me that my other account with them - the one that had been serving approximately 50% of the package zip files - was causing the same problems, on that server, as the other site was previously. They didn’t, however, give me any advance warning and they throttled the site in such a way that I can’t un-throttle it, effectively taking the site mostly down - only a few downloads were succeeding while the vast majority of the thousands and thousands of download attempts failed. Over the next few days, I gradually kept shifting most of the traffic to a server that Nate True had been letting me use. It had previously been handling around 25% of the traffic and as of last night it was handling 75% of it. The remaining 25% of the traffic was being handled by a server that the IRC user “extremis” (the guy who used to maintain all those PXL packages for everyone) let me use, and I had - at least for the time being - abandoned my DreamHost site. So, as of last evening, everyone should have been able to download zips from me again, assuming they didn’t run into the refreshing issue, mentioned above. I couldn’t, however, keep letting Nate’s server take a pounding (nor would his admin let me).

Various people came forward with offers of assistance (and I will be writing each of you a ‘thank you’ email, as soon as the dust settles - your offers meant a lot and were most appreciated). One company offered to pay for a dedicated server box - no shared hosting, no VPS (Virtual Private Server - a better form of shared hosting), but a box with no one else on it to complain that their site is being impacted by my repository. My site can use every resource the box has to offer, right up to its physical limitations. It’s not cheap, either. They ordered the server and the hosting company for it - FDCservers - had it ready for me late last night. There were some initial hiccups - the server went down while I was looking it over and so they replaced all the RAM. Later, it went down again, so they simply yanked the box and set up a new one for me. This box seems to be quite healthy. After beating my head against a wall for a bit, trying to figure out the cPanel configuration tool (I’ve decided I loathe it), I got a site created on the box, copied all the zips to it and at 3:38 am, Eastern, it started serving zip files all of you. As I write this, the server is now serving all of the zips to everyone now, even people who may not have refreshed and picked up the change, as I configured the sites that previously served the zips with permanent redirects to this server.

I have not moved the serving of the repository’s XML file to this server. The server is handling the load well (though it might need more RAM), but adding the XML to it will add an immediate 500,000+ connections per day to the box. I’m not convinced that this box can handle that and still leave enough system overhead for the inevitable increase in the number of refreshes and package downloads that is always occurring.  Plus, should I decide to start using per-connection PHP scripts again, it would consume CPU and memory that I’m not sure I want to take away from this server. So, for the moment, the serving of the XML file is still mostly being handled by the imperfect CoralCDN system and my original site with DreamHost. Personally, I think the wisest move would be to take someone else up on their offer of help and get another dedicated box. That also has the advantage of my not having all of my eggs in one basket. So, I’m still looking for another sponsor, if anyone is feeling generous.

Since I don’t think it would be fair to announce my current sponsor at the bottom of this long post, where many might not see it, please see the announcement of it in my next post. :)

-ste