Ste Packaging

While my site is having issues (which I’m working hard to resolve), you will get told that package downloads have failed, for at least some of my packages.

The package still remains in the new Installer.app queue, however, which causes problems when you try to download packages from other places.

The fix, I’m told, is to clear your queue, in Installer.app, before trying to download something from another repository than mine.

-ste

Recently, I had to move the serving of my repository’s XML file from the Dreamhost server that was serving it, to a free global caching system (http://coralcdn.org), as my hosting provider was threatening to disable the site due to the high number of connections per second. That move, even though the caching system isn’t perfect, saved the site and Dreamhost left the site alone.

Today, they have throttled a different account of mine - the one that serves 50% of all the zip files. They said that there were too many connections and it was driving up server load. Consequently, the vast majority of all downloads from that server are failing now, and there’s nothing I can do about it. There are two other server, that two other folks graciously let me use, that are currently each serving about 25% of the downloads. They cannot handle more traffic.

I have complained to Dreamhost, pointing out that they have essentially destroyed my site. I also have pointed out to them that they are not providing me the service that my account is supposed to have - namely, 10TB of data transfer a month. I’m not even close to that and they have cut me off and rather ruined things.

I’m not expecting them to change their minds, however.

Sigh.

I’ll try to figure *something* out …

-ste

UPDATE: 2008-02-03, 14:55 - I shifted half of the load from the throttled server to one of the other two, temporarily. Now that server is serving 50% on my zip files and the throttled server is only serving 25%. This means that fewer downloads will fail. Also, because it’s an important package, I hard-coded  the OpenSSH package to only come from an un-throttled server, so that people can get to it. Make sure you refresh your Installer.app, so that you pick up the change, before trying. -ste

Between folks upgrading to the newest Installer.app, with its extended auto-refresh time (24 hours, instead of 30 minutes), and folks upgrading to the newest Community Sources package, which changed where Installer.app gets my repository’s XML file (from 250 caching servers, world-wide, instead of directly from my server), the traffic to my main server has been reduced.

It hasn’t gone down as much as I’d like (if you haven’t upgraded your Installer.app and Community Sources package, please do!) but it’s definitely headed in the right direction:

2008-01-20: 464,518 refreshes
2008-01-21: 316,634 refreshes
2008-01-22: 245,471 refreshes

The result of this reduction is that the average hits per second on my site (well, the server that handles the refreshes, anyway) has dropped from 6 to 3, which is excellent.

The goal is zero direct refreshes. Clearly, there are a lot of people who have not upgraded their Community Sources package, unless there are a lot of IP addresses pulling my XML file, that are pretending to be Installer.app.

When I watch the logs for just package downloads, I see that most appear to be coming from the newer version of Installer.app, but there are still a large number coming from older versions. Again, it would seem that a lot of people have yet to upgrade to the newer Installer.app, unless there are a lot of IP addresses pulling my zip files, that are pretending to be Installer.app.

To give you some figures on that, as of 2:38 PM Eastern, today, 365,747 people have upgraded to the newer Installer.app. Wow. There are a lot of hacked iPhones and iPod Touches out there. As for the impact of that on my zip downloads, look at this:

-ste

It seems to be working pretty good for the most part. There have only been a couple of problem reports. As I write this, it’s been 11 1/2 hours since the caching system went into effect when Nullriver updated the “Community Sources” package for me. Since then, 200 cache servers around the world have picked up and started serving the XML file, and the number is climbing all the time. Much better than just my one server serving it!

Already, I am seeing the access.log scroll by slower as I continually tail it. I even see the occasional brief pause in the output - something I haven’t seen in months. So, it looks like the caches are starting to take away some of the load and traffic, as hoped.

If you haven’t already updated your “Community Sources” package to the latest version, please do so. The faster people do, the faster the load on my server goes down. Thanks!

-ste

Please upgrade to Installer.app 3.0b9 as soon as possible. Nullriver has released this update to try and help reduce the traffic on my site so that DreamHost doesn’t shut it down.

What they did, was increase the refresh timeout from 30 minutes to 24 hours. So now, when you run Installer.app, it will not automatically refresh unless it has been more than 24 hours since it last refreshed. Unless people start manually refreshing more (please don’t!), this should drasticly reduce traffic to my main server, as people upgrade to the new version.

My thanks go out to Nullriver for this update!

UPDATE: While I was waiting for this update to come out, I discovered a bug in Installer.app’s MovePath and CopyPath commands. I reported it to Nullriver and this release fixes that bug, in addition to changing the refresh timeout, as described above.

The bug was that, in this XML code:

MovePath
/var/root/Media/Labyrinth/settings.plist
/var/root/Library/Labyrinth/settings.plist

if the “/var/root/Library/Labyrinth” directory didn’t exist, Installer.app would not create it, as it should have. The behavior was the same if you changed MovePath to CopyPath. This bug did not occur when the source was the package being installed.

-ste