|
So ever since we moved to the new server, we have been starting to see intermittent problems with posting: duplicate posts, "failed" posts, etc. I think that I have that fixed, now. So if you have been seeing the website hang after you post something, that should go away now.
What was the cause? Wasn't the new server supposed to be so much better and faster? Well yeah, it is. But it is also a shared resource, and that caused some new problems that never showed up on my server at home. The problem: the File Cache.
Disks are really slow, compared to memory in modern computers. So operating systems include a thing called a "File Cache" which is a memory copy of what is on disk. The first time that you read something, it is read from disk; after that it is read from memory. Things stay in cache until the cache fills up too much, at which point the old things start to get dumped from cache.
In a webserver, all of the pages are files, organized into directories. When lots of people all look at the same page, the webserver reads it once from the disk and then all of the other users get to access the memory copy. For something like VillagersOnline, this means that the home page, and the recent blogs, and such, are almost always in cache because somebody else looked at them recently. Sometimes, even things like recently uploaded mp3's might be in cache, if the server has enough memory.
Now, about our architecture...LatestUploads attempts to summarize everything that exists in our 1000s of uploads. It used to be dynamically generated every time that you visited the site, but eventually that took too much time to calculate for each page. So, I implemented my own sort of cache: there is a file which stores the most recent version of LatestUploads (and this file is almost always in the cache of the webserver). Whenever you upload something new, I delete the old file and build a new one. Thus, the uploader experiences a small delay, but nobody else does.
Now, this worked very well on my old system because the entire information about our directory tree full of uploads generally sat in the File Cache at all times. Even though it wasn't being used, there was plenty of memory, and so there was never any reason to get rid of it. So when we needed to build a new LatestUploads, we had to scan 1000s of directories (which is slow) but each directory read was in File Cache (so was fast). There was a pause, but it wasn't too terrible.
When we moved to the new server, however, we are now on shared hardware somewhere over in Georgia. This means that there are many websites served off of this same machine, not just VillagersOnline. This means that if I don't scan the directory structure once in a while, all that information gets dumped from the File Cache; thus, when you need to rebuild LatestUploads, it has to scan 1000s of directories, and *every one* of the directory reads requires a read from disk! This caused the LatestUploads generation time to stretch into minutes, sometimes.
That's why people would often post things and think that they failed...because the LatestUploads update was taking so long. The thing was actually posted just fine (that is really quick, typically) but the cleanup afterwards was taking a while. So people would hit Reload, or retype the message, and upload again...and we'd get duplicates.
I've worked out a pretty simple solution, now. I've moved the LatestUploads updating into an asynchronous process...this means that even if it takes a long time, your "upload successful" page will show up quickly. So if you see a message about "LatestUploads is updating"...don't hit Reload, don't post again, just wait a couple and your blog will be there.
Sorry for all the trouble this caused. The problem was obvious in hindsight, but it never occurred to me beforehand. That's often how things are with programming.
Russ |
