Rinsefirst

Ok Nikki, if you thought my last post was boring try this one. This is a detailed description of how we moved with so little downtime. It turned out to be a lot simpler than I thought it would be, and it worked a lot better than I thought it would. So lets talk about today.

I went to MSA to see if Joe was there and if he wanted to go to implex. He managed to get out of the building faster than I could catch him, and so I was forced to go it alone. That\’s ok. The real reason I wanted to talk to Joe was I knew he was the only one I didn\’t have control of the DNS of, so there wasn\’t much I could do when the switch was made. I probably should have sent out the emails a little earlier. In either case.

This process actually took days to make it happen, it\’s not a quick thing to do by any means, nor do I claim to be a DNS expert at all, but I will describe it the best way I know how.

Lets start with what DNS is. DNS is a way to mapping a name to a number. For example, jrcorps.com goes to 69.54.36.67. The number is what the computer(s) need to know to find what your looking for. There\’s a problem though… Who do you ask to link the number to the name? Well, you ask your DNS server. What if your DNS server doesn\’t know? It asks it\’s DNS server… So on and so forth. All the way to the root server. There are like 10 root servers per tld (.com, .org, .net). They are more or less all knowing. They know where to find the information for any domain name.

Ok, so it would be impossible to try to keep these servers updated with everyones info as well as serve out all the information. There would be to much traffic flow. So simply, there is a caching technology implemented with DNS. When you request the ip address for a name, it goes and looks for it. It then remembers it for a given amount of time. That given amount of time is determined by a value called the TTL (time to live). It basically counts down the seconds until the DNS server goes out looking for the informatino again. This helps limit the number of DNS queries that need to be performed in a given period, as well as speeds up surfing for the user.

Most TTLs are set such that they are between 3 and 48 hours. You want it to be \"in theory\" longer than the length of somethings internet session, so that they don\’t need to query while they are still surfing your site. JR\’s for example I believe are now set somewhere close to 12 hours. This would mean that if I were to go and switch all the DNS records on the server as well as the ip addresses, most people would not see the changes for at least 12 hours. That\’s just not good when you dealing with technologies like email and sites with traffic in the thousands per day. You also have another issue to deal with. Some DNS servers will ignore the TTL and wait for things to expire (it\’s another time setting). For this reason, if you want to make sure your initial change is done and working properly, you should do it at least 48 hours in advance.

In either case, 2 days ago, I lowered the TTL to 3 hours. This ment that most servers should be checking for updates every 3 hours. That\’s good, that gives me enough time to pull one more slick trick. At lunch today, I dropped the TTL down to 3 minutes. I figured 1 minute would be way to short and would just cause our traffice to go through the roof, I figured that the 3 hour window would expire for most people about an hour before I got to implex, and I figured that it would probably take me 3 minutes to switch the IP addresses anyway (if I had joe with me it would have been faster ;-)). In either case, when I arrived at impex, I switch the dns records, then switch the ip addresses and litterally within seconds of doing this, I could already see traffic coming in on the new ip. Within 3 minutes the traffic lights were back to where they had been just moments sooner. I checked over the server to see what I might have forgotten (for example, an apache setting) as well as set the TTL back where they should be (get our traffic back down to normal). All in all, I was impressed at how well the DNS caching timeouts really did work. I dont think I will be so worried about moving servers like this again (as long as I have fair notice).

One more note or caveat if you will with this whole process. You\’ll notice…. I never said I changed the ip address on my DNS server or took down my DNS server(s). That\’s true. In fact, for those of you who knew our DNS was on princess before hand should be scratching their heads right now thinking, how did the internet figure out the DNS server changed IP\’s as well. Well, as it turns out, I moved our DNS server to mario late last week and updated the root servers to reflect that. You don\’t want your DNS to go down in the middle of this change, that will just cause servers to use yet another set of timeouts which is generally longer than the ttls. In either case, that is the way the cookie crumbles or in this case, gets devoured for desert.

The move is scheduled tomorrow at 4:00. Hopefully it goes without a hitch. I apologize for any downtime one might see. That would really be lowsy. I wonder if I can fool the mailservers…. That might be worth trying.

Let\’s see, what else is new…. Really not much. I should probably remove the video test because I\’m not broadcasting anymore. I should probably design half a dozen webpages for myself and friends, but I\’m lazy and only like working when I have to (which is all the time). Poker is poker. Football is football. Church is church. Msa is Msa.

I\’m not sure what else there is to say. See I could try to talk about something witty for once. Something that would interest people. Kinda like Tumbledry or Slashdot.

I could just write junk.

Maybe I should just go to sleep.

I\’ve hit a dilemma. I need to move this server to a new IP address. In no way, is this an easy task, nor did I ever expect it to be. The way I wanted to do it, I can\’t, so life it would seem be telling me to try something else. Let me describe the problem for everyone:

The internet, as many of you know is really nothing more than a bunch of computers all tied together using routers, hubs, and switches. The only thing that keeps the thing going is the fact that everyone has a magical number. This magical number helps other computers find your computer, and vise versa. Of course, numbers are only good to machines, so the internet created a rather nifty way of handling names. This process is done VIA something called DNS or domain name service.

There\’s a problem though. There is a delay between the DNS and the time it takes the ip addresses to change. That means anytime a change is made to a DNS record, one of two things can happened. Either it sees the old version or the new version, depending on how long it has been since it\’s name changed. Because of this, if I were to just change the nameservers right now, as well as the IP address, it would look like JR was down for up to 48 hours to some people. This is of course, not acceptable. There are some tricks to reduce the time on this, but no matter how you look at it, it adds up.

So instead, I think I have decided on another plan which takes about 100 times more work, but may provide me with some other interesting options. I plan on moving each site, 1 by 1 to a new server, taking it down in a fashion that no matter what happens it should remain up. I will start with the JR site, and go on to sites that are easy to move (the ones where downtime isn\’t going to hurt me), and then onto the harder ones.

This may mean that FTP or something stops working, email will however continue to work… It must.

This means I can always dump the crap on my current server that is no longer needed (sites that died out years ago). I think in fact, I\’m going to start doing this now… so later everyone.

It\’s been an interesting week I must say. One that deserves some writeup in the blog. JR had a very good week in just about every respect. We got 2 new clients. We had 3 or 4 clients pay us significant amounts of money. We had 2 new servers installed and implex and the third one was moved with only 4 minutes of downtime. We upgraded Spam Assassin, but I think it needs to be retrained because I think it\’s doing a worse job than before. All in all, if every week was like this now, it would be great. Of course, that\’s not the way life works.

Many of you have noticed that the video test has been off more than one lately. Sorry about that. I only turn it on when I bring my laptop to MSA now. I don\’t have it plugged in to the beast at the moment because I thought flash was crashing my computer. It turns out, I may have been wrong. I should have been taking my own advice… let me explain.

When I go and fix someones computer, I often find that their problem is the hard drive is too full. One person imparticular stands out in my mind for when I went to look at his computer he had 10 megs of free space… Windows XP will not like that at all. My normal recommendation is to keep it at least about 5% free, or 15% of you can do it. Now, I never had actually looked up these numbers. They were more of a guess based on personal experience. In fact, somewhere in the back of my mind, the percent of freespace did not matter. It was the amount of freespace. That was of course what my computer science mind would expect.

Turns out I was somewhat right, but I\’m not entirely sure why. I knew my performance was not up to par with where it normally was. I started looking for causes. I thought maybe my ram was acting up again, what not… Didn\’t seem to matter. The computer would still function just fine, but it was a little slow to open things like my documents as well as a little more prone to crashing. I looked towards the hard drive. At first glance, I was fine. 15+ Gigs free.

I looked again. 3% FREE?!?! I wondered if it was that I had fallen below that magical 5% mark where I swear windows had always slowed down. Actually, my initial guess was that I needed to defrag. Turns out though you need a certain percentage of disk space to do that ;-). So I started deleting things. 86 gigs later I was back close to 25% free. Everything was running faster. My Documents still takes a little bit to load (probably has something to do with the massive sized folders below it.) but overall, it works really well. So people, I\’m thinking the 5% is really 5%.

I\’m still thinking though I should check out my raid card.

In other news, I\’m heading north today to go to a party for my Grandma Kathy. She\’s not really my grandma, but she raised me and my sister from the ages of 1 to 6 maybe longer in some cases. She was our day care provider while my parents worked. It\’s where I learned to play Super Nintendo, work on a Apple II, and just have fun building forts. I haven\’t built forts in awhile… I should do that. She\’s celebrating her 50th wedding aniversary, very cool.

The Nintendo Revolution controller is out if you haven\’t seen it. It\’s one handed with attachments that can make it two handed. It\’s wireless. It\’s gyro\’d meaning you can use it like a sword. All in all, it\’s going to be different and unique. It will be interesting to see if the other video game systems try to follow suit, or if Nintendo\’s Innovation pays off.

I think I better go get a start on my day. I should really try to start updating more, and I need to remember to keep business cards in my wallet. I keep having dreams about how I never have them when I need them.

Ok, so the webcam seems to be some sort of success. Not sure why. So much stuff that makes that cool. I apologize if it glitches out and pauses on you, it has problems with that from time to time when more people are watching then the current server allows. That said, enjoy what you can, reloading always seems to help… But its fun to know that at this exact moment there are 2 other people watching me.

I did something I thought I would never do. I joined facebook. Fun not? We\’ll see how it goes, but I feel its good to connect and I\’m not a very personal guy so really, what do I have to hide? Good question.

Computer Club was good yesterday.

Nintendog\’s is ruling my world. I\’m not addicted, but you know, I love games that are the 30 minutes a day, quick pick up and play kind of game. You should check it out if you have a DS.

Well, I best get back to whatever it is I\’m supposed to be doing.

Some of you may have been wondering what the box on the right is… That\’s live from the server room at MSA. It\’s not fully available to the public persay due to bandwidth restrictions, but as soon as 2.0 comes out, I plan on either buying it for JR (with Jon and Blaines Permission) or buying it personally and then giving it to JR. We\’ll se how it goes. For now, enjoy me if I\’m here.