A DNS Move (For Joe)

Ok Nikki, if you thought my last post was boring try this one. This is a detailed description of how we moved with so little downtime. It turned out to be a lot simpler than I thought it would be, and it worked a lot better than I thought it would. So lets talk about today.

I went to MSA to see if Joe was there and if he wanted to go to implex. He managed to get out of the building faster than I could catch him, and so I was forced to go it alone. That\’s ok. The real reason I wanted to talk to Joe was I knew he was the only one I didn\’t have control of the DNS of, so there wasn\’t much I could do when the switch was made. I probably should have sent out the emails a little earlier. In either case.

This process actually took days to make it happen, it\’s not a quick thing to do by any means, nor do I claim to be a DNS expert at all, but I will describe it the best way I know how.

Lets start with what DNS is. DNS is a way to mapping a name to a number. For example, jrcorps.com goes to 69.54.36.67. The number is what the computer(s) need to know to find what your looking for. There\’s a problem though… Who do you ask to link the number to the name? Well, you ask your DNS server. What if your DNS server doesn\’t know? It asks it\’s DNS server… So on and so forth. All the way to the root server. There are like 10 root servers per tld (.com, .org, .net). They are more or less all knowing. They know where to find the information for any domain name.

Ok, so it would be impossible to try to keep these servers updated with everyones info as well as serve out all the information. There would be to much traffic flow. So simply, there is a caching technology implemented with DNS. When you request the ip address for a name, it goes and looks for it. It then remembers it for a given amount of time. That given amount of time is determined by a value called the TTL (time to live). It basically counts down the seconds until the DNS server goes out looking for the informatino again. This helps limit the number of DNS queries that need to be performed in a given period, as well as speeds up surfing for the user.

Most TTLs are set such that they are between 3 and 48 hours. You want it to be \"in theory\" longer than the length of somethings internet session, so that they don\’t need to query while they are still surfing your site. JR\’s for example I believe are now set somewhere close to 12 hours. This would mean that if I were to go and switch all the DNS records on the server as well as the ip addresses, most people would not see the changes for at least 12 hours. That\’s just not good when you dealing with technologies like email and sites with traffic in the thousands per day. You also have another issue to deal with. Some DNS servers will ignore the TTL and wait for things to expire (it\’s another time setting). For this reason, if you want to make sure your initial change is done and working properly, you should do it at least 48 hours in advance.

In either case, 2 days ago, I lowered the TTL to 3 hours. This ment that most servers should be checking for updates every 3 hours. That\’s good, that gives me enough time to pull one more slick trick. At lunch today, I dropped the TTL down to 3 minutes. I figured 1 minute would be way to short and would just cause our traffice to go through the roof, I figured that the 3 hour window would expire for most people about an hour before I got to implex, and I figured that it would probably take me 3 minutes to switch the IP addresses anyway (if I had joe with me it would have been faster ;-)). In either case, when I arrived at impex, I switch the dns records, then switch the ip addresses and litterally within seconds of doing this, I could already see traffic coming in on the new ip. Within 3 minutes the traffic lights were back to where they had been just moments sooner. I checked over the server to see what I might have forgotten (for example, an apache setting) as well as set the TTL back where they should be (get our traffic back down to normal). All in all, I was impressed at how well the DNS caching timeouts really did work. I dont think I will be so worried about moving servers like this again (as long as I have fair notice).

One more note or caveat if you will with this whole process. You\’ll notice…. I never said I changed the ip address on my DNS server or took down my DNS server(s). That\’s true. In fact, for those of you who knew our DNS was on princess before hand should be scratching their heads right now thinking, how did the internet figure out the DNS server changed IP\’s as well. Well, as it turns out, I moved our DNS server to mario late last week and updated the root servers to reflect that. You don\’t want your DNS to go down in the middle of this change, that will just cause servers to use yet another set of timeouts which is generally longer than the ttls. In either case, that is the way the cookie crumbles or in this case, gets devoured for desert.