Todays Time Log

9:15 AM: Phone call from Joel – Needs wireless remote… I accidently brought it home. Not the best way to start the day… Disappoint Joel, but get it to him after church.

10:45 AM: Church – Something told me today wasn\’t going to be very good, but God is great…

12:01 PM: Lunch with the family. Days still going pretty well, but I\’m unhappy about somethings that have been going on.

1:30 PM: Start watching NASCAR. Richard Lost, John Lost, Dad Lost… Meaning, Ninjor Won, I won, and Amber won… Translation: The top 3 people lost, the bottom 3 won. The running order stays the same, but it will cause everyone to be withing 1.5 games of each other (which is cool considering it\’s already week 7).

2:00 PM: Play some tetris. Fun, but definently not the worlds best player (anymore).

3:10 +/- 15 PM: Check emai… (wait, interupted)

3:11 +/- 15 PM: Pings showing intermident server access.

3:11 +/- 15 PM: Jon Calls about something else, I ask him if he just stared having ping trouble… Yep…

The next few hours are a blur but it went something like this:

Try to get a shell (nope)

Try to get ftp (nope)

Call Implex (no answer)

Try to get another shell (nope)

Try another way of getting another shell (nope)

Call Impex (no answer)

Check Bandwidth (shit! we\’re AVERAGING 7.2 Mbps. That\’s faster than a lot of your home networks, and I\’m doing it over the net.)

Get frustrated…

Call Implex (no answer)…

Realize there is nothing I can do but wait to see what happens.

Bandwidth continues to be at 10 MBPS

7:30pm: Go to Dairy Queen, Call Implex on a whim. Notice that there call center has changed messages (from daytime to nighttime afterhours) which causes it to provide a new phone number to try for after hours support.

8:30pmish: Online with implex support. Try killing the port to interupt the traffic long enough for me to get access. No luck. Realization hits… I\’m going to have to go to implex.

9:00pmish: On my way to implex.

9:45pm: Inside implex. Servers are running fine, except for this one rogue process on the primary server. I spend the next few minutes isolating the process. Removing it from running processes. Start looking through logs but can\’t find anything that looks horrible.

10:45: Everything is back on. No fixes in place. Additional monitor is put in place to make sure we can track if it happens again (Implex put\’s a packet sniffer on, so we aren\’t relying on our machines). Remove a few processes that we don\’t need that have known to been vulnerable in the past.

11:45 Arrive at MSA. Start killing the machines more…

1:45 get bored of looking at logs and upgrading servers (ok, not bored… tired). Write blog post to take mind off things… But must continue until at least a few more upgrades have been made. There are some people who will be talked with tomorrow because I\’m almost 90% sure it\’s your code that did it, not mine (people who are running phpBB and Drupal come to mind).

That\’s my spiel… Thanks all for the patience and the kind words… It\’s keeping me alive at this point.