A Day in the Life

Ever have one of those days where nothing turned out the way you expected when you woke up in the morning?  Probably – most of us have.  Today was my turn.  I woke up to an email (which was followed by a phone call) saying that we had a mail server down at work.  And not just any mail server – it was the one that all our big wigs use.  Now I’m not involved with servers, storage, or backups on a day-to-day basis, but this was an “all hands” call kind of problem.

What started as a reboot lead to mail databases that required consistency checks.  Just about when we had those almost cleared up, then we realized no mail was routing (in, out, or intra).  Couldn’t solve the mail routing issue without bouncing the server again.  So now mail was flowing again.  But it seems that a large number of users mailboxes are damaged (inbox and folders missing or missing content).  Throughout the afternoon the problem got worse as the email server software self restarts (multiple times).

So right now the server is down for “offline” maintenance.  That should take 5 or 6 hours to run.  We will be backing off a software patch from the previous weekend (to rule out any possible connection that update might have to the events of today).  After the maintenance and patch back-out are done, we will restore (from tape) about 30 mailboxes.  Then we have to compare inbox and folder contents from the repaired mailboxes to the restored copies.  Sounds like fun, right?

So I’m going to taking a nap.  I expect to begin working around midnight or 1 AM.  Sometimes this is how it goes in the IT business.  Hope your day was less stressful than mine.  🙂

1 Comment »

  1. berzins said,

    January 16, 2008 @ 5:08 am

    Dude, It wa a black tuesday for everyone.

    Talk about 12~14 hours of meetings with civil construction sharks trying to negotiate a USD1 bi deal alone with a room full of chinese.

    I really would prefere dealing with a buggy server 😉

RSS feed for comments on this post · TrackBack URI

Leave a Comment

Creative Commons License Creative Commons License