Need to move mailboxes quickly?
I'm supposed to have this week off for our company's annual winter break, but like most sysadmins I will end up having to work when everyone else doesn't.
I get the pleasure of moving about 250GB worth of mail around in an effort to get rid of 2 corrupted mailstores. Mailstores that even Microsoft can't fix. I have spent about 15 hours on the phone with Microsoft since Friday - it is now Monday. The original call was to remove a couple of mailboxes that weren't being deleted normally. Somehow a user (it's always the users fault... right? 
) of mine managed to create a looped folder in her deleted items box. Imagine a folder structure like this:
Deleted Items
A+
B+
A+
B+
A...B...A...B... for eternity.
I tried everything. Moving the mailbox, disconnecting and reconnecting the mailbox, mfcmapi, pfdavadmin, screaming and pounding on the keyboard. Nothing worked. Microsoft couldn't figure it out either. I now believe that this may be the cause of my latest BackupExec issues alluded to in my previous blog post- a belief that is strong enough that I have actually halted publishing it until I'm sure. Don't get me wrong though, Symantec hasn't made a good product in years...
Microsoft's solution is the same one that I had come up with before I had called them. I have to move all of my mailboxes off the affected mail stores. Which means about 400 mailboxes and 250GB.
Not wanting to sit there and manually move one mailbox at a time, I decided to seek out a powershell script to do the moving for me. I needed something that could read from a CSV file and do a multithreaded (more than one a time) moves. The multithreaded part turned out to be the big problem. Writing a powershell script to move 1 mailbox at a time is something that takes no real effort, but multithreaded requires a little bit more complexity. Since I'm never one to reinvent the wheel, I turned to Google and stumbled (after way too much keyword manipulation) to find a MSExchangeTeam blog article on it.
It allows you to import a CSV file with the following format (I'll save you from trying to find it in the documentation):
Identity,targetmbserver,targetmbsg,targetmbdb
Bob Smith,mailserver,storagegroup,mailboxdatabase
Jane Doe,mailserver,storagegroup,mailboxdatabase
It will then move 4 at a time and display the progress. Perfect. Now if only they could move faster...
Backup Exec – the bane of my existence. Part 1.
This is one of those times I'm glad to have a blog - a place that I can write down my thoughts for Google eternity to go back on.
Backup Exec. I've been using it since version 9 when it was still Veritas and unlike a fine wine, it has not been getting better with age. 9 was the last good version. I never had any problems with it and backups and restores were never a hassle.Version 9 and I were friends.
I don't remember 10 and 10d very well, but I do remember the amount of pain they caused me. 10d was also when Symantec re-branded the product and marks when the software really started to take a nose dive.
Fast forward to 11. It's now Symantec Backup exec. Symantec adds a really nice (sounding) feature called GRT, or Granular Restore Technology. In previous versions, Backup Exec had to back up the entire mail store and you had to go through a lengthy process if you wanted to restore individual mailboxes or email messages. Not so with GRT.
I remember being very excited about this feature and the moment I got the upgrade I turned it on. Backups worked well enough. The usual random failures for no particular reason.
One day I received a call from my boss asking if I could restore a mailbox for an ex-employee. The ex-employee needed a couple of emails for a personal matter and had left on good terms. I said "No problem boss, I'll get right on it," like the good PFY I was.
I set up the restore job, ran it, and Backup Exec told me which tape to insert. I slapped the tape the tape drive. The job starts, gets almost all the way to the end and then fails. Oh crap.
Typical error message from hell... you know, the one that doesn't actually tell you anything. This was before they had a functional knowledge base.
I called Symantec for support and was connected to the usual under-trained tech support employee. To make a long story short, after a fair amount of troubleshooting the guy said that it was a known issue. What was a known issue? You couldn't restore a mailbox if the mailbox owner had an email signature. Right.
In the next part of this series, I go over Backup Exec 12.5 and my constant battle for working backups as well as outline a few fixes that seem to resolve most Backup Exec errors.
vSphere U1, 1 down, 1 to go.
Good news first - in my previous blog post I had mentioned that there was a serious bug with updating to ESX 4 U1 and HP SIM agents. This has been resolved, U1 has been re-released and people should not have this issue any more (I have not tested this myself). That KB article has been updated and is available here.
The bad news is that there is now a bug with U1 and vCenter if it manages ESXi hosts.
It does not seem nearly as drastic as the previous issue I blogged about, but you should be aware of it. For now, it seems that you can avoid running into it by not adding, removing and then readding an ESXi host to vCenter.
Stay tuned for my next post (possibly a series?) about backups.