How netflow made my users happy.
Recently I had been receiving automated bandwidth alerts for a couple of our offices, so I decided to take a deeper look at what was generating enough traffic to saturate 4 T1's.
How?
The first technology that came to mind was netflow, but I had absolutely no experience with it. I only knew the basics - it would show me source and destination IP's and the type of traffic being transmitted. I also knew that it worked with flows, unlike SPAN ports which forward all traffic and something I wasn't interested in doing.
Since netflow isn't something that's useful without something collecting the data (netflow data is pushed, not polled), the first thing I did was install ManageEngine's Netflow Analyzer. They have a free version that is good up to 2 monitored interfaces (note this is not the same thing as monitored IP's - so you can theoretically just monitor your routers public interface to get useful data!), also I've used manageengine demos in the past and I've always been pleased with their products.
Once I had the Netflow Analyzer installed it was time to enable netflow on my router - a Cisco 2811 running IP Base 12.4(3f). Initially I was pretty concerned about CPU usage because we're already doing PPP multilink which is handled by the CPU and netflow can be CPU intensive, but Cisco's documentation indicated I should be good. Netflow was surprisingly easy to set up:
router#enable
Password:*****
router#configure terminal
router(config)#interface [INTERFACE YOU WISH TO MONITOR]
router(config-if)#ip route-cache flow
router(config-if)#exit
router(config)#ip flow-export destination [YOUR ANALYZER] 9996
router(config)#ip flow-export source [INTERFACE YOU WISH TO MONITOR]
router(config)#ip flow-export version 9
router(config)#ip flow-cache timeout active 1
router(config)#ip flow-cache timeout inactive 15
Note: If ip flow-export version 9 doesn't work, try version 5
In short order, I was in netflow land. Because it works via streams, it took some time for all the data to come available. Roughly an hour later I already had a big enough picture to start acting (IP addresses obfuscated to protect the guilty):
The top bandwidth destination - 192.168.23.5 - is a NAS device in another office (the other office that we were also having issues with).
All of our offices are connected via MPLS and can talk to each other as if they were on the same network. In every office, we have a NAS device for backups. We use a piece of freeware called Cobian to backup that uses a hard coded path for its destination (it isn't location aware).
As it turned out, we had a couple of users move without informing IT, so we never changed their backup locations:
I did some quick math, and figured out that these two users were maxing out our bandwidth in BOTH offices experiencing regular bandwidth shortages. Holy smokes!
Without netflow, I would have never known this was the issue (I always blamed sporting events 
- and with it, I can take corrective action:
1. By changing the backup drives, we can make this go away.
2. I set up an automatic report which runs nightly that sends an email to the department with all traffic destined to backup drives at the wrong location.
3. We can start looking for backup software that limits bandwidth usage (we've needed a new backup package for awhile).
Bandwidth is no longer in such short supply, which cuts down on latency and makes users happy!
If you have any netflow stories, please comment!
HP ESX Management agents won’t uninstall.
Earlier this week I was attempting to upgrade the HP agents on my ESX servers and on one host, for no particularly good reason I was getting a really weird error:
This script will now attempt to uninstall the HP Insight Manager Agents.
Do you wish to continue? (y/n) y
Uninstalling HP Insight Manager Agents bulletin (hp-classic-mgmt-solution-825.10.1344)
Removing (hp-classic-mgmt-solution-825.10.1344)
This script will now attempt to uninstall the HP Insight Manager Agents.
Do you wish to continue? (y/n) y
[hp-classic-mgmt-solution-v825] bulletin found.
Uninstalling ... [hp-classic-mgmt-solution-v825] bulletin
Removing {hp-classic-mgmt-solution-v825} ...
Encountered error NoMatchError:
The error data is:
Id - hp-classic-mgmt-solution-v825
Message - 1 of 6 Vibs in this bulletin have been superseded. Please try
removing the newest bulletin for this component instead.
Errno - 13
Description - No matching bulletin or VIB was found in the metadata.
Unable to remove {hp-classic-mgmt-solution-v825}. esxupdate status {13}
[ FAILED ]
Exit 0
No matter what I tried, I couldn't get it to go away. I tried removing the RPMs and that wouldn't work. I also tried to remove the bulletin's manually, but I couldn't - it yielded the exact same error as above (turns out the HP utility does anyway is make a call to esxupdate).
A bulletin is essentially a rollup of packages (RPM's in this case) which VMWare calls a VIB or vSphere Installation Bundle. In this particular instance, a single VIB was stopping the process and needed to be removed manually. How though?
I was totally stumped, so I placed a call to VMWare support, and they showed me this nifty UNDOCUMENTED command to list the VIBs:
[root@vistsfo1 831]# esxupdate --vib-view query | grep hp
rpm_hp-smh-templates_8.2.5-51@noarch installed 2009-07-13T10:07:01.211124-07:00
rpm_hp-snmp-agents_8.2.5-50.vmware4x@x86_64 installed 2009-07-13T10:07:01.141505-07:00
cross_hpilo_400.1.1.1.1VMW-00001 retired 2009-07-13T10:07:01.275159-07:00
rpm_vmware-esx-drivers-scsi-hpsa_400.3.6.14.27vmw-1.0.7.193498@x86_64 retired 2009-09-26T22:04:40.219785-07:00
rpm_vmware-esx-drivers-scsi-hpsa_400.3.6.14.28vmw-2vmw.1.9.208167@x86_64 installed 2009-11-27T10:40:13.064463-08:00
rpm_hpsmh_3.0.1-73@x86_64 installed 2009-07-13T10:07:01.490639-07:00
rpm_hp-health_8.2.5-50.vmware4x@x86_64 installed 2009-07-13T10:07:01.224935-07:00
rpm_hp-agents-config_8.2.5-24@noarch installed 2009-07-13T10:07:01.148444-07:00
Great! Now, to remove the offending package. In my case I knew it was the right VIB because it was there were only 2 non driver related VIBs installed.
[root@vistsfo1 831]# esxupdate remove -b rpm_hp-snmp-agents_8.2.5-50.vmware4x@x86_64
So now onto the interesting part. The --vib-view command is a hidden command for esxupdate. It's not in the man page, so how could anyone know about this handy flag without calling VMWare? Well, hindsight brings up and interesting method... open esxupdate on your ESX server. On line 146 (ESX 4U1) you can see this:
# Hidden options
parser.add_option('--HA', action='store_true', help=optparse.SUPPRESS_HELP)
parser.add_option('--vib-view', action='store_true', dest='vibview',
help=optparse.SUPPRESS_HELP)
parser.add_option('--maintenancemode', action='store_true',
help=optparse.SUPPRESS_HELP)
I suppose I could have just looked that the script to find that out!
Hopefully this will help, I saw a few posts out there about it.
Sync Active Directory with Postini
About a year ago I had the opportunity to set up Postini for my organization as a part of my Exchange 2007 setup/migration.
I selected Postini as our anti-spam service - mainly because our users were already used to the interface. One of the biggest challenges during planning was how to keep Postini current with my organizations mailboxes. I could let Postini create an account when an email is received, but that sounded like a really bad idea. Thankfully, in my research I found out about an application that will sync AD to Postini: Google Apps Directory Sync for Email Security.
The tool isn't overly difficult to set up, but it does have a few quirks that I feel I should share - especially since Postini's paid support is awful. We paid $750 for support and when I needed help getting it working I was informed that they didn't support syncing and referred me to knowledge base articles. I managed to figure it out on my own (on time!) and I've had it working perfectly ever since. Below you can find some set up information that may help you. Please keep in mind that this setup works in my environment and may not work in yours.
Once you download the application (link above), you will need to create a "profile" that you will use to run the syncing application.
The first screen you will see is Authentication. This is where you input your administrator login for your Postini organization.
The second screen you will see is Orgs - you can select a specific organization to sync to (for instance, if you had multiple domains). I have it set to "All users in all orgs under your account," and it works perfectly with a 3 org nested setup like mine.
Exclusion filters. I don't use these, but it will allow you to exempt accounts from being syncronized.
LDAP settings: This is where you input your Active Directory information. You can use either LDAP or LDAPS (SSL LDAP). If you opt to use LDAPS, don't forget to change the port to 636. Input a domain controller IP under Host Name, add in your base domain - for example: DC=subdomain,DC=domain,DC=com. Authentication is an account with read access to your AD - use the DOMAIN\user format.
For LDAP User Attributes, select MS Active Directory. The email address attribute is "mail." Alias Address Attributes are "proxyAddresses."
For user sync - this is an important part: If you don't add a search string for users, it will also pull computer accounts. It won't add them, but it will error your log report. I added my Org Name, SUBTREE, (sAMAccountType=805306368) - this will filter out everything, but users. You could add any kind of filter you wanted here. More about LDAP filters.
Mailing Lists - I don't sync them. They are almost all internal only. Since you have to also make a change in Exchange 2007 to allow for non-authenticated users, I make exceptions for dist lists through the web interface for public facing mailing lists. This is up to you.
Notification - This will email you a report every time it runs that tells you if there were any errors, how many users were added/removed, etc. I recommend this.
Delete limit - I have this left at the default 5% - I can see the wisdom in this setting 
Log files - lets you set the file path/limit on the synchronization log. I left as is.
Now you should be done. Go ahead and run the sync simulation. At the bottom of the log in the Sync Log tab, you should see: Simulation completed successfully - good job! Go ahead and save the profile by going to the file menu to a location you will remember easily.
The sync tool will not run automatically on its own - and for that I set up a scheduled task that runs every day at midnight and runs a cmd script that contains the following:
sync-cmd -a -c emailsync.xml
WAIT
sync-cmd is located in your Postini sync tool location - most likely C:\Program Files\Google Apps Directory Sync for Email Security. The -a option commits the changes and the -c option specifies the configuration file you created above. Voila! Every midnight you should get an email letting you know what has changed.
Hope that helps!
Need to move mailboxes quickly?
I'm supposed to have this week off for our company's annual winter break, but like most sysadmins I will end up having to work when everyone else doesn't.
I get the pleasure of moving about 250GB worth of mail around in an effort to get rid of 2 corrupted mailstores. Mailstores that even Microsoft can't fix. I have spent about 15 hours on the phone with Microsoft since Friday - it is now Monday. The original call was to remove a couple of mailboxes that weren't being deleted normally. Somehow a user (it's always the users fault... right? ) of mine managed to create a looped folder in her deleted items box. Imagine a folder structure like this:
Deleted Items
A+
B+
A+
B+
A...B...A...B... for eternity.
I tried everything. Moving the mailbox, disconnecting and reconnecting the mailbox, mfcmapi, pfdavadmin, screaming and pounding on the keyboard. Nothing worked. Microsoft couldn't figure it out either. I now believe that this may be the cause of my latest BackupExec issues alluded to in my previous blog post- a belief that is strong enough that I have actually halted publishing it until I'm sure. Don't get me wrong though, Symantec hasn't made a good product in years...
Microsoft's solution is the same one that I had come up with before I had called them. I have to move all of my mailboxes off the affected mail stores. Which means about 400 mailboxes and 250GB.
Not wanting to sit there and manually move one mailbox at a time, I decided to seek out a powershell script to do the moving for me. I needed something that could read from a CSV file and do a multithreaded (more than one a time) moves. The multithreaded part turned out to be the big problem. Writing a powershell script to move 1 mailbox at a time is something that takes no real effort, but multithreaded requires a little bit more complexity. Since I'm never one to reinvent the wheel, I turned to Google and stumbled (after way too much keyword manipulation) to find a MSExchangeTeam blog article on it.
It allows you to import a CSV file with the following format (I'll save you from trying to find it in the documentation):
Identity,targetmbserver,targetmbsg,targetmbdb
Bob Smith,mailserver,storagegroup,mailboxdatabase
Jane Doe,mailserver,storagegroup,mailboxdatabase
It will then move 4 at a time and display the progress. Perfect. Now if only they could move faster...
Backup Exec – the bane of my existence. Part 1.
This is one of those times I'm glad to have a blog - a place that I can write down my thoughts for Google eternity to go back on.
Backup Exec. I've been using it since version 9 when it was still Veritas and unlike a fine wine, it has not been getting better with age. 9 was the last good version. I never had any problems with it and backups and restores were never a hassle.Version 9 and I were friends.
I don't remember 10 and 10d very well, but I do remember the amount of pain they caused me. 10d was also when Symantec re-branded the product and marks when the software really started to take a nose dive.
Fast forward to 11. It's now Symantec Backup exec. Symantec adds a really nice (sounding) feature called GRT, or Granular Restore Technology. In previous versions, Backup Exec had to back up the entire mail store and you had to go through a lengthy process if you wanted to restore individual mailboxes or email messages. Not so with GRT.
I remember being very excited about this feature and the moment I got the upgrade I turned it on. Backups worked well enough. The usual random failures for no particular reason.
One day I received a call from my boss asking if I could restore a mailbox for an ex-employee. The ex-employee needed a couple of emails for a personal matter and had left on good terms. I said "No problem boss, I'll get right on it," like the good PFY I was.
I set up the restore job, ran it, and Backup Exec told me which tape to insert. I slapped the tape the tape drive. The job starts, gets almost all the way to the end and then fails. Oh crap.
Typical error message from hell... you know, the one that doesn't actually tell you anything. This was before they had a functional knowledge base.
I called Symantec for support and was connected to the usual under-trained tech support employee. To make a long story short, after a fair amount of troubleshooting the guy said that it was a known issue. What was a known issue? You couldn't restore a mailbox if the mailbox owner had an email signature. Right.
In the next part of this series, I go over Backup Exec 12.5 and my constant battle for working backups as well as outline a few fixes that seem to resolve most Backup Exec errors.