How to lose $1,000 in 30 seconds.
A little over a month ago I was asked to set up a SFTP server so that our clients could transfer files securely. SFTP is a bit of a misnomer, you would expect it to be a subset of FTP, but it's not. SFTP is actually its own protocol designed as an extension to SSH. The further confuse the issue, SFTP is not the same as FTPS - a little used SSL version of FTP.
I knew that SFTP was more on the Linux side of things, so I decided at first to use linux for this. Even though I'm primarily a Windows shop, I firmly believe in the right tool for the job. I even got a bit of budget to buy RHEL for it. Unfortunately, as I came to find out after trying to set it up, there is a limitation that had me running to another solution. The limitation was part of a requirement handed to me - I couldn't allow our clients to traverse directories and get a list of our other clients. The way openssh implements chroot allows for this to happen and there's no way around it. Your SFTP users will end up in their own directory, but a simple "cd ../ls" will show them a list of your clients (or root directory). I later read that proftpd may not do that, but by then I had opted for a Windows solution.
There are a number of products out there that will do SFTP on Windows - some free, some not. Because this would be something I would be running in a production environment that is client facing, the solution had to include a support option. This narrowed my choices to Serv-U and WS_FTP. I've used both in the past and I always had a pretty decent impression of Serv-U so I installed a demo and started to run it through its paces.
Part of my requirements was that it had to play nice with my large prosumer NAS that I use for cheap disk space. Serv-U was working beautifully up to that point, but from what I could tell it wouldn't do impersonation and it relied upon the service credentials to work properly. This wouldn't normally be an issue if the space was located on a Windows server that was on my domain, but the NAS device has never played nicely with active directory and user credentials. So I decided to give Serv-U support a call to see if they had a quick answer.
I placed the call, and in short order was connected to customer service (funny enough, I was talking to the same guy that did the phone tree!). I informed him that I had a quick 30 second presales question and I was ready to purchase his product immediately if I could get a quick answer.
I was shocked when the gentleman told me that they wouldn't take my question because they didn't do presales support via phone and I had to send in an email and would get a response within a couple of days. When I told him (politely, seriously) that I had my credit card in hand and was ready to purchase the product if I could get an answer to my question he balked and told me again that I could only send an email. He actually managed to sound annoyed that I had even called to begin with.
I know, some of you readers may be asking, "Why not just send in an email? He gave you an option! Stop being so unreasonable" Well, a few reasons - for one, I had to get a solution in that day. The second reason, and something more important to me personally - the guy was just plain rude about it. Here I am, a potential customer ready to purchase his product for $1000 - no small sum - and he was annoyed I was calling for a presales question!
Serv-U being out of the running, I installed the WS_FTP demo and it worked beautifully and I purchased it later that day. It was more expensive - in fact, almost $500 more, but I was willing to pay for it if it worked.
And that my friends, is how you lose $1,000 in 30 seconds.
How netflow made my users happy.
Recently I had been receiving automated bandwidth alerts for a couple of our offices, so I decided to take a deeper look at what was generating enough traffic to saturate 4 T1's.
How?
The first technology that came to mind was netflow, but I had absolutely no experience with it. I only knew the basics - it would show me source and destination IP's and the type of traffic being transmitted. I also knew that it worked with flows, unlike SPAN ports which forward all traffic and something I wasn't interested in doing.
Since netflow isn't something that's useful without something collecting the data (netflow data is pushed, not polled), the first thing I did was install ManageEngine's Netflow Analyzer. They have a free version that is good up to 2 monitored interfaces (note this is not the same thing as monitored IP's - so you can theoretically just monitor your routers public interface to get useful data!), also I've used manageengine demos in the past and I've always been pleased with their products.
Once I had the Netflow Analyzer installed it was time to enable netflow on my router - a Cisco 2811 running IP Base 12.4(3f). Initially I was pretty concerned about CPU usage because we're already doing PPP multilink which is handled by the CPU and netflow can be CPU intensive, but Cisco's documentation indicated I should be good. Netflow was surprisingly easy to set up:
router#enable
Password:*****
router#configure terminal
router(config)#interface [INTERFACE YOU WISH TO MONITOR]
router(config-if)#ip route-cache flow
router(config-if)#exit
router(config)#ip flow-export destination [YOUR ANALYZER] 9996
router(config)#ip flow-export source [INTERFACE YOU WISH TO MONITOR]
router(config)#ip flow-export version 9
router(config)#ip flow-cache timeout active 1
router(config)#ip flow-cache timeout inactive 15
Note: If ip flow-export version 9 doesn't work, try version 5
In short order, I was in netflow land. Because it works via streams, it took some time for all the data to come available. Roughly an hour later I already had a big enough picture to start acting (IP addresses obfuscated to protect the guilty):
The top bandwidth destination - 192.168.23.5 - is a NAS device in another office (the other office that we were also having issues with).
All of our offices are connected via MPLS and can talk to each other as if they were on the same network. In every office, we have a NAS device for backups. We use a piece of freeware called Cobian to backup that uses a hard coded path for its destination (it isn't location aware).
As it turned out, we had a couple of users move without informing IT, so we never changed their backup locations:
I did some quick math, and figured out that these two users were maxing out our bandwidth in BOTH offices experiencing regular bandwidth shortages. Holy smokes!
Without netflow, I would have never known this was the issue (I always blamed sporting events 
- and with it, I can take corrective action:
1. By changing the backup drives, we can make this go away.
2. I set up an automatic report which runs nightly that sends an email to the department with all traffic destined to backup drives at the wrong location.
3. We can start looking for backup software that limits bandwidth usage (we've needed a new backup package for awhile).
Bandwidth is no longer in such short supply, which cuts down on latency and makes users happy!
If you have any netflow stories, please comment!
HP ESX Management agents won’t uninstall.
Earlier this week I was attempting to upgrade the HP agents on my ESX servers and on one host, for no particularly good reason I was getting a really weird error:
This script will now attempt to uninstall the HP Insight Manager Agents.
Do you wish to continue? (y/n) y
Uninstalling HP Insight Manager Agents bulletin (hp-classic-mgmt-solution-825.10.1344)
Removing (hp-classic-mgmt-solution-825.10.1344)
This script will now attempt to uninstall the HP Insight Manager Agents.
Do you wish to continue? (y/n) y
[hp-classic-mgmt-solution-v825] bulletin found.
Uninstalling ... [hp-classic-mgmt-solution-v825] bulletin
Removing {hp-classic-mgmt-solution-v825} ...
Encountered error NoMatchError:
The error data is:
Id - hp-classic-mgmt-solution-v825
Message - 1 of 6 Vibs in this bulletin have been superseded. Please try
removing the newest bulletin for this component instead.
Errno - 13
Description - No matching bulletin or VIB was found in the metadata.
Unable to remove {hp-classic-mgmt-solution-v825}. esxupdate status {13}
[ FAILED ]
Exit 0
No matter what I tried, I couldn't get it to go away. I tried removing the RPMs and that wouldn't work. I also tried to remove the bulletin's manually, but I couldn't - it yielded the exact same error as above (turns out the HP utility does anyway is make a call to esxupdate).
A bulletin is essentially a rollup of packages (RPM's in this case) which VMWare calls a VIB or vSphere Installation Bundle. In this particular instance, a single VIB was stopping the process and needed to be removed manually. How though?
I was totally stumped, so I placed a call to VMWare support, and they showed me this nifty UNDOCUMENTED command to list the VIBs:
[root@vistsfo1 831]# esxupdate --vib-view query | grep hp
rpm_hp-smh-templates_8.2.5-51@noarch installed 2009-07-13T10:07:01.211124-07:00
rpm_hp-snmp-agents_8.2.5-50.vmware4x@x86_64 installed 2009-07-13T10:07:01.141505-07:00
cross_hpilo_400.1.1.1.1VMW-00001 retired 2009-07-13T10:07:01.275159-07:00
rpm_vmware-esx-drivers-scsi-hpsa_400.3.6.14.27vmw-1.0.7.193498@x86_64 retired 2009-09-26T22:04:40.219785-07:00
rpm_vmware-esx-drivers-scsi-hpsa_400.3.6.14.28vmw-2vmw.1.9.208167@x86_64 installed 2009-11-27T10:40:13.064463-08:00
rpm_hpsmh_3.0.1-73@x86_64 installed 2009-07-13T10:07:01.490639-07:00
rpm_hp-health_8.2.5-50.vmware4x@x86_64 installed 2009-07-13T10:07:01.224935-07:00
rpm_hp-agents-config_8.2.5-24@noarch installed 2009-07-13T10:07:01.148444-07:00
Great! Now, to remove the offending package. In my case I knew it was the right VIB because it was there were only 2 non driver related VIBs installed.
[root@vistsfo1 831]# esxupdate remove -b rpm_hp-snmp-agents_8.2.5-50.vmware4x@x86_64
So now onto the interesting part. The --vib-view command is a hidden command for esxupdate. It's not in the man page, so how could anyone know about this handy flag without calling VMWare? Well, hindsight brings up and interesting method... open esxupdate on your ESX server. On line 146 (ESX 4U1) you can see this:
# Hidden options
parser.add_option('--HA', action='store_true', help=optparse.SUPPRESS_HELP)
parser.add_option('--vib-view', action='store_true', dest='vibview',
help=optparse.SUPPRESS_HELP)
parser.add_option('--maintenancemode', action='store_true',
help=optparse.SUPPRESS_HELP)
I suppose I could have just looked that the script to find that out!
Hopefully this will help, I saw a few posts out there about it.
Sync Active Directory with Postini
About a year ago I had the opportunity to set up Postini for my organization as a part of my Exchange 2007 setup/migration.
I selected Postini as our anti-spam service - mainly because our users were already used to the interface. One of the biggest challenges during planning was how to keep Postini current with my organizations mailboxes. I could let Postini create an account when an email is received, but that sounded like a really bad idea. Thankfully, in my research I found out about an application that will sync AD to Postini: Google Apps Directory Sync for Email Security.
The tool isn't overly difficult to set up, but it does have a few quirks that I feel I should share - especially since Postini's paid support is awful. We paid $750 for support and when I needed help getting it working I was informed that they didn't support syncing and referred me to knowledge base articles. I managed to figure it out on my own (on time!) and I've had it working perfectly ever since. Below you can find some set up information that may help you. Please keep in mind that this setup works in my environment and may not work in yours.
Once you download the application (link above), you will need to create a "profile" that you will use to run the syncing application.
The first screen you will see is Authentication. This is where you input your administrator login for your Postini organization.
The second screen you will see is Orgs - you can select a specific organization to sync to (for instance, if you had multiple domains). I have it set to "All users in all orgs under your account," and it works perfectly with a 3 org nested setup like mine.
Exclusion filters. I don't use these, but it will allow you to exempt accounts from being syncronized.
LDAP settings: This is where you input your Active Directory information. You can use either LDAP or LDAPS (SSL LDAP). If you opt to use LDAPS, don't forget to change the port to 636. Input a domain controller IP under Host Name, add in your base domain - for example: DC=subdomain,DC=domain,DC=com. Authentication is an account with read access to your AD - use the DOMAIN\user format.
For LDAP User Attributes, select MS Active Directory. The email address attribute is "mail." Alias Address Attributes are "proxyAddresses."
For user sync - this is an important part: If you don't add a search string for users, it will also pull computer accounts. It won't add them, but it will error your log report. I added my Org Name, SUBTREE, (sAMAccountType=805306368) - this will filter out everything, but users. You could add any kind of filter you wanted here. More about LDAP filters.
Mailing Lists - I don't sync them. They are almost all internal only. Since you have to also make a change in Exchange 2007 to allow for non-authenticated users, I make exceptions for dist lists through the web interface for public facing mailing lists. This is up to you.
Notification - This will email you a report every time it runs that tells you if there were any errors, how many users were added/removed, etc. I recommend this.
Delete limit - I have this left at the default 5% - I can see the wisdom in this setting 
Log files - lets you set the file path/limit on the synchronization log. I left as is.
Now you should be done. Go ahead and run the sync simulation. At the bottom of the log in the Sync Log tab, you should see: Simulation completed successfully - good job! Go ahead and save the profile by going to the file menu to a location you will remember easily.
The sync tool will not run automatically on its own - and for that I set up a scheduled task that runs every day at midnight and runs a cmd script that contains the following:
sync-cmd -a -c emailsync.xml
WAIT
sync-cmd is located in your Postini sync tool location - most likely C:\Program Files\Google Apps Directory Sync for Email Security. The -a option commits the changes and the -c option specifies the configuration file you created above. Voila! Every midnight you should get an email letting you know what has changed.
Hope that helps!
Need to move mailboxes quickly?
I'm supposed to have this week off for our company's annual winter break, but like most sysadmins I will end up having to work when everyone else doesn't.
I get the pleasure of moving about 250GB worth of mail around in an effort to get rid of 2 corrupted mailstores. Mailstores that even Microsoft can't fix. I have spent about 15 hours on the phone with Microsoft since Friday - it is now Monday. The original call was to remove a couple of mailboxes that weren't being deleted normally. Somehow a user (it's always the users fault... right? ) of mine managed to create a looped folder in her deleted items box. Imagine a folder structure like this:
Deleted Items
A+
B+
A+
B+
A...B...A...B... for eternity.
I tried everything. Moving the mailbox, disconnecting and reconnecting the mailbox, mfcmapi, pfdavadmin, screaming and pounding on the keyboard. Nothing worked. Microsoft couldn't figure it out either. I now believe that this may be the cause of my latest BackupExec issues alluded to in my previous blog post- a belief that is strong enough that I have actually halted publishing it until I'm sure. Don't get me wrong though, Symantec hasn't made a good product in years...
Microsoft's solution is the same one that I had come up with before I had called them. I have to move all of my mailboxes off the affected mail stores. Which means about 400 mailboxes and 250GB.
Not wanting to sit there and manually move one mailbox at a time, I decided to seek out a powershell script to do the moving for me. I needed something that could read from a CSV file and do a multithreaded (more than one a time) moves. The multithreaded part turned out to be the big problem. Writing a powershell script to move 1 mailbox at a time is something that takes no real effort, but multithreaded requires a little bit more complexity. Since I'm never one to reinvent the wheel, I turned to Google and stumbled (after way too much keyword manipulation) to find a MSExchangeTeam blog article on it.
It allows you to import a CSV file with the following format (I'll save you from trying to find it in the documentation):
Identity,targetmbserver,targetmbsg,targetmbdb
Bob Smith,mailserver,storagegroup,mailboxdatabase
Jane Doe,mailserver,storagegroup,mailboxdatabase
It will then move 4 at a time and display the progress. Perfect. Now if only they could move faster...