LDAPS Integrated SVN with Ubuntu 10.04 LTS
And now for something completely different!
I'm a firm believer in the right tool for the job, and when I was recently placed in charge of a couple of development efforts I wanted to make sure the right tools were in place to do my job effectively. The first tool that I needed was a revision control system so that we could keep track of changes. It used to be that CVS was the common choice, but from my quick and dirty research it looked SVN had taken the top spot (later, as it turns out, there are some pretty heated debates about this... some people will tell you git or mercurial).
Since SVN was going to be the system of choice, I decided to put it together on a Linux based virtual machine - mainly because I'm a Windows systems administrator by trade and I like to keep all of my skills sharpened. Some may say it is foolish - especially with such an important system - to run it on a platform that is unfamiliar. The beauty of subversion is that it runs on more than one platform (Windows, even) and you can migrate easily if needed.
The biggest requirement I had was integrating SVN with LDAP(S). I wanted to be able to control access, but through Active Directory and not some sort of file on my SVN server. As it turns out, it wasn't too difficult to do - once I figured out how to do it 
. Here are the commands in a nutshell (all of these commands are to be run as a root user - sudo or just logged in as root doesn't matter):
Install subversion and apache2 with the SVN module:
apt-get install subversion
apt-get install apache2 libapache2-svn
Enable the SVN Apache module:
a2enmod authnz_ldap
Since I use a self-signed certs and I'm too lazy to install my SSL chain, I have to turn off Apache's checks (you may not want to do this). Add to the end of /etc/apache2/apache2.conf:
LDAPVerifyServerCert Off
After troubleshooting, you need to set this in your /etc/ldap/ldap.conf file to avoid errors similar to "[warn] [client x.x.x.x] [636] auth_ldap authenticate: user foo authentication failed; URI /secret [ldap_search_ext_s() for user failed][Operations error]"
REFERRALS off
Now, you need to add it to your apache2 configuration so that you can access it via http. You will need to edit this to suit your own needs, I hope it's relatively self-explanatory, but I've made some comments in red. Add this after your last </directory> statement:
DAV svn
SVNParentPath /data/svn - Your SVN repository data
SVNListparentPath on
SVNAutoversioning On
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthName "svn"
AuthUserFile /dev/null
AuthLDAPURL "ldaps://DOMAINCONTROLLER/DC=domain,DC=com?sAMAccountName?sub?(objectClass=*)" - You want this pointed at your base DN. Also, some (even most) of you may not be running LDAPS (SSL LDAP) - you may need to change this to ldap://.
AuthLDAPBindDN "DOMAIN\User" - Any user will do, since by default Windows allows any user to query active directory for auth. I suggest making a separate user for just this.
AuthLDAPBindPassword Password
AuthLDAPGroupAttributeIsDN on
AuthLDAPGroupAttribute member
Require ldap-group cn=svn,ou=securitygroups,ou=IT,dc=domain,dc=com - This is optional, but allows you to limit SVN access to a specific group.
Once you're done, it's time to create your first repository:
svncreate /data/svn firstrepository
Now you need to change the owner to www-data so apache2 can read/write it:
chown -R www-data svn
chgrp -R www-data svn
chmod -R g+rws svn
And voila! You should be able to go to http://yourIPorHostname/svn and view your first repository!
iPhone users can’t hide from netflow!
I couldn't resist blogging about this one.
Last Friday I wrote a blog post about how netflow was making my users happy. This week I'm going to talk about how netflow is still making my users happy - except for one.
After a few days of watching CPU usage on my routers, I added a second site to my netflow collector and started investigating.
The first thing I noticed was more users backing up to the wrong location (we're now performing an audit). The second thing I noticed was someone transferring a lot of data from Akamai - and I started looking into what it was.
The IP in question didn't have a DNS entry, so I got curious. I used nmap to find out more information and lo and behold:
Running: Apple iPhone OS 3.X
OS details: Apple iPhone mobile phone (iPhone OS 3.0 - 3.0.1)
An iPhone?!
After realizing it was an iphone, and not even a user's computer - I decided to take action against the offending device. After all, we're having bandwidth issues! Because we're using DHCP, I took a look at our DHCP leases to find out the MAC address of the iPhone:
Now that I had the MAC address, I could send it to the /dev/null blackhole. I logged into my wireless access point and went to work. I would outline the CLI commands, but I must confess to using the web interface. As you can tell from these instructions, it's much easier (on the surface) to do it that way. Also, as it turns out Cisco WAP's don't play nice if you configure through both the GUI and CLI.
Essentially what I did was create a filter for the particular offending MAC address. If you are doing this yourself, be careful! By default it will set the Default Action to "Block All," meaning you will knock everyone off.
This is how I set it up:
Yeah, I know - I'm a terrible person - the poor user can't get on our ultra fast (not really) wireless network! Well, the way I see it is there isn't a single business use case where a user would need to transfer 120+ MB worth of data to their phone. That's what the cell phone carrier network is for! Let AT&T handle it says I!
How to lose $1,000 in 30 seconds.
A little over a month ago I was asked to set up a SFTP server so that our clients could transfer files securely. SFTP is a bit of a misnomer, you would expect it to be a subset of FTP, but it's not. SFTP is actually its own protocol designed as an extension to SSH. The further confuse the issue, SFTP is not the same as FTPS - a little used SSL version of FTP.
I knew that SFTP was more on the Linux side of things, so I decided at first to use linux for this. Even though I'm primarily a Windows shop, I firmly believe in the right tool for the job. I even got a bit of budget to buy RHEL for it. Unfortunately, as I came to find out after trying to set it up, there is a limitation that had me running to another solution. The limitation was part of a requirement handed to me - I couldn't allow our clients to traverse directories and get a list of our other clients. The way openssh implements chroot allows for this to happen and there's no way around it. Your SFTP users will end up in their own directory, but a simple "cd ../ls" will show them a list of your clients (or root directory). I later read that proftpd may not do that, but by then I had opted for a Windows solution.
There are a number of products out there that will do SFTP on Windows - some free, some not. Because this would be something I would be running in a production environment that is client facing, the solution had to include a support option. This narrowed my choices to Serv-U and WS_FTP. I've used both in the past and I always had a pretty decent impression of Serv-U so I installed a demo and started to run it through its paces.
Part of my requirements was that it had to play nice with my large prosumer NAS that I use for cheap disk space. Serv-U was working beautifully up to that point, but from what I could tell it wouldn't do impersonation and it relied upon the service credentials to work properly. This wouldn't normally be an issue if the space was located on a Windows server that was on my domain, but the NAS device has never played nicely with active directory and user credentials. So I decided to give Serv-U support a call to see if they had a quick answer.
I placed the call, and in short order was connected to customer service (funny enough, I was talking to the same guy that did the phone tree!). I informed him that I had a quick 30 second presales question and I was ready to purchase his product immediately if I could get a quick answer.
I was shocked when the gentleman told me that they wouldn't take my question because they didn't do presales support via phone and I had to send in an email and would get a response within a couple of days. When I told him (politely, seriously) that I had my credit card in hand and was ready to purchase the product if I could get an answer to my question he balked and told me again that I could only send an email. He actually managed to sound annoyed that I had even called to begin with.
I know, some of you readers may be asking, "Why not just send in an email? He gave you an option! Stop being so unreasonable" Well, a few reasons - for one, I had to get a solution in that day. The second reason, and something more important to me personally - the guy was just plain rude about it. Here I am, a potential customer ready to purchase his product for $1000 - no small sum - and he was annoyed I was calling for a presales question!
Serv-U being out of the running, I installed the WS_FTP demo and it worked beautifully and I purchased it later that day. It was more expensive - in fact, almost $500 more, but I was willing to pay for it if it worked.
And that my friends, is how you lose $1,000 in 30 seconds.
How netflow made my users happy.
Recently I had been receiving automated bandwidth alerts for a couple of our offices, so I decided to take a deeper look at what was generating enough traffic to saturate 4 T1's.
How?
The first technology that came to mind was netflow, but I had absolutely no experience with it. I only knew the basics - it would show me source and destination IP's and the type of traffic being transmitted. I also knew that it worked with flows, unlike SPAN ports which forward all traffic and something I wasn't interested in doing.
Since netflow isn't something that's useful without something collecting the data (netflow data is pushed, not polled), the first thing I did was install ManageEngine's Netflow Analyzer. They have a free version that is good up to 2 monitored interfaces (note this is not the same thing as monitored IP's - so you can theoretically just monitor your routers public interface to get useful data!), also I've used manageengine demos in the past and I've always been pleased with their products.
Once I had the Netflow Analyzer installed it was time to enable netflow on my router - a Cisco 2811 running IP Base 12.4(3f). Initially I was pretty concerned about CPU usage because we're already doing PPP multilink which is handled by the CPU and netflow can be CPU intensive, but Cisco's documentation indicated I should be good. Netflow was surprisingly easy to set up:
router#enable
Password:*****
router#configure terminal
router(config)#interface [INTERFACE YOU WISH TO MONITOR]
router(config-if)#ip route-cache flow
router(config-if)#exit
router(config)#ip flow-export destination [YOUR ANALYZER] 9996
router(config)#ip flow-export source [INTERFACE YOU WISH TO MONITOR]
router(config)#ip flow-export version 9
router(config)#ip flow-cache timeout active 1
router(config)#ip flow-cache timeout inactive 15
Note: If ip flow-export version 9 doesn't work, try version 5
In short order, I was in netflow land. Because it works via streams, it took some time for all the data to come available. Roughly an hour later I already had a big enough picture to start acting (IP addresses obfuscated to protect the guilty):
The top bandwidth destination - 192.168.23.5 - is a NAS device in another office (the other office that we were also having issues with).
All of our offices are connected via MPLS and can talk to each other as if they were on the same network. In every office, we have a NAS device for backups. We use a piece of freeware called Cobian to backup that uses a hard coded path for its destination (it isn't location aware).
As it turned out, we had a couple of users move without informing IT, so we never changed their backup locations:
I did some quick math, and figured out that these two users were maxing out our bandwidth in BOTH offices experiencing regular bandwidth shortages. Holy smokes!
Without netflow, I would have never known this was the issue (I always blamed sporting events 
- and with it, I can take corrective action:
1. By changing the backup drives, we can make this go away.
2. I set up an automatic report which runs nightly that sends an email to the department with all traffic destined to backup drives at the wrong location.
3. We can start looking for backup software that limits bandwidth usage (we've needed a new backup package for awhile).
Bandwidth is no longer in such short supply, which cuts down on latency and makes users happy!
If you have any netflow stories, please comment!
HP ESX Management agents won’t uninstall.
Earlier this week I was attempting to upgrade the HP agents on my ESX servers and on one host, for no particularly good reason I was getting a really weird error:
This script will now attempt to uninstall the HP Insight Manager Agents.
Do you wish to continue? (y/n) y
Uninstalling HP Insight Manager Agents bulletin (hp-classic-mgmt-solution-825.10.1344)
Removing (hp-classic-mgmt-solution-825.10.1344)
This script will now attempt to uninstall the HP Insight Manager Agents.
Do you wish to continue? (y/n) y
[hp-classic-mgmt-solution-v825] bulletin found.
Uninstalling ... [hp-classic-mgmt-solution-v825] bulletin
Removing {hp-classic-mgmt-solution-v825} ...
Encountered error NoMatchError:
The error data is:
Id - hp-classic-mgmt-solution-v825
Message - 1 of 6 Vibs in this bulletin have been superseded. Please try
removing the newest bulletin for this component instead.
Errno - 13
Description - No matching bulletin or VIB was found in the metadata.
Unable to remove {hp-classic-mgmt-solution-v825}. esxupdate status {13}
[ FAILED ]
Exit 0
No matter what I tried, I couldn't get it to go away. I tried removing the RPMs and that wouldn't work. I also tried to remove the bulletin's manually, but I couldn't - it yielded the exact same error as above (turns out the HP utility does anyway is make a call to esxupdate).
A bulletin is essentially a rollup of packages (RPM's in this case) which VMWare calls a VIB or vSphere Installation Bundle. In this particular instance, a single VIB was stopping the process and needed to be removed manually. How though?
I was totally stumped, so I placed a call to VMWare support, and they showed me this nifty UNDOCUMENTED command to list the VIBs:
[root@vistsfo1 831]# esxupdate --vib-view query | grep hp
rpm_hp-smh-templates_8.2.5-51@noarch installed 2009-07-13T10:07:01.211124-07:00
rpm_hp-snmp-agents_8.2.5-50.vmware4x@x86_64 installed 2009-07-13T10:07:01.141505-07:00
cross_hpilo_400.1.1.1.1VMW-00001 retired 2009-07-13T10:07:01.275159-07:00
rpm_vmware-esx-drivers-scsi-hpsa_400.3.6.14.27vmw-1.0.7.193498@x86_64 retired 2009-09-26T22:04:40.219785-07:00
rpm_vmware-esx-drivers-scsi-hpsa_400.3.6.14.28vmw-2vmw.1.9.208167@x86_64 installed 2009-11-27T10:40:13.064463-08:00
rpm_hpsmh_3.0.1-73@x86_64 installed 2009-07-13T10:07:01.490639-07:00
rpm_hp-health_8.2.5-50.vmware4x@x86_64 installed 2009-07-13T10:07:01.224935-07:00
rpm_hp-agents-config_8.2.5-24@noarch installed 2009-07-13T10:07:01.148444-07:00
Great! Now, to remove the offending package. In my case I knew it was the right VIB because it was there were only 2 non driver related VIBs installed.
[root@vistsfo1 831]# esxupdate remove -b rpm_hp-snmp-agents_8.2.5-50.vmware4x@x86_64
So now onto the interesting part. The --vib-view command is a hidden command for esxupdate. It's not in the man page, so how could anyone know about this handy flag without calling VMWare? Well, hindsight brings up and interesting method... open esxupdate on your ESX server. On line 146 (ESX 4U1) you can see this:
# Hidden options
parser.add_option('--HA', action='store_true', help=optparse.SUPPRESS_HELP)
parser.add_option('--vib-view', action='store_true', dest='vibview',
help=optparse.SUPPRESS_HELP)
parser.add_option('--maintenancemode', action='store_true',
help=optparse.SUPPRESS_HELP)
I suppose I could have just looked that the script to find that out!
Hopefully this will help, I saw a few posts out there about it.