So I was recently tasked with building out a UAT (user acceptance testing) environment for a customer. This is basically an exact replica of the production environment we have just spent 2 years designing and implementing. This allows the customer to test any upgrades, changes, or patches before actually implementing them into production. Plus its pretty cool if you are doing some dicey shit and wanna make sure you don’t bring the entire enterprise to its knees in the middle of the day.
Now to the problem. 30% or so of the UAT environment was already built out using a SU 1 version of call manager. Now you may say “who fucking cares”? Well guess what, I do. Cisco doesn’t publish a bootable version of the SU 1 version of call manager. Now when you are adding new subscribers to an existing publisher, the versions of call manager have to be exactly the same, or the new subscribers will shit all over themselves. So we came up with the brilliant idea to switch versions on all the UAT servers that were currently up and running. (I have a bootable ISO for this version). Once I got all the existing servers back to the old version, I could install all the new servers, do a switch version at the CLI of the existing servers, and then do a simple upgrade on all the new servers to the SU 1 version of Call Manager. Sounds easy right? Well it is pretty fucking easy, the only problem is it doesn’t fucking work. I got the entire environment built out, ran the switch version commands, then set off the upgrades. Came back and hour later to find all 20 or so of my new servers had failed the upgrade process.
So back to square one. I deleted all those new servers. So started noodling this shit some more. Now I am sure we have all seen this screen when building out a new environment.
If you are like me, you typically just tab right through this shit. But hey, this sounds like my scenario right? I have existing servers running a SU 1 version. I am installing new subscribers and I am lacking the bootable ISO. So why not install the bootable version, then press yes on this screen and apply the upgrade at the end. So again this sounds easy right? Fuckin ehh its easy. Now here is the issue. I am building 25 servers, there is no fucking way I am doing this without answer files. I could not for the life of me find a way to apply this upgrade patch, and use answer files at the same time. So i said fuck it, I will build one by hand without using the answer files and just see if it works. Guess what, that shit still failed. I was all out of ideas here. Did i mention I only had 4 days to build out and fully test this giant UAT environment. I had just spend 1 and a half of those 4 days fucking around with this stupid shit.
Now if you scroll down a few blog posts and see the post entitled http://www.backdoorburglar.com/?p=144….you will see that we had a major issue with deleting a few production servers ( no big deal right)….haha this was actually a huge fucking deal, but that’s an entirely different matter. The reason I am bringing this up is because during the huge debacle, we actually got Cisco TAC to confirm that you should not apply an upgrade patch during the install ( then why the fuck is that even on the install page?). This also prompted them to give us the bootable ISO of this SU 1 version of Call Manager that I was sorely in need of. So I went back and deleted all the new servers I had built (AGAIN) and rebuilt them using that bootable ISO. Now you may be thinking to yourself “what a lazy fucker, why didn’t he troubleshoot the issue of the failed upgrades instead of being a big vagina and taking the easy way out using the bootable ISO?” To that I say “fuck you” I was trying to cram 10 lbs of shit into a 5lb bag by trying to build out this ginormous environment in 4 days. I had already spent 1.5 days getting absolutely nowhere. Sometimes you gotta know when to hold em and know when to fold em.