So I was recently tasked with building out a UAT (user acceptance testing) environment for a customer. This is basically an exact replica of the production environment we have just spent 2 years designing and implementing. This allows the customer to test any upgrades, changes, or patches before actually implementing them into production. Plus its pretty cool if you are doing some dicey shit and wanna make sure you don’t bring the entire enterprise to its knees in the middle of the day.
Now to the problem. 30% or so of the UAT environment was already built out using a SU 1 version of call manager. Now you may say “who fucking cares”? Well guess what, I do. Cisco doesn’t publish a bootable version of the SU 1 version of call manager. Now when you are adding new subscribers to an existing publisher, the versions of call manager have to be exactly the same, or the new subscribers will shit all over themselves. So we came up with the brilliant idea to switch versions on all the UAT servers that were currently up and running. (I have a bootable ISO for this version). Once I got all the existing servers back to the old version, I could install all the new servers, do a switch version at the CLI of the existing servers, and then do a simple upgrade on all the new servers to the SU 1 version of Call Manager. Sounds easy right? Well it is pretty fucking easy, the only problem is it doesn’t fucking work. I got the entire environment built out, ran the switch version commands, then set off the upgrades. Came back and hour later to find all 20 or so of my new servers had failed the upgrade process.
So back to square one. I deleted all those new servers. So started noodling this shit some more. Now I am sure we have all seen this screen when building out a new environment.
If you are like me, you typically just tab right through this shit. But hey, this sounds like my scenario right? I have existing servers running a SU 1 version. I am installing new subscribers and I am lacking the bootable ISO. So why not install the bootable version, then press yes on this screen and apply the upgrade at the end. So again this sounds easy right? Fuckin ehh its easy. Now here is the issue. I am building 25 servers, there is no fucking way I am doing this without answer files. I could not for the life of me find a way to apply this upgrade patch, and use answer files at the same time. So i said fuck it, I will build one by hand without using the answer files and just see if it works. Guess what, that shit still failed. I was all out of ideas here. Did i mention I only had 4 days to build out and fully test this giant UAT environment. I had just spend 1 and a half of those 4 days fucking around with this stupid shit.
Now if you scroll down a few blog posts and see the post entitled http://www.backdoorburglar.com/?p=144….you will see that we had a major issue with deleting a few production servers ( no big deal right)….haha this was actually a huge fucking deal, but that’s an entirely different matter. The reason I am bringing this up is because during the huge debacle, we actually got Cisco TAC to confirm that you should not apply an upgrade patch during the install ( then why the fuck is that even on the install page?). This also prompted them to give us the bootable ISO of this SU 1 version of Call Manager that I was sorely in need of. So I went back and deleted all the new servers I had built (AGAIN) and rebuilt them using that bootable ISO. Now you may be thinking to yourself “what a lazy fucker, why didn’t he troubleshoot the issue of the failed upgrades instead of being a big vagina and taking the easy way out using the bootable ISO?” To that I say “fuck you” I was trying to cram 10 lbs of shit into a 5lb bag by trying to build out this ginormous environment in 4 days. I had already spent 1.5 days getting absolutely nowhere. Sometimes you gotta know when to hold em and know when to fold em.
I was gonna do a whole giant write up on travel tips…but I don’t really fucking feel like it right now. So I have decided I will do a weekly travel tip.
So I typically travel at least 2 weeks per month….more like 4 weeks per month on this project. I thought I would share my tips and tricks for traveling.
1. Pick a company and stick with that shit. I see a lot of engineers using different hotel chains, airlines, and rental cars every week…this is pure fucking madness. My wife and I are flying to Cancun and staying in a 5 star resort in a couple months (at no cost to me) all because I stay with the same hotel chain and airline for all my trips. The reward points add up very quickly and you need to stop being a fucking tard and take advantage of this. I get upgraded to the nicest room available when I check into my hotel, I always get upgraded to a very nice rental car, and once I hit the next level on my Sky miles status, I will be upgraded to First Class while flying. If you are going to be away from your family and living on the road…you might as well live like a BOSS.
You may only be using telnet if…
…..you enable SSH version 2 but forget to add “(config)#
You might have a good project manager if she is on the phone with her mortgage company discussing mortgage insurance and can bring that man to tears in under 5 minutes for not doing his job.
We recently deployed a shit ton of Cisco 7841 phones. We ran 95% of the ATP successfully and then got to the best part. We get to knock down the WAN and send the phones into SRST mode. For the first 10 minutes everything looked peachy, then all of a sudden the SIP registrations on the gateway decided to be punk ass bitches. We were holding steady at around 200 registrations when we had close to 600 SIP endpoints. Three of us stared at Secure CRT with the same anticipation as a 6 year old on Christmas morning. Unfortunately we suffered the same disappointment as most kids who ask for a bike for Christmas and then received a box full of beaver droppings. The registrations stopped and then at one point started losing registrations. This was some bull shit. It was late and we wanted to go drinking. Well of course after an hour of troubleshooting this stupid shit, I made the brilliant assessment that I should probably call the “experts” at Cisco TAC. So after 5 hours on the call with backbone support and then getting developers involved, we came to the conclusion that the firmware load we are using has issues with SRST.
SO FYI this load “sip78xx.10-1-1SR2-1” can lick my sack.
We added this beauty “sip78xx.10-1-1ES7” and guess what….those little bitches registered. However, we didn’t want to introduce a new load cluster wide without doing a major bug scrub. So we rolled back and decided that if the system goes into SRST mode…some of those people are just fucked.
Recently, I did a project to spin up a new UC environment consisting of a SME cluster, multiple CUCM leaf clusters, and 2 big, bad-ass Unity Connections boxes. We were using the very special 18.104.22.16812-14.sgn.iso image. Well, the SME and CUCM publishers came up just fine with this ISO, but Unity was being a little bitch. The install completeed OK, and I was able to sign into the CLI via the console on the vsphere client, but not via SSH. When I tried to log in to the web administration page, however, I got a “Database Communication Error”. After about an hour I no longer get the Database Error – I simply got a 404 not found. I tried rebuilding that mammerjammer 3 different times using the following OVA’s:
After some research, I have found that when installing this version of Unity, it is necessary to install the 22.214.171.12412-9 version and do an upgrade to the -14 version.
This will most of the time just be an issue when building out a cluster for THE MAN as most normal customers have taken 8.6 and thrown it into a box with their giant Zach Morris cell phone, 1988 Z-28 Camaro, and spare can of Jolt Cola.
If you plug in your new PRI connection and it will not come up
….you may have plugged the PRI into a Gigabit interface.