Sometimes I bust so many caps I feel like I’m Woka Flokca:
Today I had to bust a cap on an Acme Packet 3820. We had been having a strange, intermittent one-way audio problem plaguing us for some time. All the SIP traces had looked good on the call examples, and we were unable to re-produce the issue. So, time went on, and once in awhile we would hear about another one. Always from the same guy, and always a wireless caller calling in from the PSTN to him.
This is one of the most frustrating things in IT. Intermittent problems with no real pattern that you cannot re-produce. Well, we had our breakthrough a couple of days ago. We had a few more people reporting problems, and found a pattern to it. It was happening only to the people who had mobility (Single Number Reach) enabled on their phones. We were able to re-produce it with the client, so I set up a test phone to mirror their configuration, and sure enough, I could re-produce it on my test phone, too. Jackpot!
So, I started to look at the traces. It was quite the call flow between 4 pairs of SBC’s, a CUCM SME cluster, a Siemens OSV, and a CUCM Leaf cluster. SIP messaging was bouncing around like hollow points ricocheting.
We were able to simplify it a little bit – get it down to 2 SBC pairs, take the OSV out of the call flow, and modify the route group so at least we could chose the SBC pair for the outbound call to the remote destination.
After doing this, and making several test calls, it was still very hit or miss. One in 4 or so would fail, with no real pattern to it. All the signaling looked good when scrutinizing the SDP. So what next?
I did what any good little UC engineer would do, and started busting caps. First, I put the 6945 I was using for testing on to a switch I had hanging out with me, and did a little monitor-session config:
First – filter those nasty BPDU’s from your uplink port so you don’t put the port on switch in the closet into an errdisable state:
switchport mode access
power inline never
spanning-tree bpdufilter enable
Then, pick your nose, and the ports that you want to use:
no monitor session 1
monitor session 1 source interface FastEthernet 0/1 both
monitor session 1 destination interface Fa0/2
Now, it was time to shark my wire, and what did I see? RTP in both directions:
Digging into the details – those little UDP packets were headed for the right port on the SBC, so it was time to capture on the other side.
Here’s what I did to send a copy of all the packets in/out of the SBC to my laptop:
3820_HOSTNAME# configure terminal
3820_HOSTNAME(capture-receiver)# state enabled
3820_HOSTNAME(capture-receiver)# address 10.98.1.198
3820_HOSTNAME(capture-receiver)# network-interface s1p0:0
Save Changes [y/n]?: y
last-modified-date 2015-04-25 18:57:06
Save-Config received, processing.
waiting for request to finish
Request to ‘SAVE-CONFIG’ has Finished,
Currently active and saved configurations do not match!
To sync & activate, run ‘activate-config’ or ‘reboot activate’.
Activate-Config received, processing.
waiting for request to finish
Request to ‘ACTIVATE-CONFIG’ has Finished,
3820_HOSTNAME# packet-trace remote start s1p0 10.98.61.45
Trace started for 10.98.61.45
3820_HOSTNAME# packet-trace remote start s0p0 192.168.205.12
Trace started for 192.168.205.12