This was an article I was working on whilst reverse engineering a Sonus SBC that ended up electrocuting me. So I’ve kinda fallen out of love with the project for now, Part 2 in the future? Maybe.
For those that aren’t aware of Sonus/Ribbon devices, many SBCs include a small internal log cache for troubleshooting. You simply download these and import them into your favourite analysis tools like LX or “NewSyslog” (Great name Audiocodes…) or maybe even my Search-UxCallLog utility.
But there are times, especially on the lower end devices like the SBC 1000 whereas a cost-cutting measure there is no local storage, despite there being locations on the board for PCI and USB DOM (Disk on Module) devices, there are no headers installed.
Sonus offers a version of their devices with local storage, known as ‘eUSB’ but I’ve not personally seen any in the wild. A bit of googling shows that there are common modules both in a 2x5pin header format and a mini-PCI module format
Sure, you can configure LX to be a Syslog server and set up remote logging, but it’s a pain in the ass, the SBC is usually somewhere far away from the device you’re administering, and LX doesn’t play nice with multiple users on the same management host. So if you install it for yourself, you break it for the rest of your team. If you have an ASM, you can just install LX there and let it log for you. but they will likely trip the support alarm on the ASM and if you install it on your machine, you’re stuck connected to the customer’s VPN until whatever issue your troubleshooting repeats.
What are we supposed to do then?
Well, a recent tweet from John Cook and a call to action from Alex Holmeset reminded me that Sonus Support will sometimes ask for a USB stick to be inserted for dumping crashdumps and that the SBC supports a USB stick for the AD Cache
After looking here, I can see that the SBC does indeed support USB flash drives. They must be FAT 32 formatted and over 1GB in size. So we know that part should work. Now we just need it to think it has an eUSB module.
Of course, the first thing I did was, email my contact at Ribbon asking for help. But I’m impatient, it was the weekend and I was putting off the 50 other projects I’m working on, including certain videos. So I kept digging.
As we can see on both my lab SBC1Ks, I don’t have an eUSB module.
I found a spare USB key, formatted it as fat32 and fired up LX
And sure enough, it popped up in the logs.
Interestingly we can see the SBC starts a “Fake Alarm” service when the USB is attached, from what I can tell it does this to set the Alarm LED green when it mounts the USB device.
We can also see that the device appears in the partition table
After having the USB attached for a minute or two, we can see the SBC creates a folder called cap which I assume are for core dumps (it also creates dropbox on reboot), but there are no additional options in logging and the eUSB flag is still “No”
Alright then, let’s restart the SBC itself,
Bugger.
No Bueno, Bugger.
Can we trick it?
Okay then, let’s capture everything the SBC outputs on boot via Syslog. Hopefully, we can find something juicy in those log messages we can use to our advantage
The first thing I notice is this message pops up constantly, probably something to do with the fact I put Doom on this one a while ago
I can also see that the SBC notes I don’t have Music on Hold… Hmmm
And of course, it finds the USB on boot
I can also see the SBA module is aware of the USB and broadcasts this over the SMBus
But still nothing of interest after trawling the whole log file.
So I created some folders with paths that were in error messages and tried again (Spoiler, it didn’t work)
Getting to the hardware level
This is the part where I warn you, VERY SERIOUSLY that there is an exposed high voltage power supply in here!
A single mistake and you MAY NEVER HAVE THE OPPERTUNITY TO WORK ON THESE AGAIN.
The below content is so you DONT have to take an SBC apart yourself and is NOT A GUIDE.
If you arent comfortable working at high voltages. DO NOT ATTEMPT TO REPLICATE anything in this article.
Fine then, with nothing helpful in the normal logs, I decided to look for a debug header to watch the boot process.
Once we remove everything from the SBC (and unplug most of the fans) we are left with an open case, the logic board and a pretty dangerous power supply. I have four kids so appropriate safety measures were deployed
There were a few pins of interest, what appears like a mini PCI style slot for an eUSB/DOM. Some debug style headers, a jumper set of 3 pins on the front labelled JP1 (one of which has negative 5v on it) and the ASM, interconnect bus.
I took the liberty of reverse-engineering the motherboard and assumed some interesting things from my findings.
- The SBC runs a 650Mhz ARM CPU with 2GB of DDR2 RAM and 512MB of Flash Storage and an internal 8 port Ethernet Switch
- The DSP cards appear to be connected internally via I2C and Ethernet
- As are the Telephony cards
- These cards appear to share some DDR1 memory with the main CPU using a DMA controller.
Time to break out the logic analyzer
Eventually, after enough poking around, I found a few interesting buses on the ASM connector. Including what appeared to be serial and SPI.
Unfortunately, after a fair bit of faffing about, I couldn’t make sense of the packet captures across the ASM bus. Whilst probing around the board I ended up giving myself a good shock on the exposed power supply. So much for that protection paper…
Summary
Could I find an easy way to add USB logging? No. but we did learn some interesting things about the internals of the Sonus SBC.
I plan on revisiting this in the future, probably once I get my hands on an SBC with an eUSB module installed. I’ve just gotta find one at a reasonable price or once I finish the Teams SIP gateway stuff.
Anyway, hope this helps someone.