There is a ‘talk franchise’ that has started titled ‘Switches Get Stitches.’ Started by Eireann Leverett and Colin Cassidy, it showcases problems in industrial network switch hardware and firmware. Digital Bond Labs offers a humble contribution to the cause: a demonstration of a firmware rootkit for an (admittedly somewhat dated) industrial switch. If you are attending Defcon 23, be sure to check out the ‘official’ SGS talk there.
One of the components in this year’s ICS Village CTF is going to be pretty unique: we have modified a network switch firmware. This ends up giving a lot of interesting leeway: we can now mangle packets, talk to a command and control server, and make a few other interesting flags for participants to find.
Most ICS equipment lacks any kind of firmware protection. Scarier is the fact that some operators, including a very small subset of utility operators, purchase safety-critical equipment from dubious sources such as eBay.
So, let’s take apart a network switch and show just how easy it is to trojan a device!
Embedded versus Deeply Embedded
The word ‘embedded system’ is bandied about a lot in the world. In my mind it always means a system whose software (firmware) is tightly coupled to the hardware. As an old colleague is fond of saying, a new reality is emerging: there are big computers and little computers, but fewer and fewer are what I would call ‘embedded’.
Thanks to a much wider adoption of operating systems such as Linux and BSD, particularly in the network device world, the tight coupling of programs to hardware is a little less prevelant. In reality, a lot of newer ‘embedded systems’ are just small computers, which, modulo the tight coupling of their bootloaders to the hardware, allow us to think of trojans and firmware rootkits at a higher level.
There is a whole other series of techniques used for the ‘deeply embedded’ stuff, which maybe would be good material for another blog post.
Gaining a shell
For this experiment, let’s look at an industrial Ethernet switch. The Sixnet EtherTRAK line caught my eye a few years ago: it is a managed Ethernet switch, it runs Linux (it has the Linux penguin logo right on the front), and it can regularly be found on eBay for cheap. Sixnet decided on Linux as the operating system for this line of switches and promised, a long time ago, to open up an SDK so that end users could install their own data management software on the switch. Alas, the SDK was lost to time, but the switch works so we’ll see what we can do to, er, improve it.
If you participated in the ICS Village CTF in 2014, you may have run into this switch. You also might have been the person who succeeded in changing the password on me, which took me a little while to figure out how to recover…
Anway, we know that the switch runs Linux, which is nice. So, we flit about the web interface and see if we can find any vulnerabilities that will let us root it.
It has the ability to backup and restore configuration data. This is commonly a poorly-done thing, and that is the case here. We can download a backup of the device configuration, which includes the Apache configuration. Looking at the Apache configuration, the web server is already configured to allow symbolic links, which is very convenient for us. So, we try to add a file to the configuration backup archive: the file will be located in the webserver root directory and will be a symlink to /etc/passwd.
That works! We are able to retrieve the /etc/passwd file from the device. (As an aside, the device actually runs the lighttpd webserver, but we got lucky and this trick still worked).
We notice that the default accounts use shells that restrict user activity to configuring the device. We’d like to add an account that lets us login and have a real command shell. So we build a new file, hopefully without breaking anything on the switch. We add a new user, ‘reid’ to the end of the passwd file and copy the hashed default password (which is ‘admin’), giving the new account uid 0 (root privileges). Using the same trick that we used to retrieve the passwd file, we will place a modified /etc/passwd into the configuration backup archive which contains a new user account. We didn’t want to do this as a first step in case something important was in that file that needed to be preserved. The new user account will hopefully allow us to login with administrative privileges and will give us a shell if all goes well.
Success!
Now we have the ability to log in to the device and to have a root shell. We look around and see that it is a typical ‘embedded’ Linux system running BusyBox. Thankfully, Sixnet even left us the netcat utility compiled for this particular distribution, as well as a native gdbserver. This is all that we could hope for if we want to maintain access to the system. We have 3.8MB of free space on which to put additional tools and scripts to maintain access across a variety of scenarios.
So, we build a cross-compiler toolchain and grab the Linux 2.6.21 source tree. We then compile up a tap/tun driver and install one of my favorite toys: iodine. The ‘tun’ kernel driver compiles to be about 45kB for ARM, and an iodine client takes only 70kB. That leaves us with 3.7MB still for other tools. ICMP tunneling software, perhaps a copy of nmap, and maybe even some exploitation tools will also be installed. We even have a 2MB ram filesystem for use for scratch, which is unused by the stock firmware.
This network switch is now a small weapons platform, no soldering required. If we had malicious intent, we might try and resell it, and then wait for it ‘phone home’ so that we could begin scanning or attacking an industrial network.
A wired network switch is one thing, but imagine applying this technique to a device with cellular communication…
Lessons to Learn
Unsigned firmware isn’t anything new. Digital Bond Alum Daniel Peck was the first in the field to really look at this issue with any eye toward informing the ICS security community. Very few vendors have started digital firmware signing programs, and those that do so are still in the teething stages of the process.
A trouble with simply ‘signing updates’, which is a predominant practice among device vendors, is that this post-update exploitation is still possible. So, for end users who are doing critical work it is important to always buy hardware from trusted sources.
In the case of the Sixnet switch, adding a firmware signing program to the switch after-the-fact turns out to be pretty hard. The device only has one flash memory: the main NAND flash. Configuration is stored on the main filesystem. Verification of the system at boot-time would have to exclude all of the device configuration files, some of which may be useful for rootkit deployment. There is no secure memory in the device to act as a trusted bootloader, so if the device is compromised, an attacker could overwrite this attestation code with their own code anyway. A far better choice, security-wise, would be to split the firmware and the configuration filesystems, and to use a processor that has write-protectable memory to use as a secure bootloader. That way, a secure bootloader could at least verify the integrity of the operating system. A followup exercise would be ensuring that none of the configuration data could be used to execute code.
Corey Doctorow already wrote a nice three-item wishlist, and I concur: actually having the signed firmware that has been reviewed is great, but even better would be his #3 item: tools to allow us to verify that firmware on embedded systems have not been tampered with.
While more deeply embedded systems will be a lot harder to exploit than this Sixnet switch, they too can be modified. Perhaps the changes could not be made on a Friday afternoon of downtime, but with enough funding that is not really a problem for an attacker.
tree root image by atoach, switch image by sixnet