The Switches Get Stitches crew has been hard at work on quite a few switching projects. Indeed they released a new exploit tool against GE and GarrettCom switches early this morning, after attempting to get a fix for a Denial of Service bug for at least one year according to the team.
Backdoors, authentication bypass, and lack of firmware signing are all fine and good, but we wonder: what can you really do with this kind of access?
It turns out, quite nasty things.
Most end users who are taking a strong defensive stance on their networks are deploying IDS or NSM. Feeding these systems with data requires configuring various managed switches around the network to mirror traffic from switch ports.
One of the configuration questions that comes up with NMS and IDS is, what to mirror? Most switch manufacturers document their SPAN and Mirroring configuration guidelines with a small caveat: if you are mirroring more than one source port to a given destination port, you may end up dropping packets on the mirror (if, for example, all of your sources are saturating a 1Gb link, then the 1Gb mirror port cannot receive data quickly enough to see everything).
So when IDS and NMS systems are deployed, we often set them up like this:
The assumption being that field devices won’t be attacking each other directly — attacks should result in some observable traffic from an upstream system.
When we developed our first persistent root access to an ethernet switch in June, we began exploring the switching fabric to see what could be done against field devices directly from the switch. Being an embedded Linux switch, the Sixnet model made a great target because it is incredibly easy to develop and deploy additional software into the firmware image.
Pop Quiz
A pop quiz when it comes to port mirroring is this: our managed Ethernet switch has 9 ports. How many ports does it have?
Be honest when you answer before highlighting below for the answer.
11 ports
If you said 9 ports, we might forgive you for being wrong. In truth, a managed switch has two hidden interfaces, so in this example we have 10 Ethernet ports and 1 serial port.
Wait, you say, where are these hidden ports? Here is a diagram of the switching chipset inside of our Sixnet switch (which is sold as a 9-port Ethernet switch and that uses a Marvell 6083 switching chipset).
image source: Marvell 88E6083 Product Overview
Notice the 10 Ethernet interfaces (ports 0-9) on our ‘9 port’ Ethernet switch?
Notice that you have an extra field device on your Ethernet network, and you didn’t even know it! This interface is hidden because there is no RJ45 jack associated with it, and there is no option to configure it via the Switch management web interface.
The ‘extra’ interface of course is connected to the microcontroller inside the switch. This is the microcontroller which runs BusyBox/Linux and allows us to actually configure the switch via HTTP, SSH, Telnet, and other protocols. This extra field device can do a few nasty things that we will get to in a bit.
One of the first tasks with our newly rooted switch was to learn to interface with the switching chipset itself. The switch already had a shared library for this, called ‘libswm’ to perform such exploration. This library provides an API for interfacing with the switching fabric’s MDIO and MDC interfaces (which is the serial interface for configuring the switching engine that we hinted at above).
A main application on the switch is called each time a switch configuration change is made, and at boot up. This application reads in the switch configuration file from the Linux system, and issues a series of commands to the switch chip via the MDIO/MDC interface to actually configure it. These commands include setting switch fabric control registers to determine what ports to enable and disable, what MAC address restrictions to apply to ports, how to configure port mirroring and port selection, etc.
The Third Man
Since we have a rootkit on the switch, we actually can do some funny things using a combination of this library and traditional Ethernet attacks. These include:
– Running a cron job which gives us a repeating window of time wherein port mirroring is disabled.
– Disabling ports entirely at times of our choosing.
– *** Mirroring traffic to the switch MCU itself. ***
– ***ARP poisoning switch comms between field devices for filtering on the MCU.***
Mirroring traffic to the switch MCU is interesting because we can decide to perform switch configuration changes based on what we see. For example if we monitor a group of PLCs, we can determine when an operator is doing something on an HMI (when we see out-of-sequence Output assertions) and we can decide to attack a system then. This will make it harder for operators to understand why a process control change happened — maybe they clicked something wrong, or some equipment failure happened during a manual operation.
One-way ARP poisoning is also interesting, because we gain the ability to manipulate traffic in a way that cannot be detected by IDS/NSM. We simply tell the field device that our switch management microcontroller is the IP device it wishes to send packets to, then we modify those packets and forward them right back to the switching fabric — with the correct MAC for the upstream server. The only observable artifact of this will be a slight timing delay of the packet.
What else is interesting is that the data sheet for the Marvell switch in this case contains configuration settings which are ‘Reserved’ and require an NDA from the fabric maker in order to learn the meaning of. An open area of research is whether we can dupe the switching fabric into routing all traffic through the switch main MCU using any of these reserved configuration fields. This would allow us to filter traffic for example to a mirror port, so that we could for example craft a special MAC address or IP options field which means, “don’t mirror this packet, please.”
It is pretty likely that we can achieve this filtering goal without any hidden features via reconfiguring the switch fabric at the right times. Timing port disablement and re-enablement correctly should allow an attacker the ability to perform MITM attacks that cannot be detected via normal means (only via timing analysis).
Mitigation Today
There isn’t much that you can do today to prevent an already-infected switch from giving you a bad time. These tips will make it (slightly) harder for an attacker to actually infect your switch, and will it (slightly) more likely that you will detect an infected switch.
All of these fun attacks were premised on the fact that only the upstream network connection will be mirrored. While this is definitely true on high-bandwidth networks, ICS networks tend not to be saturating their network links (nor getting anywhere close). A best practice today might be to mirror every one of your field device interfaces to your IDS or NMS collection port, instead of just mirroring the upstream interface. This makes it more likely to catch a switch-intitiated ARP poison attack, but of course the switch micro, if infected, could just disable the mirroring while it does its attacking. So this mitigation only works if the attacker isn’t particularly smart.
The other thing to do is ensure that your network equipment is not managed in-band with the rest of your network operations. Some switches allow you to specify a particular Ethernet port as ‘the’ management interface. If you can, do this with your equipment. Don’t connect this newly-configured management interface to your larger network — if you can, set aside a management LAN and restrict access to this LAN. This mitigation makes it a little less likely (maybe) that your switch will be rootkitted.
Finally, talk to your vendor. If they aren’t signing their firmware (both during updates and via a secure bootloader) and performing negative testing against their equipment, start asking that they do so. The time of best leverage for this is when holding purse strings to a large order.
Conclusion
Switch security is an interesting area of research, one that I hope will see more attention in the coming years. For sure, having digitally signed and attested firmware in managed Ethernet switches would be nice. Having a way to ‘tap’ communications between the switch MCU and the switching fabric would also be a wonderful feature to have in future network switches.
We certainly hope that switches will not be targeted, if only because these attacks are very difficult to mitigate now that so many insecure switches are deployed. Without going the route of dumping the switch firmware, likely making the switch unusable for production, there is no way to determine if a firmware has been changed today.
mirror image by Natalie’s New York