Portaledge is Digital Bond’s control system security research project funded by the US Department of Energy. We recently issued the first release and are nearing the second, so this is a good time to discuss with practical examples, what Portaledge is, how it works, and how asset owners can implement it to detect cyber attacks in this multi-part blog post.
In the simplest terms, Portaledge aggregates security events from a variety of control system and IT sources and then correlates these security events to detect attacks. Portaledge is not a stand alone software package. Instead, Digital Bond leveraged the aggregation and correlation capabilities of OSIsoft’s PI Historian, which enjoys near ubiquity in the energy sector. The robust and mature PI development environment limited the amount of code we needed to write, and the market share provides a large number of ready users.
Security events can be actual directed “hacking attempts”, non-directed malware or scanning, non-malicious mistakes such as fat fingered password, the unintentional DOSing of the network by a process gone wild, or a gamut of other happenings. Security Events can be created by control and IT applications and devices of all types. The ability to detect, correlate and aggregate security events is a paramount for both improving security and providing forensic audibility, but it is an ability lacking in most control systems. Fortunately, Portaledge provides this capability while creating a recorded log of events.
As Portaledge consists of PI ACE modules their is no additional software to buy if an owner/operator already has deployed PI with the appropriate licenses. All that is needed is Digital Bond’s subscriber content ($100 annual subscription fee), download the modules, and install them into ACE. Modules currently exist for System Availability events and Enumeration (Scanning) events will soon follow. A wide range of additional event classes are planned.
Continuing the blog series about Portaledge (Digital Bond’s US Department of Energy funded cyber attack detection project), this weeks entry discusses how Portaledge aggregates IT security events using the PI IT Monitor Interfaces.
Portaledge uses the OSISoft’s PI IT Monitor Interfaces to collect data from a variety of classic IT data sources. Examples include:
- Session Data from the IP Flow interface
Session Data can be used to identify new systems joining the network, communication on new ports and services, increases or decreases in the data volume between two systems and a number of other signs of an attacker on the network.
- Availability Data from the Ping, TCP Response, Windows Performance Monitor, and SNMP interfaces
Availability Data can identify when a workstation, server or other system is unavailable or suffering from degraded performance. This may or may not be the result of an attack.
- Escalation Data from the Syslog and Windows Event log interfaces
Attackers take some common actions after gaining a foothold in a system, for example, adding a new user with administrative privileges.
Data for the Portaledge calculations is polled from PI Tags. Creating PI Tags is a common task, familiar to PI administrators. Tags are created by the administrator utlizing the PI System Management Tool, or they can be imported via an Excel plugin using templates.
The Excel plugin simplifies the task of creating tags related to the Portaledge modules. For each Porteladge module we provide a template file and instructions on how to modify the template file specific to your installation. As this manipulation is performed in Excel it is possible to quickly modify the tags using standard cut and paste functions. The data can also be manipulated as individual cells, rows, or columns, simplifying the tag creation process.
For each IT Monitor Interface there is a PI document describing the installation of the interface and the creation of points that are associated with the particular interface. These tags configure the behavior of the interface and act as a storage location for the interface’s data.
For example, with the IP Flow Interface there are several associated tags that store the flow data: Source Name, Source IP, Source Port, Destination Name, Destination IP, Destination Port, Protocol, and Octets (session length in bytes). As the IP Flow Interface observes the flow data it sends data points to the proper tags and stores the data point with a time stamp.
Once the data is in the PI server we can begin to analyze it and create Events, Event Class Events and Meta Events. This analysis will be covered in future blog posts.
Our goal in our tool development research projects is to make the installation process as painless as possible. This in a challenge in Portaledge as we need to be able to poll and write security log data, but the location and naming of the data is dependent upon the user and will be different from installation to installation.
Solution? Enter the PI SMT Module Database and Aliases, with Aliases we can “normalize” the name of a PI Tag. To understand the process of creating and using an Alias to a PI Tag using the Module Database, we need to first understand what a PI Point is, what a PI Tag is, and then what an Alias to a PI Tag is.
A PI Point is the underlying data structure with its associated types and settings, that stores datum into the PI historian. A PI Tag is then a label for the PI Point. Finally, an Alias is a shortened name reference to the PI Tag. Most people use the terms PI Point and PI Tag interchangeably.
For Portaledge Event Module installation we include a Tag Creator Excel template that can be used in conjunction with the PI Excel Plugin to rapidly create the PI Tags associated with the Portaledge Event, or the PI Tags can be created by hand in the PI SMT Tool using the point builder. The PI Tag names for any given Event have to be customized for the individual installation, as seen in the example below where the systems’ IP addresses compose a portion of the Tag’s information.
How do we as coders then poll and write data to tags that have names that we can not determine ahead of time? Simple, we create an Alias to the tag. We use the Module Database tool of the PI SMT to create the Alias. It takes a name and a PI Tag that it will reference. We can then poll data from the Tag by referencing the Alias.
When a user installs a Portaledge module, the user then needs to assure that the Alias refers to the PI Tag name as it exists on their particular installation. Instructions and templates to do this are provided with each module.
Through the use of the Aliases we can poll the data from the associated tags using names that are “normalized” across custom installations. And the user does not need to customize the Event detection and correlation code!
The PI Advanced Computing Engine (ACE) is the environment in which Portaledge modules are developed, execute and are managed. ACE allows for the creation of modules to perform complex calculations, systems and process monitoring, and communications between systems and data sources. In Portaledge with use ACE to correlate security events from a variety of data sources to detect cyber attacks.
These custom modules are built using an OSIsoft provided SDK in the Microsoft Visual Studio environment and are typically developed in Visual Basic. The 3 primary components that comprise ACE are the Wizard and SDK, the Manager and the Scheduler.
The Wizard and SDK provide a framework for developing ACE modules. The Wizard allows for the selection of input and output tags either directly to the tag or through an alias. It also provides a template file for custom development with the entry and exit points into the module and function stubs for the initialization of variables and garbage collection of data. The key functionality of a module is also stubbed out by the Wizard in the form of the “Calculation” method, which is called each time the module runs and is where calculations and calls to other functions pertinent to the custom calculations are performed. The SDK contains the API and library functions upon which modules are built and that the Wizard relies.
The ACE Manager allows for the control of modules. Via the Manager the period of a module’s execution can be controlled, modules can be started and suspended, and module execution can be monitored. The Manager also monitors the status of a module and will note if it has errored out and can supply some information in reference to the error.
The ACE Scheduler is the system service that actually executes the modules. It executes the modules based on the parameters specified in the Manager such as frequency, time offset, PI Context, etc. There are two versions of the Scheduler and SDK depending on what version of ACE is employed.
Custom ACE modules are useful as they allow a user to develop solutions to problem that may be unique to their deployment. Such solutions could be the conversion of units, monitoring and storing new records into the historian based on custom events, or updating the Module Database with data from an external source. Portaledge is such a custom solution that uses the power of ACE to monitor and correlate custom security events.
As our second release of Portaledge Event Modules is forthcoming, I am continuing with a series of posts on Portaledge fundamentals. My goal is to provide an overview of how Portaledge functions, and it role as a Security Event Manager for control systems.
Portaledge relies on a variety of data sources to monitor a system and detect security events. For each security event detected and recorded by Portaledge there is a corresponding event trigger. Each event module in a Portaldge event class monitors the historical data for one or more sets of criteria, watching for potential event triggers and correspondingly, creating and logging the associated event.
A Porteledge event trigger is a single data point or set of aggregated data points sent from the various PI interfaces, such as the PI IP Flow, Ping, and Syslog IT Monitor Interfaces, the OPC and other SCADA Protocol Interfaces, SCADA vendor specific interfaces or other security event data sources. Portaledge polls data from the historical data base and evaluates the data points or aggregated data set against specific criteria. If the criteria meets a trigger definition then an event is created and, the data point/set that caused the event is the event trigger.
Portaledge aggregates data from various Pi points to monitor for triggers. For example; the majority of the enumeration events poll data from over 8 data sources to create a data set representative of a network communication session. The data polled and aggregated is: source system name, source system IP address, source system port, destination system name, destination system IP address, destination system port, session length, and session protocol e.g. TCP, UDP, ICMP etc.
As Portaledge creates and stores events, an “event” is then the base level of correlated data. An event is targeted at a single system or device, i.e. a single IP address. Correlated and chained events within an event class or across event classes will be discussed in future postings.
To better understand the trigger to event process consider the following example: The Enumeration Traffic Monitor module monitors for communications between systems that are “out of bounds” meaning they are not on a list of “approved” communications. For each system in a network the administrator can create a system entry, and with that entry keep a list of systems that are allowed to talk to the system, their associated source and destination ports, and what protocol.
The Enumeration Traffic Monitor, when it executes, creates and parses a list of all the session that occurred over a given time slice. It creates the list by polling the 8 data points mentioned above from the historical data base. It then sorts and parses the list, comparing the sessions against the list of allowed communications. When a sessions is detected that is not allowed an event alert is created and logged.
The trigger is the set of of data creating the out of bounds communication fed to the historian by the IP Flow Interface. The event is a “Enumeration Traffic Monitor Event” noted with an alert that shows the source and destination IPs, the ports, the sessions size and protocol and the time stamp of when the session info was written to the historian.
Portaledge has an event hierarchy. The hierarchy (from smallest to largest) consists of: Event Triggers, which cause Events, which are correlated in a class into Event Class Events. Events and Event Class Events can be correlated across classes into Meta Events.
Today I am going to discuss Event Class Events. Triggers and Events were covered last week in Part 5 of this series. Meta Events will be described in an upcoming post.
To date for Portaledge we have released one set of Event Class modules, the Availability Class. And there is another set, the Enumeration Class forthcoming. The other classes to be developed are: Communication, Escalation, Exploitation, Obfuscation, Process Manipulation, and possibly Reconnaissance. More information on the Portaledge classes and hierarchy can be seen on the SCADApedia.
Each Event occurs in one and only one Event Class. For example in the Enumeration Event Class there are the; UDP Port Scan, SYN Port Scan, ICMP Scan, TCP Port Scan, and FIN Port Scan events as well as the associated Port Sweep events. The Enumeration Class also contains a Finger Detection Event and an Event that monitors for out of range communications, namely the Traffic Monitor Event. These Enumeration Events, scheduled for release shortly, only represent a subsection of the total events defined in the Enumeration Event Class.
As Events occur they are logged and written to the historical database. When the Event Class Event module periodically fires it examines the Events associated with it’s Event Class across a time slice and correlates the Events on commonalities. The Events sharing commonalities are “chained” into Event Chains within the Event Class.
For clarification of this process, consider the following examples:
An attacker runs a Nmap UDP scan on a subset of systems on a network segment. As each system in the subset is scanned an Enumeration Class Event reporting that the system has been scanned, the source IP and port of the scan, and the number of ports touched on the system being scanned is created, and written to the historical database. When the Enumeration Event Class module fires it gathers all of these data points across a time slice and correlates them on the common source IP and protocol of the scan. The common events are sorted on time and chained into an “Enumeration Event Class Chain.” The chain is composed of the same number of parts as the number of events detected and each part contains data relative to the event that caused it.
In the second example consider the case where an attacker performs a TCP port scan of a single system. This port scan creates an TCP Port Scan Enumeration Event indicating that a number of ports were scanned from the attacker’s system and port. The attacker, noticing that the finger port is enable next tries to enumerate user accounts using finger. This creates a number of Finger Enumeration Events. The Event Class Event module correlates these events based on the common source IP of the attackers system and the common destination IP of the system scanned and fingered. An Enumeration Event Class Chain is created containing as its first part the TCP port scan and then the subsequent finger events.
As discussed in Part 6 of our on going series on the inner workings of Portaledge, Portaledge has an event hierarchy. The hierarchy (from smallest to largest) consists of: Event Triggers, which cause Events, which are correlated in a class into Event Class Events. Events and Event Class Events can be correlated across Event Classes into Meta Events.
Meta Events are the most telling of Portaledge’s alerts. They indicate more complete attacks, by this I mean if you see a Meta Event chain composed of an Enumeration Event followed by a Penetration Event and then an Escalation Event, you can be fairly certain that an attacker is playing on your control system. While individual Events and Event Class Events will reveal attacks, they also are more susceptible to false positives and may not provide enough information on what an attacker is doing. The Meta Events through their correlation help to gain better insight into the bigger picture of what is occurring.
A meta event occurs when two or more Event Class Events can be correlated on a commonality. There are no specific names for meta events. Instead the meta event is named by the Event Class Event chain that triggered the Meta Event.
Some generic examples would be:
- Meta Event Chain: Reconnaissance>Enumeration>Exploitation
- Meta Event Chain: Enumeration>Availability
- Meta Event Chain: Escalation>Communication>Obfuscation
The Meta Event module correlates events on commonalities. Such commonalities may include: source IP, destination IP, source port, destination port, time, or event type, such as Availability Event alerts of the same type at about the same time. The Portaledge ACE module that produces the Meta Events will be more “intelligent” with each release. The initial release will provide simple correlations as noted above.
For a more specific example consider the case where an attacker port scans a PLC that resides on a control system at 192.168.10.25. The IP Flow monitor feeds the scan data to the Enumeration Event Class TCP Port scan event which sees the port scan and throws an alert. The comm stack of the PLC is overloaded by the port scan introducing latency into the PLC’s comm stack. The Field Device Network Latency alert is then triggered by the introduced latency on the field device. The Meta Event module sees the two event and correlates the destination IP of the scan with the IP address of the device that has incurred a Field Device Network Latency event, sorting the events by time stamp and chaining them into an Enumeration>Availability Meta Event. TCP Port scan and Field Device Network Latency are the specific components of the Meta Event chain.