SolarWinds Trouble: It Rolls Downhill

By Dr. Wesley McGrew, Senior Cyber Fellow; w.mcgrew@martinfed.com

If it wasn’t already part of your intuition, then your career has probably taught you that when there’s trouble, the saying goes—put mildly here—that “it rolls downhill.” The problems you deal with daily are often not of your creation, but they become your responsibility. Supply-chain vulnerabilities and breaches in cybersecurity work like this too. Cybersecurity & Infrastructure Security Agency (CISA) has issued Emergency Directive 21-01, “Mitigate SolarWinds Orion Code Compromise” (https://cyber.dhs.gov/ed/21-01/), which, if you work in cybersecurity at a federal agency, rolls some heavy information your way for a Monday morning.

The Problem

This is what we call a supply-chain attack, where a product or service you rely on, such as a popular security monitoring product, falls victim to a breach, and that access is leveraged by the attacker to compromise you. SolarWinds Orion is a widely used platform for managing network assets and monitoring their logs, traffic, and configuration. From March to May of 2020, updates for SolarWinds Orion had backdoors inserted into the code by an advanced persistent threat (Russian foreign intelligence—SVR—according to sources that spoke to the Washington Post).

Even if you implement best practices of hardening, testing, and monitoring on your network, supply-chain attacks should always be a concern. You have very little control over the code that goes into products you rely on, and you likely do not have the resources (or even the permission in your license) to comprehensively reverse-engineer and test every third-party or open-source product you deploy to your own network. You will have imperfect knowledge and trust of code in your organization, and all you can do is follow defense-in-depth: layer security around potential problems to reduce risk. That and be ready to respond when something like this SolarWinds compromise happens.

Immediate Action

The direction given by CISA is required to be acted on by federal agencies, exempting systems defined as “national security systems”, and those operated by the Department of Defense and the Intelligence Community (presumably, these actions and more are being taken there). Even if you’re not in the federal space, it’s good guidance. Here are the main points, summarized and discussed:

• If you have the expertise to do so, make a forensically-sound image of memory and disk of hosts running SolarWinds Orion. If you have logs of network traffic from compromised systems, save and analyze them for the provided indicators of compromise.
• Shut off systems running the impacted versions. CISA takes the additional step of directing agencies to not rejoin those hosts to the enterprise domain until further direction is given. This gives agencies a little time to breathe and assess how to rebuild these systems.
• Block all traffic to and from hosts where the SolarWinds Orion product has been installed, including versions not specified as being compromised.
• Look for the provided indicators-of-compromise (IOCs) and remove the accounts and persistence created by the attackers.
• Report identified IOCs to CISA. This provides higher-level awareness among agencies of the breadth and depth of the compromise.
• Once the previous recommendations are implemented to isolate the compromise, steps can be taken to prepare to bring systems back online (with the final go-ahead being given by CISA at some point). Notably, CISA recommends that all hosts monitored by SolarWinds Orion, not just those with SolarWinds installed, should be treated as potentially compromised. SolarWinds is given a lot of trust on networks it monitors, and as trust is spread around, so is suspicion in the event of a compromise.

Readiness

You can’t eliminate supply-chain threats, but you can prepare to detect and respond to incidents involving them. Based on the recommended response actions, think about what you need to have in-place:

• If you don’t have the in-house capability to capture forensic evidence from memory, disk, and network, you need to establish a relationship with a service provider that can help you do so quickly. This is not something you have the time or capacity to figure out “on the spot” during incident response. If you don’t have this relationship established, you’ll be forced to move on with response and recovery in a way that loses evidence and visibility of the compromise.
• Keep a good inventory of third-party systems and software on your network. This may seem obvious, but when a breach of a vendor occurs, you need to know at a glance whether or not you rely on that vendor. Good asset management and audits of your physical space, network ranges, and software installations will make sure you don’t get caught unaware. Identify and subscribe to notification systems for each of your vendors to maintain awareness. SolarWinds is big enough that you read about it this morning in the Washington Post, but the next one might be more niche, leaving you in just as much of a mess.
• Plan and practice for temporarily operating your network in configurations that do not include software from some of your key vendors. What is the minimal configuration in which you can continue operations?
• Egress (outbound) filtering of traffic can go a long way to helping you isolate the impact of a compromised product and potentially identify malicious activity. The more trust you place in a system on your network, the more you should control, monitor, and restrict the activities of that system.
• Testing engagements, such as penetration testing and red team exercises, should involve starting points and scope that realistically emulate advanced threats already in-place on systems internal to the network.

Conclusions

Supply-chain attacks can and will happen, and while the fault may not be with you or your organization, the responsibility for responding to the attack (or potential for one) will be yours. Without preparation, there will be aspects of your response, recovery, and investigation that will suffer from a lack of resources. For many organizations, it’s expensive and inefficient to keep staff and resources on hand full-time to execute all of this (or test for the potential issues in the first place). Build relationships and agreements with vendors ahead of time to test, prepare, monitor, and respond to supply-chain attacks. Maintain awareness and capability so that “surprise” is short-lived, and work can begin to secure your networks.