Stuxnet marks the Emergence of Real-World SCADA Security Challenges

What kind of Smart Grid security blog would this blog be if it didn't comment on the Stuxnet worm? The short story includes a couple of key players:

• Buried (previously undisclosed, aka "Zero Day") vulnerabilities in Windows. And Windows' security weaknesses used as a starting point for a SCADA attack
• Using USB drives to cross the air gaps and transport the worm from the networked world to the SCADA world
• Attackers acquiring (via $$$ or theft) trusted digital certificates and building them into the attack
• Hard-coded passwords in a Siemens-built SCADA system

If you want a thorough account of how Stuxnet works, Symantec did a bang up job here. But be forewarned, unless you've got some solid development chops, it may be more detail than you can handle!

A treatment better for business folks and arm chair grid security generalists comes McAfee here, or from ComputerWorld, with an initial article here, then this follow-up one week later, here, with input from SCADA security guru Joe Weiss. For the moment, the storm seems to have passed, with Siemens and security product co's offering solutions to clean up Stuxnet code from infected machines, and block it from others. But this story is far from over.

Weiss calls out 170 cyber related outages in the US to date, with 3 of them serious enough to have caused significant (read: expensive) regional outages. He also notes that it's currently impossible to discern cyber attacks from accidental glitches because of the weak state of digital forensics in the power industry to date.

By the way, the 2-way power and data flow Smart Grid, great enabler hacking and attacking, will also improve our ability to do post mortems on cyber incidents, though as with many other types of cyber crime across the Web, it will often be super difficult to pin down the originator.

For me, the big take away comes from the praise security analysts are bestowing on the Stuxnet architects. I don't mean to suggest they support this type of work, not at all. But rather, that this was no casual side-project of some mis-directed youth. Stuxnet is heavy, heavy duty malware. Which means, to me anyway, that there's much more to come, and that the USG and FERC in particular, need to get way more serious about energy control system security, and issue mandatory policy that gets it done throughout the bulk power system and across the distribution network.

We may get some more insight from the cyber security conferences Black Hat and Defcon starting this week in Vegas, where Jonathan Pollet of Red Tiger Security, will discuss (and potentially reveal) SCADA vulnerabilities in utility control systems. Stay tuned ... this is exactly what Joe has been warning us about all along.

Demand Response: Refreshingly Effective

This one's short and sweet: demand response (DR) works. It may not be the answer to all our energy woes, but it afford-ably accomplishes what it seeks to accomplish, simultaneously reducing its customers' costs and emissions. In case you missed the recent heat wave recounted in the article (you were probably having one of your own), here's a good part:

When two power plants failed in ISO’s New England service region in late June, energy prices momentarily spiked to $1,000 per megawatt-hours. EnerNOC’s DemandSMART network sprang into action and ISO was able to meet demand and avoid a hit to its bottom line by reducing demand by 380 megawatts. According to EnerNOC, this was accomplished largely because of its network operations center, which remotely enlisted over 1,000 assets during a two and half hour dispatch and managed 500,000 data transactions to manage the situation in real-time.

More on this here, and even more now awaits folks with Earth2tech's premium subscription. As with other new forms of data-driven grid functionality, DR applications open new doors through which bad actors might enter. So far, though, it seems the DR folks at EnerNOC and elsewhere are running a tight (enough) ship.

Photo credit: liz west at Flickr.com

Smart Grid Confidence Game 3: Hacking the US Grid Annoyingly Difficult, Ridiculously Time Consuming, says Wired's Danger Room

Much appreciation to Wired's Michael Tanji for recently observing that the sky remains in its nominal coordinates and that we should proceed with our lives, and previously scheduled Smart Grid initiatives, already in progress. Here's the opening:

People have claimed in the past to be able to turn off the internet, there are reports of foreign penetrations into government systems, “proof” of foreign interest in attacking U.S. critical infrastructure based on studies, and concerns about adversary capabilities based on allegations of successful critical infrastructure attacks. Which begs the question: If it’s so easy to turn off the lights using your laptop, how come it doesn’t happen more often?

Remember, it serves no one's interests to deny that the grid and Smart Grid face many significant threats. It's just that by subjecting ourselves to jarring FUD alarms only, we lose balance, perspective and the ability to believe what our eyes are telling us is really going on in the world.

So why is it, then, that despite daily media claxons and vuvuzelas signaling that the end (of the grid) is nigh, that our massive and complex electrical generating, transmitting, distributing and consuming systems mainly keep working? The answer lies, at least in part, in their very complexity. Tanji continues:

The fact of the matter is that it isn’t easy to do any of these things. Your average power grid or drinking-water system isn’t analogous to a PC or even to a corporate network. The complexity of such systems, and the use of proprietary operating systems and applications that are not readily available for study by your average hacker, make the development of exploits for any uncovered vulnerabilities much more difficult than using Metasploit.

Now here comes the tricky part, where isolation from the Internet is given some of the credit:

... these systems are rarely connected directly to the public internet. And that makes gaining access to grid-controlling networks a challenge for all but the most dedicated, motivated and skilled — nation-states, in other words.

While isolation may be the current state, I don't think you can bet on it as a constant. The temptations to connect are too many, and one-off connections to the Internet often go undetected by internal security staff and auditors. Better to stick with the complexity/diversity message than the "disconnected today/always will be disconnected" pledge.

The full piece is here, followed, as per usual, by a comment chorus from the bitter and bizarre (with a couple of regular folks sprinkled in).

Also, if you want to get a better feel for this complexity yourself, give the Google Tech Talk on "Smart Grid, Utilities, and Internet Protocols" a look. The presenter is Enernex's Erich Gunther. As the saying goes, he'll forget more about our electric infrastructure and the Smart Grid than most of us will ever learn. In addition to the complexity arguments made earlier, Gunther, and others like him on the "good guys" team, are another reason why I'm confident that attackers' impacts will be moderate and the sky will remain aloft as we develop and deploy the Smart Grid. Hope you're confident too.

Photo credit: Dominic Alves at Flickr.com

Changing of the Guard: Weatherford Replaces Assante as NERC CISO

Just so you know, there was a shift in the force recently as Michael Assante stepped down from the CISO position and NERC sought an able replacement. This post (and this NERC announcement) informs you that, happily, the new CISO has been installed and we're back on track.

Good thing too, cause the electricity generating, transmitting (if not yet, distributing) industry is being pulled in two seemingly opposing directions: on one hand, the desire the demonstrate compliance with CIPS 002-009; while on the other, high anxiety that:

• CIPS 010 and 011 are much different than 002-009 (see summary from James Holler here) and unless they're phased in VERY gradually, that means trouble
• The new CIPS are based largely on security control standards like those in NIST SP 800-53 "Recommended Security Controls for Federal Information Systems and Organizations." Again, a whole different enchilada in terms of detail than what's in 002-009
• This will force huge changes (and likely, commensurate new expenses) for utilities trying make the best of limited human resources, time and funds

Maybe there's a loose connection of sorts here. I recall that the SP 800-53 controls are referenced in DOD 8500.x security policies (see DITSCAP and DIACAP). Michael Assante was a Naval intel officer and seems to me he did a great job during his tenure at NERC. Now Mark Weatherford, recently the CISO for the states of California and Colorado, also comes to the office with a solid Navy pedigree. From the NERC announcement on him:

Weatherford began his career as a Naval Cryptologic Officer, where he led the Navy’s Computer Network Defense operations and the Naval Computer Incident Response Team. Weatherford has a bachelor’s degree from the University of Arizona and a master’s degree from the Naval Postgraduate School.

One thing we've seen in our talks with CISOs and other security professionals in the utilities and ISO/RTOs is the prevalence of prior military (though not always Naval) experience, including folks who did crypto and other cyber security related jobs when they were slightly less "seasoned."

Well, as you'll see from Holler's summary, if not your own hands-on experience in the compliance trenches, it may well be a rough ride moving from the relatively light-weight original CIPS, which really just went fully live on 1 Jan of this year, to the industrial strength 010 and 011. I for one am pulling for Mark to do a great job and wish him every success. We all have a job to do, but his is a key role in this.

Webcast: Smart Grid IT Systems Security

Just a reminder - this is a very high level intro to this topic, most appropriate for business folks and new initiates. If you're looking for more meat, much more detailed guidance is referenced in the presentation.

Also, looks like we've found a format that'll work for the webcasts. For best results, recommend you click on the "full screen" icon located in the extreme lower right-hand corner. OK then? Here's the latest from the series ... see what you think:

SGSB Webcast 3: Smart Grid IT Systems Security

View more webinars from Andy Bochman.

Do Androids Dream of Electric Cars?

Q: I'll repeat the question - Do androids dream of electric cars?

A: I don't know, but I do.

With apologies to the Philip K Dick novel that inspired Blade Runner's spinners (air cars), while they may not fly, the propulsion systems of mainstream automobiles are about to undergo a major transformation. Over the next few years our everyday vehicles are poised to make the leap from Popular Science to Car and Driver. But the implications for critical electrical infrastructure go far beyond quiet motors and cleaner air; the grid itself will transform to accommodate the new loads, and grid IT systems will be upgraded to take advantage of some exciting new grid management capabilities Plug-in Electric Vehicles (PEVs) will enable.

To whit, see the North American ISOs and RTO's "Assessment of Plug-in Electric Vehicle Integration with ISO/RTO Systems." Below are a couple of security related snippets from this report, but the whole thing makes exciting reading for anyone interested in building and/or using the energy future.

Recommended standard communication interfaces:

• DNP3
• ICCP or IEC 60870-6/TASE.2
• XML/HTTPS

Recommended encryption standards:

• Secure ICCP
• Secure DNP3; compliant with IEC 62351-5 for Secure Authentication
• HTTPS with digital certificates

The authors make it plain that there's a ton of work to be done and that these are just a few baby steps when they note:

In addition to the identified communication interfaces and security requirements (including standards in development for smart grid and the NERC CIP 002-009 Standards), there are other integration requirements either not covered or partially covered by existing standards or developing standards.

Jack and I will be beating a drum ... and watching ... to make sure software security requirements get prioritized. But from complex systems and business process engineering perspectives, not to mention the attention paid to interstate coordination and market signals:

Because PEVs are mobile loads, and because aggregators will serve as liaisons between PEVs and ISO/RTOs, consistency across ISO/RTOs is a concern. As such, standard processes, including validation and settlement processes, and common communication protocols, including security requirements and communication interfaces, are desirable. Therefore, the project team recommends continued participation by the IRC in ongoing standards development, such as with SAE, NIST, NAESB, IEC and IEEE. The project team also recommends ISO/RTO investments in IT and communications infrastructure to meet the unique needs of PEV resources and aggregators and ultimately to enhance system reliability and enable participation of PEV resources in ISO/RTO markets.

It may not be a moon shot, but the scale of this project, especially when as we go from hundreds, to hundreds of thousands, to many millions of electric cars, sometimes seems similar. Suffice it to say, keeping it all secure, while getting all the other parts right, will be a grand challenge.

Photo Credit: Tesla Model S at Industry.Bnet.com

Opinion: Industry, not Utilities, Needs to Make and Better Articulate Business Case for Smart Grid Changes

Have you noticed there are rich veins of knowledge and experience in certain parts of the Web that aren't visible to Google? Well, if like me you dedicate some small part of your life to reviewing the reader comments that follow online articles on grid security and privacy, you'll feel like we ought to just throw in the towel on this whole Smart Grid thing right now. Actually, I've got my fingers crossed that those doom-spouting commenters are not representative of the general population.

However, you'll find a few places of real value for would-be Smart Grid implementors and advocates, similarly obscure to most search engines. Here are a few Linked-in groups, notably: SmartGrids - Energy & Water, and Smart Grid Security (you'll have to sign up for LinkedIn to participate but that's not hard). There's some marketing going on, but also substantive discussions and debates on the present and future of the grid, led by folks who know the space first hand.

One of these nuggets was posted in the Energy & Water group by Paul Duncan of GSD Energy Consultants, a former Navy Chief Petty Officer whose career also includes several years building demand response solutions for GridPoint customers. On a thread posing the question of whether utilities personnel need to do a better job articulating ROI for Smart Grid projects, here's his response, from a sympathetic and pointedly self-critical perspective:

Therein lies the problem.

With rare exception, it is nearly impossible for personnel at a utility to keep up with the repeated shotgun blasts of information and technologies that are coming their way day after day. Not only are utility personnel generally ill-equipped to understand all of this new hardware and software, we, as an industry, have been evolving at a high rate of development speed, resulting in rapid transformation of capabilities and further confusion on the utility-side of the table. Industry folks love to talk about advanced power electronics, advanced software systems to aggregate distributed energy resources, real-time decision processes, and more.

Yet I see utilities struggling with the value propositions and implementations of AMI networks, let alone anything downstream (and technologically more advanced) of those platforms. This is because (although many of us hesitate to admit it) AMI does represent a large change within a utility -- a change in billing processes, a change in work flow, and a change in data management.

When the utilities' "do-nothing/zero change" model of risk avoidance yields a non-zero, positive fixed rate of return on deployed assets, we end up competing against the legacy knowledge level with technology and business processes that have higher perceived risk than the "do-nothing" alternative. In my opinion, we in the industry have done an extremely poor job at helping the sponsor within the utility get their hands around risk mitigation issues, resulting in limited pilots, limited results, and a failure to show convincing scalability.

I am a firm believer that until we, not utilities, can further quantify the ROI of our products and services, and do so at a risk-differential level that is not too far from the normal business risk-level of the utility, that we will be stuck with limited pilots as well as slow adoption by our utility clients. We must do a better job in our product architecture to quantify the ROI of our products and services, so that the utility manager can reduce their learning curve, compare and contrast alternatives, and in the end, have a greater understanding of the technologies available from industry. Until then, I feel that "Smart Grid" will remain largely external to the utility, resulting in slow adoption.

I immediately responded to this piece because it speaks to what I have seen in the field as well. Basically, that there is no place in a utility for technology for technology's sake. And that risk tolerance compared to other sectors is super low ... and thank goodness for that. It's industry's job to formulate and clearly articulate low risk solutions that improve the lives of utilities personnel and their clients, and to arm their champions with the compelling evidence they'll need to get their projects prioritized.