A Novel Approach to Grid Cybersecurity Awareness

Not long ago I was in a meeting with the CIO of a large electric utility and when I inquired as to the cybersecurity awareness of the board of directors, was told it had recently skyrocketed.

Why the sudden shift I asked?  Had the company just endured a serious and/or highly public breach? Nope, things had been mercifully static on that front. A classified threat briefing by DHS? No, not that either. Well, what was it then?

Apparently one board member had read the latest Tom Clancy book, Threat Vector and once exposed to Clancy's fictional vision of how the US could be brought low through largely cyber means, it changed his thinking. Spoke in language he could understand, and captured his imagination too. It soon spread to the rest of the board.

Now comes former Senator Byron Dorgan with a cautionary novel of his own, and this one is much more grid-centric, from the title on. I later read Threat Vector myself ... 900 pages or so if I remember right, looking for power sector specific attacks and breaches and they were few. I've read some of the reviews of Gridlock, though, and in it the US grid is front and center and not doing so well.

Dorgan and co-author David Hagberg don't have anywhere near Clancy's readership, not close. But if an executive in your company were to happen upon a copy, well, apparently it's quite a page turner, and you might have a new, more cybersecurity-aware board to work with in a few weeks.

Looking Again at the Markey-Waxman Grid Vulnerability Publication

Where would I be without feedback? Many thanks to SGSB readers who chimed in on this.

I recently published a post titled "House of Reps Report Reams Utilities on Cybersecurity." Not accurate and all you have to do is read the cover page which, just below the House seal, says "A Report written by the staff of congressmen Edward J. Markey (D-MA) and Henry A. Waxman (D-CA)". Mea Gulpa.

So on second look I looked a little closer and found some things to like and some things I had to wonder about. For example, I'm happy to see congressmen seeking more information about the current state of security in our sector. Who could argue with that?

But their methods are not fully sound.

House of Reps Report Reams Utilities on Cybersecurity

Was trying to capture spirit of Jesse Berst's headline on the same subject:

Utilities to FERC: Take your security measures and shove it

That's not very nice, is it?  I think they toned it down with a later change, but this headline was what was in my inbox in this morning's SmartGridNews.com newsletter. The subject is a recent report published by the House of Representatives that's highly critical of electric utilities behavior to date re: grid cybersecurity.

Moving on! The Wall Street Journal's Rachel King did a fine write-up of recent testimony from the CEO of the American Gas Association (AGA), Dave McCurdy. King began by noting that:

The oil and gas sector faces many of the same cyber security challenges as the electric industry. Yet, there’s one major difference between the industries, both of which need to secure software-based industrial control systems from intruders. There are no regulations governing cyber security among the oil and gas companies.

Sanity Check: Nuclear Cyber Security Should be the Best, Right?

A few recent missile launchings notwithstanding, you may recall a little over a month ago things were hot and heavy in the North vs. South Korea showdown. On April 15th Japan Times published this account: South Korea Bolsters Security of Nuclear Plant Network, which opened thusly:

SEOUL – The state-run operator of South Korea’s nuclear power plants has separated its internal computer network from the Internet in an effort to guard against possible North Korean cyber attacks, Yonhap News Agency reported Sunday.

and continued:

It said Korea Hydro & Nuclear Power Co. has also completely divided its nuclear plant control systems from its internal computer networks and restricted both systems’ access to the Internet, while USB ports of the plant control systems have also been sealed.

Another Disclosure, this time with ICS CERT's Blessing

We're only a few months past Basecamp, and here we go again. Only this time there are fewer voices urging restraint.

Wired's Threat Level blog put up a story of a certain control system OEM that seemed uniquely unaware of the risks it had built into its products, and unwilling to make a change of any kind. At the time of publication, 25 April 2012, the company still hadn't budged.

Then, on 1 May 2012, the Christian Science Monitor was telling a different story: the vendor pledged to make and distribute a fix.

The Wired article ended with a couple of sentences that concisely capture this problem and make you want to laugh and cry at the same time:

Numerous researchers have been warning about the vulnerabilities for years.  But vendors have largely ignored the warnings and criticism because customers haven’t demanded that the vendors secure their products.

Have your heard the term "goat rope"?  How about "goat rodeo"?  This situation is definitely one of those ... and maybe both. Hope both the vendor and user sides figure out how to get their ducks in line, and fast.

Photo credit: Mike Baird at Flickr.com

Full Disclosure from 2012 Distributech's Keynote Security Panel

It's fun to connect with and catch-up with energy sector security friends, and not always at security conferences. I think we all get a kick out of seeing each other and then dispersing back out into the world to promote the cause and fight our battles in all the different ways we do it.

In fact, it feels a little more special when gather inside a larger conference context, which without a doubt is what you get at the mighty annual Distributech, which took place this year in sunny San Antonio, Texas.

So, enough chit chat. Let's dive into what was discussed on Thursday morning by these folks. Moderator Mike Ahmadi of GraniteKey expertly led a panel of experts on the topic of Security Standards, including:

• Bobby Brown, Enernex 
• Alan Rivaldo, Texas PUC 
• Nate Kube, Wurldtech 
• Darren Highfill, Man of Many Hats 

The guys covered several different topics in depth, including security metrics, vulnerability handling in IT vs. OT, social engineering, and perhaps, most provocatively, security information disclosure ethics and ramifications. Below find a few highlights for each one:

Metrics and Measurement

• In the shadow of Basecamp (which we'll get to shortly), trying to gauge industry progress on security or lack thereof, Mike asked: "are products getting better?" and the response surprised some of us I think. Nate, who has been testing grid products and systems since he was knee high said "absolutely!"
• Others chimed in that, slowly but surely, increased awareness has raised the bar for what's expected from vendors. Sometimes it's because utilities' RFPs' demand it, other times it comes from the vendors themselves. Altogether it's certainly too slowly for many of us, but the consensus seemed to be: tangible improvement is happening out there
• Darren introduced the new DOE RMMM (in early development), referenced other maturity models and frameworks, and he and the panel seemed to contend that all of these, to a greater or lesser extent, help organizations baseline and roadmap their security functions and goals ... and who wouldn't want that!
• Bobby Brown got some laughs (from me, anyway) when he likened the concept of security maturity standards for SG products to the carnival sign we all know that says "You must be this tall to ride this ride"
• Nate praised an audience member's phrase: "at the speed of Metasploit". This set the stage for the later discussion on disclosure. (There's more on the Metasploit vulnerability and exploit development framework HERE if this is your first time hearing the term.)
• Much to my delight, much was said about metrics and measurement in the early going, as we moved back and forth between contrasting the development and evolution of standards and guidelines (e.g., NERC CIPs, NISTIR 7628, IEC 62443 2-4, etc.) with demonstrable improvement in the security posture of utilities

Vulnerabilities in IT vs. OT

This may be obvious to many folks, and I've heard it mentioned quite a bit myself especially concerning meters. But the point was made that in the IT universe, one of the primary modes for dealing with newly surfaced vulnerabilities as well as new types of threats, was rapid change. Rapid change of hardware (we all want the latest gadgets, laptops and servers) is facilitated and driven by customer expectations a refresh on these items every few years or so.

And we see even more rapid change in IT software, as patches to some systems are generated once a month, once a week or pretty much any time. We not only tolerate this pattern, we've come to expect it as a natural part of using the latest and greatest (and safest) software.

That of course brought us back to the OT part of our world, and its intrinsically different set of economics, values and certainly, hardware and software lifecycles. For many good reasons, the systems that support our operations centers, generators, transmission and distribution functions, to include both the hardware and the software, have simply not been built to accommodate frequent change. 

And the culture which wraps around these systems, both the users and the suppliers, is still largely hard-wired to make decisions based on comparatively very lengthy spans of time elapsing between changes.

According to Darren, factors that play into the longer OT hardware and software version lifecycles include:

• How a system is built
• How systems around that system are built
• How we use these systems

And a question arose: are systems that are being designed today looking like they're more able to facilitate faster change cycles? Don't think we arrived at an answer on that ... and that means the answer might be "no"

Social Engineering

The panel got a question from an attendee on social engineering, that is, using plain old people skills (e.g., charm, friendliness, charisma, urgency, faux credentials, etc.) to gain physical access to secure areas, access control information, system configuration information, and just about anything else.

All agreed that typical utility workers' (stereotype to follow) inherent goodness and sense of trust and helpfulness made the energy sector more susceptible to this type of threat than say financial services on Wall Street, where (only slight exaggeration to follow) everyone is mean, greedy and suspicious of everyone else

One of the panelists from a testing org said social engineering is 100% whenever they use it (ouch). Though the same person that social engineering assessments often one of the first services lined out by a utility when negotiating a contract for a comprehensive assessment.

Allan Rivaldo, the Texas PUC representative, after he made it perfectly clear that his statements made on the panel were not necessarily representative of his org, followed by saying that Texas takes insider and social engineering threats very seriously.

Disclosure and Information Sharing

Someone dropped a bomb (of a question) near the end. The panel was asked what it thought about the recent public disclose of PLC/SCADA vulnerabilities in the OT products of half a dozen vendors, to include the attack code for each crafted in Metasploit. 

While it seemed like most panelists believed that Dale Peterson of Digital Bond had acted with good intent: to speed up the remediation of the vulnerabilities by their respective vendors, there was substantial disagreement on whether this approach was justified and on whether it would induce the result Peterson said he sought.

One panelist contended that this action was necessary and valuable for "shining a light" on a broken process related to how DHS's ICS Cert works with vendors to resolve known vulnerabilities. The point being, I think, that following the official policies, many vulnerabilities go unremediated if the vendor provides a reason for leaving the vulnerability alone.

But another said that the Basecamp project researchers' unilateral release of vulnerability details and exploits did little except increase the level of risk to asset owners.

The thing that got me was that, knowing the guys on the panel as well as I do, knowing that they are all men of extremely high intelligence and good will, and that they only want what's best for the community, I was really surprised that they disagreed substantially on the issues that the Basecamp disclosure episode surfaced. 

Clearly this is complicated stuff: ethically, technically, culturally. But I think there's no doubt that our thinking is maturing in some respects, and that the industry community, both the users and the vendors, is responding. It will take a long time for Basecamp to fully play out. Hopefully we'll mainly agree, when it does, that it had a net-positive affect on the electric sector's security posture.

MIT Palantir Reveals Future Views of Grid and Grid Security

And as in the Lord of the Rings, few can look into a palantir and walk away unscathed. That's true for this recently released grid forecast from MIT, and especially for the sections on cyber security, which have served as the justification for many alarmist articles since, including:

• Electric Grid's Future: Increased Risk of Attack
• Smart Grid: There will be a Successful Attack
• US Power Grid is a Big, Soft Target for Cyberattack, MIT Study Shows
• Is Smart Grid Security a Losing Game?

What the hell does that last title even mean?  I read the article and still don't get the point.

It's funny but I just went through the security section of the MIT document and couldn't find anything faintly, and nothing that would strike the regular readers of this blog as in any way surprising.

The part that seemed to stir the press pot the most was in the conclusions and recommendations section - it began by stating that no one organization today makes and enforces grid security rules for the entire (US) grid, not FERC or NERC since they only have authority to regulate the bulk grid. Not other groups in DOE. Not DHS. Nor NIST, as its cyber security working groups as they can only recommend, not mandate, protective actions.

So this prompts the MIT report team to conclude:

This lack of a single operational entity with responsibility for grid cybersecurity preparedness as well as response and recovery creates a security vulnerability in a highly interconnected electric power system comprising generation, transmission, and distribution.

And recommend:

The federal government should designate a single agency to have responsibility for working with industry and to have appropriate regulatory authority to enhance cybersecurity preparedness, response, and recovery across the electric power sector, including bulk power and distribution systems.

This sounds right on one level (single source of truth and control) and yet wrong on many others, particularly, as the authors themselves point out, that they are hard pressed to imagine which government organization is equipped or ever could be equipped to take on so monumental a task.

But seriously folks, the MIT report is well worth a look, not so much for its cyber security content, as for its informed prognostications on other aspects of the future grid. There's no need to worry about the Eye of Sauron, or anything else unusually alarming, in this quest for knowledge.

You'll find the full report and some supplementary materials HERE, and the security section begins on page 208.

Image credit: Wikia

Conference Alert: EnerSec Smart Grid Security Summit West 2011

This conference series, the first ever dedicated to Smart Grid Security and Privacy, had a great start last year in San Jose and now returns to California with a head of steam after robust attendance and some very strong content earlier this year in Knoxville.

The lineup keeps getting stronger and this session promises a compelling mix of workshops on day 1, followed by days 2 and 3 with regulator and industry updates, round table discussions and lots of back and forth with what has been in the past a very energized audience.

You can expect a bunch of utilities will be present, and not just the big 3 from California, plus state regulators from CA and TX, fed folks from DOE, NERC, FERC and NIST.  Also, owing to proximity to one of the largest USN bases in the world, we'll likely see some energy-minded sailors present too.

Here's the basic facts for you:

• Dates: 3-5 Oct 2011
• Location: San Diego
• Venue: Town and Country Hotel - click HERE to reserve a room
• For more info and to register for the conference, click HERE

Next week I plan on throwing a few trivia questions at you. Correct answers may earn you a significantly reduced rate for the conference, or at the very least, hearty congratulations.

Photo credit: http2007 on Flickr.com

The Value of Black Hat for Smart Grid Security

When it comes to spotting flies in the energy sector security ointment, perhaps regulators are too polite to utilities, and utilities too polite to their suppliers. No such problem with the security hackers who jump up on Black Hat's global soap box every year and show the world what they've found.

The conference wrapped up last week, and I've got two completely different types of finding for you. One has to do with huge vulnerabilities in the systems related to home networks at the edge of the Smart Grid. The other is targeted at the heart of the legacy grid itself: SCADA systems and the programmable logic controllers (PLCs) that run important transmission and distribution equipment.

• Click HERE for the home network piece
• And HERE for the grid equipment vulnerability demo

Two years ago it was Smart Meter vendors who found themselves embarrassed, in the cross hairs of security pro's, who showed how easy it was to exploit weaknesses in their products. Now attention has shifted to other grid elements. And the beatings continue!

Suppliers thinking they'll save money by moving slowing on improving the security characteristics of their products are playing with fire. The lesson of Black Hat is that they'll be found out. It may not be by NERC. And their utility customers may be focusing on other pressing challenges. But man, sooner or later, the Black Hat crew will be on your case and when they do it'll take more than tons of money to get your troubles behind you.

For this, we should be grateful. Keep it up guys!

Stuxnet marks the Emergence of Real-World SCADA Security Challenges

What kind of Smart Grid security blog would this blog be if it didn't comment on the Stuxnet worm? The short story includes a couple of key players:

• Buried (previously undisclosed, aka "Zero Day") vulnerabilities in Windows. And Windows' security weaknesses used as a starting point for a SCADA attack
• Using USB drives to cross the air gaps and transport the worm from the networked world to the SCADA world
• Attackers acquiring (via $$$ or theft) trusted digital certificates and building them into the attack
• Hard-coded passwords in a Siemens-built SCADA system

If you want a thorough account of how Stuxnet works, Symantec did a bang up job here. But be forewarned, unless you've got some solid development chops, it may be more detail than you can handle!

A treatment better for business folks and arm chair grid security generalists comes McAfee here, or from ComputerWorld, with an initial article here, then this follow-up one week later, here, with input from SCADA security guru Joe Weiss. For the moment, the storm seems to have passed, with Siemens and security product co's offering solutions to clean up Stuxnet code from infected machines, and block it from others. But this story is far from over.

Weiss calls out 170 cyber related outages in the US to date, with 3 of them serious enough to have caused significant (read: expensive) regional outages. He also notes that it's currently impossible to discern cyber attacks from accidental glitches because of the weak state of digital forensics in the power industry to date.

By the way, the 2-way power and data flow Smart Grid, great enabler hacking and attacking, will also improve our ability to do post mortems on cyber incidents, though as with many other types of cyber crime across the Web, it will often be super difficult to pin down the originator.

For me, the big take away comes from the praise security analysts are bestowing on the Stuxnet architects. I don't mean to suggest they support this type of work, not at all. But rather, that this was no casual side-project of some mis-directed youth. Stuxnet is heavy, heavy duty malware. Which means, to me anyway, that there's much more to come, and that the USG and FERC in particular, need to get way more serious about energy control system security, and issue mandatory policy that gets it done throughout the bulk power system and across the distribution network.

We may get some more insight from the cyber security conferences Black Hat and Defcon starting this week in Vegas, where Jonathan Pollet of Red Tiger Security, will discuss (and potentially reveal) SCADA vulnerabilities in utility control systems. Stay tuned ... this is exactly what Joe has been warning us about all along.

More Smart Grid Security Fun: V2G Hacking and Cyber Car Jacking

Thanks to Forrester analyst Usman Sindhu for zeroing in on risks emerging from new sources on the Smart Grid edge. Namely, those related to our increasingly (wirelessly) wired automobiles. At the IBM Innovate conference Jack and I are attending this week, cars came into focus in a way I don't think they have before. You see, this is a conference devoted almost fully to the art and science of software, and cars are made out of steel, right?

Well, for time being, yes. But that's not the end of the story. Besides steel, the typical car of 2010 has over 200 million lines of code. And though ferrying payloads to low earth orbit and docking with the International Space Station are beyond most 2010 models' capabilities, this is far more software than it takes to run the space shuttles. With dozens of applications and interfaces, not only is each one a highly complex system in itself, but if you think about it, each is an intelligent node in a system of systems. Improvements are now rolling out with increasing frequency to safety, navigation and propulsion systems, among others.

Jack has recently developed an auto-fixation, and as he said in a presentation earlier today, the ability to monitor, diagnose, and repair many vehicular problems without expensive, inconvenient trips to the repair shop is a major win for car makers and customers alike. The way he described it, it was almost like techno-nirvana. Until, that is, he mentioned the likely frailty of the software upon which all of this great new functionality depends.

As recent recalls have demonstrated, the cost of loving what software enables is realizing what happens when it goes wrong, whether by accident or from malicious intent. For a drill down, recommend you see this from the Economist on Cars and software bugs, as well as the Discovery Channel's "This Car runs on Code". Karl Koscher et al from the University of Washington spell it out in plain English in their recent paper: "Experimental Analysis of a Modern Automobile":

While the automotive industry has always considered safety a critical engineering concern (indeed, much of this new software has been introduced specifically to increase safety, e.g., Anti-lock Brake Systems) it is not clear whether vehicle manufacturers have anticipated in their designs the possibility of an adversary. Indeed, it seems likely that this increasing degree of computerized control also brings with it a corresponding array of potential threats.

Threats from bad guys are one thing; threats from poor coding, configuration errors and other unintentional companions of complexity are likely a bigger challenge in the near term. Nevertheless, could an attacker work his/her way through less-than-secure automotive communications networks to put drivers in harm's way or adversely impact a utility? Sounds exotic, but when Vehicle-to-Grid (V2G) dreams start becoming reality, and electric cars draw their power from the grid while fulfilling important energy storage functions upon which we come to rely, this is one area we want to make sure doesn't get overlooked. In fact, just like in everything else, we'd recommend minimizing the drama and designing security in from the word go.

Photo Credit: So Fast it Hertz Blog

60 Minutes Sounds Grid Security Alarm

Hat tip to my classmate and former Discovery Channel Powrtalk colleague Chris Davis for alerting me to the show that aired tonight. The popular news journal interviews former Director of National Intelligence (DNI) Mike McConnell, FBI Cyber Division Assistant Director Shawn Henry and others. It begins with cyber crime in the DOD world, goes through some real-world financial services industry examples, and concludes with conviction that the computers that run the Grid have been seriously compromised and that there's little the US government has been able to do to make private operators close out their vulnerabilities.

Remember, the subject here is the current Grid, the pre-cursor to the future Smart Grid, which will bring with it new types of additional abilities but also better ways of isolating some of them when necessary. The segment is called "Sabotaging the System" and you can watch it in its entirety right here, right now ... after a brief commercial, that is.


Watch CBS News Videos Online

Darknet Hackers Grok Smart Grid "Opportunities" for Badness

Darknet is where the cyber good guys explore the dark side, with an eye on rooting out risk and shoring up defenses. This post makes it clear that one of the first thoughts an ethical hacker thinks when they imagine the power grid becoming a giant computer network, is: "[this] is a hackers playground!"

Also this:

The scary part is there’s no encryption and many things are done without authentication, meaning with a little reverse engineering you can probably shut down the power to anyone on the not-so-smart grid.

Here We Go Again

Within an otherwise fine tech intro article on SmartSynch's Universal Communications Model (UCM), comes the type of observation I wish were just fear mongering hyperbole:

Smart grid systems are currently riddled with security holes, but that hasn't stopped utilities from rapidly rolling out smart meters.

You think there's any research to back up this assertion? Or is it likely true cause that's the way we always build: capability first, security last ... if at all.

Danahy: Old Security Habits and the New Smart Grid

JD: This weekend brought us a new security vulnerability message about next generation power, wrapped in the traditional trappings of today's Internet and cybersecurity messaging. The CNN headline reads 'Smart Grid' may be vulnerable to hackers' and the story looked like any of a hundred similar flags waved over software applications, newly delivered services, government infrastructure, etc.

and ...

I think that model is wrong. I am not saying that third-party testing isn't important, but it misses the underlying problems that have allowed the insecure system to exist in the first place. Systems like the Smart Grid need to be developed with a fuller understanding of the purpose, threats, and environment, in which these components will be working. 

Read full post here.