The Harsh (Security) Reality of Old Software in the Current and Future Grid

You know I'm always advocating for getting better security awareness and process into everything we do with Smart Grid software. Whether it's developing policy, getting tools and building secure processes into the software development lifecycle (SDLC), and/or educating and arming orgs on what questions to ask software vendors re: the security rigor they include (or fail to include) in the development and integration of the products they market.

Sounds like a pretty good idea. Could make the Smart Grid and utilities' worlds and the North American grid infrastructure a whole lot safer and better, right?  

But then someone comes along and rains on my parade. And not just anyone, but a card carrying grid guru with more experience in this field than just about any other mortal. And what does he say?  This:

One must keep in mind that there will be far more poorly coded, totally untrustworthy firmware and software in the field for decades (the existing installed base) than new, more secure systems following sound development practices installed over the same time period. Dealing with this reality and the fact that the old stuff will not be ripped out should be a priority.

"Thanks" to Erich Gunther of Enernex. So, sports fans, while I and others keep beating the drum for more-secure new software, would a few of you mind getting on the challenge Erich's pointing out? Like, right away please.

Beating Stuxnet to Death (Before it Beats Us)

If it feels like I'm belaboring the importance of understanding Stuxnet, it's because, IHMO, it's a threat well worth belaboring. Stuxnet is Mother of all industrial and utility sector cyber wake-up calls. And if you're an asset owner asleep at the wheel, it could be your momma, and your daddy too (see: who's your daddy?)

As I mentioned in a previous Stuxnet rant, good security tools and best "defense in depth" practices are a less-than-complete defense:

No matter how solid an org's security policies, no matter the level of adherence to defense in depth principles and security best practices, no matter how much security technology was deployed and how up-to-date it was kept, it is very likely that Stuxnet would have found a way in.

Now here's a real expert, Andrew Ginter of Industrial Defender on the excellent Findings from the Field blog, laying out the harsh reality of the Stuxnet wake-up from a (NERC and DHS) security standards point of view:

A site protected with whitelisting/HIPS ... would have been CFATS or NERC compliant, and would have been protected from Stuxnet. Unfortunately, I am aware of only a handful of such sites, and no HIPS protection is mandated by NERC or CFATS. Sites with only anti-virus deployed are seen by today’s regulations as having adequate malware protection, but that protection would not have prevented Stuxnet compromises in the first year the worm circulated.

If you're new to whitelisting, here's a ZDNet blast from the past in 2008, featuring Microsoft security guru Scott Charney making the case that whitelisting is the future for most/all successful cyber security strategies. From my understanding of this approach, it's a huge step forward from where many orgs are today. But I also recall hearing Symantec's reverse engineer and Stuxnet expert Liam O' Murchu saying he thought Stuxnet could/would potentially morph to circumvent whitelisting defenses. Yikes.

Nevertheless, NERC and NERC CSO Mark Weatherford have been busy issuing guidance to utilities on how to best combat Stuxnet and Stuxnet-like threats. We're not privy to the actual details of that guidance, but you can gain a little insight into NERC's actions here and here.  I'm not sure it's a Stuxnet defeater, but I for one am quite happy to hear Weatherford calling for more security in software development and sourcing processes.

Regarding preparations for future versions of Stuxnet targeting electrical infrastructure, forget compact fluorescents for the moment. Got midnight oil? Start burning it.

Much improved sub-optimal defenses and recovery plans are vastly more desirable than what we've got in the field today.

Utilities could shoot to Roll with Stuxnet Junior's Punch - an SGSB Reader Chimes In

Got this comment in response to my most recent Stuxnet post - Surviving Stuxnet and its Offspring. It's from an IT security pro at AEP:

A viable question is:

If we know we can't practically defend against Stuxnet or its spawn, what is our approach? Giving up is not an option. "Roll with the punch" may end up being a viable strategy. How could we design control systems, or other IT environments for that matter, to be resilient enough to take a potential knock out punch and yet be able to come back up swinging? Another question may be, "in the end, can we optimize our investment by planning to take the punch rather than futilely hiding from it?" 

I think this is a great way of conjuring where we were trying to go (mentally) at the recent Smart Grid Survivability workshop, and where we need to get to asap as an industry. 

Too (Much) Smart: Meters, Grids, Cars, Phones

"Smart" in the electronics sector generally connotes a device with a processor and some built-in communications, though sometimes it's just meant to convey coolness. But as the media increasingly links "smart" with "dangerous", marketers may need to find another strategy soon.

Of course, this doesn't bode well for consumer adoption of Smart Meters and the Smart Grid. Angst is bubbling up in the ranks of those who leave comments below cautionary and increasingly inflammatory online articles. For example, here's a surprisingly coherent entry found beneath a recent post on looming cyber issues with "smart" cars:

If we're not careful, we'll end up changing the definition of the word "smart". "Smart" = dumb enough to be cracked and hacked. We'll have this issue with smart phones, smart cars, the smart grid, smart appliances, not to mention our regular computers.

He's right of course, and that's a big part of the challenge, along with the media's desire to document and propagate this assertion, and drive fear, uncertainty and doubt (FUD) deep into the mass market.

Like successful TV shows that eventually Jump the Shark (wander too far from their original concept), all marketing fads also eventually run out of steam, after which point they become comical if not pitiful. This will eventually happen (if it hasn't started already) with the prefix "smart" automatically placed in front of every new gadget and appliance.

And when that happens if not sooner, we might want to find a new term for what we now call Smart Grid. It's been called other things before; another name isn't going to hurt. And no, I don't think "Super Smart Grid" will do.

Photo credit: Ivan Walsh on Flickr.com

Stuxnet Update V: Surviving Stuxnet and its Offspring

Though I wouldn't look for a movie version any time soon, like the Davinci Code for Smart Grid and other cyber sleuths, the story of the Stuxnet worm keeps getting more and more mysterious.

At the IEEE Smart Grid Surivivability workshop held at SEI in Arlington, VA last week, we had a front row seat for a great presentation by Symantec's Liam O'Murchu, one of three Stuxnet reverse engineers Symantec has had on the case for over three months straight.

Though I've been following Stuxnet on the SGSB (first post HERE) since shortly after it surfaced (well after it was born circa 2009), Liam provided some insights that surprised all of us I think, including:

• To escape detection while targeting every Windows OS from 2000 to 7, the attack team purchased each and every version of all anti-virus products for each OS and then designed Stuxnet to ensure they wouldn't be noticed by any of them
• Stuxnet is evolving its capabilities to infect systems and replicate within an organization, yet its payload remains unchanged. Meaning: the target remains the same ... and maybe the attackers aren't yet satisfied they've accomplished their mission
• On the human-interest side, he noted that the reverse engineering paths he and his colleagues have been following are the same or similar to those blazed by the the team who crafted the attack.Though lots of evidence points that way, Symantec (unlike Ralph Langner and others) is not ready to say that Iran's nuclear operations are the only or primary target of Stuxnet. There are still several parts of Stuxnet they've yet to crack and their research continues
• In addition to phenomenal anti-virus evasion techniques, Stuxnet includes lots of other stealth approaches and myriad attack strategies for getting passed OS defenses, through firewalls, increasing its privileges, and much, much more

In short, no matter how solid an org's security policies, no matter the level of adherence to defense in depth principles and security best practices, no matter how much security technology was deployed and how up-to-date it was kept, it is very likely that Stuxnet would have found a way in. We're very lucky that the apparent target doesn't seem to include systems important to the US or our allies. This is clearly focused on very, very specific control elements like certain make/model pumps and actuators. If it doesn't find exactly what it wants, it does nothing else. It's polite. That's good news.  So we got our wake-up call.

But the bad news is that for aspiring bad guys, Stuxnet is a master class, a surprising visit from "attacks of the future" to present day 2010 on how to do more damage than you ever thought possible. We'll see Stuxnet again, and if it's pointed at us (US utilities, other industrial operators) next time the payload may be quite different.

Written by Liam and team, Symantec's 51-page Stuxnet Dossier remains the definitive document on Stuxnet.  We'll be hearing more from them as they (and others) make new discoveries, but there's already plenty of info available now on how to begin hardening your org against the future spawn of Stuxnet, even if those defenses might be less than complete.

Photo credit: Digipam on Flickr

Common Sense and Common Knowledge

At the 2010 RSA Conference in London this week, long-established security visionary Ira Winkler was giving a speech entitled "If you tweet what you had for lunch, you deserve to be robbed". It was a very entertaining presentation about the amount of information people are unintentionally sharing into a public environment that is populated with both well-meaning and ill-intentioned folks. Perhaps a summary would be useful here, but that isn't really the point of this piece.

During Ira's presentation, he discussed the linked concepts of "common sense" and "common knowledge". In the social networking community, a lack of knowledge among many, particularly the young, about how all of this sharing could really hurt them, leads to decisions that we see as stupid, as lacking any sort of common sense about privacy, propriety, and personal space. As he was describing the disconnect between these adult values and the narcissistic need to share, I started to think about the challenges we are seeing in achieving a real and consistent set of common goals or methodologies as we work to secure the Smart Grid.

We see some organizations expressing security in terms of reliability, others in terms of privacy, still others in terms of financial justification and utility viability. A quick couple of keystrokes brought up some examples:

• NRECA has provided some content that is customized and adapted to various smaller utility newsletters that talks about "Balancing Smart Grid Buzz with Common Sense". It presents a view of the coming Smart Grid in more conservative terms, tamping down some of the projected customer enthusiasm about new features with a strong dose of cautionary logic. The Dawson Public Power version of the piece closes with:

  "There’s a big difference between being on the cutting edge or the bleeding edge of technology. Dawson Power wants neither. We want the “proven edge”..."



• On the other hand, common sense means something very different to some Smart Grid deployers in Texas. According to an article in Electric Light and Power, It is about evolution and revolution:

  “Texas is the one I always point to, and the main reason, I would say, is they are taking a very common sense approach,” [eMeter chief regulatory officer Chris] King said. “The legislature passed a law saying, ‘We want smart meters.’ They didn’t spend 10 years trying to boil the ocean. They have home area network interfaces in the meters, as does California, but in Texas they’re already live. California is a year away, maybe two."
  
  “Texas knows they’re making mistakes—they’re small—and they make a fix.”



• In April, the New York Times carried this thought on a differing style of Smart Grid common sense:
  

  ...Ralph Izzo, chairman and CEO of New Jersey's Public Service Enterprise Group, said better marketing may not be the answer to addressing the gap in consumer understanding of electricity use or changing consumer behavior.
  
  "I think we tend to overstate the contribution that sophisticated technology can and should make," Izzo said.
  
  "I feel like just shouting, 'Stop. Apply some common sense,'" he said. "Before we start championing multibillion-dollar investments in smart grids that control set-back temperatures on refrigerators because there is or isn't going to be a Super Bowl ... we need to get folks to caulk around their windows,"
  

So what do we do with all of this?

The fact of the matter is that there does not exist a common base of knowledge, objectives, or outcomes, that can be applied to the megalithic, polymorphic, thing we think of as the Smart Grid. This means that individual organizations, regulators, customers, and implementers will likely have a different basis from which to develop appropriate solutions and timetables. As so often happens, the definition of common sense is not so common. That isn't because the concerned parties aren't sensible, it's because they are highly sensible to their own uncommon needs.

This teaches us a new lesson, that solutions and proposals need to be very specific in their goals and rationales, and organizations must establish a common base of knowledge for discussions on any proposal's merits. Only with that shared understanding can we rely on the "common sense" of good people to create solutions that will ultimately make sense for the common good.

Image courtesy of Casey Brown

Renewables Grid Giga-boost: Google and Friends Commit to Fund Undersea Wind Power

Not just offshore, mind you, but offshore and under water.  We're talking high voltage transmission lines in the deep blue sea off the USA's east coast mid section. If you're thinking this is another green jobs initiative from the current administration, you're wrong. It's the private sector doing what it does best: seeing a problem, doing some analysis, realizing it's an opportunity, and putting some skin in the game despite known and quantified risks.

Covered in all the major news outlets today, including the WSJ, this is great clean tech news as well as energy security news. Here's why:

• It's a win for renewables as it'll now be much easier and cheaper (and therefore, much less risky) to deploy big offshore wind turbines 
• It's a win for energy security as one of the most congested parts of the national grid will have more pathways and options for routing electricity, especially in the NY/NJ region
• This should help the perpetually stalled Cape Wind project get out of the blocks. If folks down south can pull off a wind infrastructure project of this magnitude, how come forward looking, business minded, PhD-educated, renewables friendly northerners have been arguing about this modest first step for 10+ years with nothing to show for it? Wind energy in Massachusetts is in danger of being OBE - overcome by current events

For me, the second point on energy security is also a boost for Smart Grid security. Absent hostile submarines with cable cutter-enabled frog men, this transmission addition will give grid operators more room to breath, even as it makes it more likely they'll be figuring out how to best manage gigawatts of new intermittent power over the next several years. We'll be relying on more technology to handle this challenge of course - here's to ensuring it's developed and deployed with security in mind: up front, built in, and by design.

Cyber Security Car Talk

Though I'm writing you from the land of Click and Clack, this piece is about a topic you'll probably not hear covered on their show. CNET journalist Elinor Mills, who I had the pleasure of meeting at the first Smart Grid Cyber Security Summit in San Jose in August, recently keyed: "Cars, the next hacking frontier." And as electric cars (and cars in general) have been on my mind lately, this really caught my eye.

As we've noted in previous posts, there are some surprising similarities in the ways previously isolated systems are being (often wirelessly) connected in the electric and automotive sectors. For most consumers, computers + code + communications = fun. But for security watchdogs, these same elements = trouble. And ultimately, cars and the grid will marry (and their coupling will produce precocious new security challenges) in a space industry calls V2G - meaning Vehicle-to-Grid.

Elinor links to an earlier CNET article of hers, "Hacking a Car", in which Stefan Savage of UC San Diego invokes history to make the connection:

If you look at PCs in the early 1990s, they had all kinds of latent software vulnerabilities. It didn't matter so much because PCs were at home and not connected to everything else. Then they were connected to the Internet and the latent vulnerabilities were exposed to outside attack. We see cars moving in much the same direction. There is a strong trend to provide pervasive connectivity in cars going forward. It would be good to start working on hardening these systems and providing defenses before it becomes a real problem.

And so it begins. I've begun research for a white paper on vehicle and V2G cyber security which I will try to have ready in early 2011, if not before 2010 is through. What's the motivation?  Here's how one gloomy CNET commenter cast it:

Someday the cyber terrorists will strike, locking everyone into their cars and disabling the engines, thus ensuring a swift and bloodless invasion of the United States. Then it will be up to the Amish to defend the country. We is doomed ..." 

I beg to disagree on three counts:

1. The Amish are tougher than you think.  See this short clip on Amish Rake Fighting
2. Bikers are even tougher than the Amish, and they won't be locked out
3. We're going to figure the security angles out up front and make sure cars remain as safe or safer than they are today -- though I'm not sure how safe that is

Photo credit: TechnaBob.com

New SGSB Webcast is Live

SGSB Webcast 5: Smart Grid Software Security

View more webinars from Andy Bochman.

While it's fun to think of all the great new gadgets and devices that are enabled by the Smart Grid (and that the Smart Grid enables), none of them could even begin to work without the "invisible glue" out of which the entire enterprise is being constructed: software.

As we rush to deploy Smart Meters by the millions, consumer portals, HANs and iPad applications that can communicate with them, meter data management systems (MDMS) to handle the tons of data that's generated, electric vehicles (EVs) to push local electric infrastructures to the limit, and synchrophasers across the continent to give us a better view of "the greatest engineering achievement of the 20th century", it's important to not forget about software just because we often can't see it.

Misconfiguration of software assets and (usually unintentional) vulnerabilities in code are the primary pathways hackers use to breach systems, alter their behavior and reach sensitive data. This presentation is more of a "why to" than a "how to" manual. There are plenty of the latter and I'd be happy to point you to some. But the reasons for taking on this challenge are compelling, and IMHO, need to get out.  

Enough already, here you go. It's about 17 minutes long, and you'll like it better if you make it bigger (click on "Full" icon in the lower righthand corner).