Tuesday, November 30, 2010

Smart Grid Security Lessons from WikiLeaks?

UPDATE: Brilliant IBM colleague Jeff Jonas post on WikiLeaks implications and some potential first steps forward for sensitive-data intensive orgs. Click HERE to read it.
We talked about this today a little on day one of the 2nd Annual Canadian Smart Grid Summit in Toronto. Not sure how the other participants felt, but for me, in the early days of designing and deploying world class security and privacy controls for the electrical utility industry in the wake of WikiLeaks makes me want to stop and reassess. Everything.

From an information security point of view WikiLeaks founder Julian Assange is a villain as dangerous as any penned by Stan Lee. And in Army Private Brad Manning, we've got the perfect lackey ... a worst-case scenario inside threat and substantially misguided youth who may not live to fully appreciate the damage he's caused his country and its allies.

Manning is no Megamind; far from it. The security flaws he overcame were policy shortcomings, not technical exposures.

While no organization is bullet proof, other sectors often point to the US DoD as an exemplar of security best practices. And who knows, maybe DoD has the best policy in DIACAP, the best internal and external guidance in the world, and the best tools and security controls money can buy. But you know what? Nothing prepares you for the thing you didn't see coming.

As North American utilities work to achieve and maintain rudimentary security via NERC CIP compliance, implement best practice cyber and physical security controls in IT and OT, and wrestle with how to best combat future threats as powerful as Stuxnet, WikiLeaks lessons should have them question every foundational assumption about what they're seeking to protect, how they're going to protect it, and from whom.

This Atlantic article, How the Pentagon Hopes to Prevent More WikiLeaks Embarrassments" tries to shine some early light on potential ways out of this morass for the Pentagon and State Department. But for me, pondering enormous Smart Grid data flows, in organizations that never had to segment and store anything like this before, has me wanting to call a time out.

We've all got a lot to learn from Stuxnet and now WikiLeaks. It's much too much in too short a period of time to assimilate. But we've got to try. We've got some big decisions to make in 2011 and we'd better get most, if not all of them right.

Photo credit: Michael Vroegop on Flickr.com

Monday, November 29, 2010

Enernex's Kevin Brown on Intersection of Physical and Cyber Security Challenges in Smart Grid Devices

Thanks to Erich Gunther for promulgating this excellent video Q&A featuring his security-minded colleague, Kevin Brown.

As a cyber guy, I've not imagined physical security as being much more than perimeter fences, surveillance cameras and good locks. Brown's discussions on battery life expectancies, how high you should mount pole-mounted devices, and how easy is to become king of reclosers were all eye openers for me.

Visually, there's not a lot more going on than in My Dinner with Andre. But the content, which truly bridges the physical and cyber worlds, is utterly compelling, fascinating stuff. It's over 20 minutes long, so make sure you find an open spot in your schedule. You won't want to multi-task through this one or you'll miss a lot.

Physical and Cyber Security for a Smart Grid from Erich Gunther on Vimeo.

Stuxnet Visualized

As one often hear's a picture is worth a thousand words, and at 30 frames per second, a good video is worth that much more. Here's Symantec's Liam O Murchu, the same engineer who presented to us at the IEEE Smart Grid Survivability Workshop last month (see post HERE), in a nicely crafted presentation showing how Stuxnet works its (black) magic.

The balloon pop at the end is a good metaphor for what is happening to industry's recently burst beliefs that control systems are safe from cyber attack.

Still looking, BTW, for a nice video, white paper, or even a scribbled note on a cocktail napkin for best practices to defend against future Stuxnets beyond banning USB drives.

Tuesday, November 23, 2010

I Mind this Gap: The Distance Between the Future Smart Grid and Today's Mix of Security Challenges

For a critic of alarmist, sensationalist Smart Grid headlines, I'm a bit surprised the blog editor in me approved this one by the blogger in me. But to dust off a 50 cent word from grade school writing class. it was the juxtaposition of two statements made in the past few days that got me going.

One is a great reminder of the very many compelling reasons we're building this thing from one of the industry's most articulate Smart Grid advocates, GTM's Senior Smart Grid Analyst David Leeds. The other is a sweeping cautionary statement on Stuxnet-like threats last week by one of the most respected security minds in the business, former AEP and NERC CSO Mike Assante, (now CEO of NBISE).

Here are a few snippets from Leeds' piece. First, what the Smart Grid will do for us:
The ... smart grid will not only bring new communication capabilities to mission-critical grid devices and end-user appliances in order to optimize energy efficiency, reliability and security, but will also serve as the enabling platform to plug in the next generation on clean energy technologies, such as rooftop solar systems, wind farms and electric vehicles.
And from an economic perspective, why we need to build it now:
While today’s distribution grids, lacking real-time visibility and control, are largely running blind and consequently costing the U.S. economy approximately $100 billion to $150 billion each year in power outages, tomorrow’s grid, much like the human body’s own nervous system, will have sensory intelligence embedded throughout, giving the grid the ability to anticipate disruptions, and even to self-heal.  
OK, I'm motivated ... let's build this sucker stat!  But hold on ... the gap I'm referring to in the title, is, of course, the yawning chasm between what you hear Leeds' saying must be done, and Assante's message (which we're about to get to), which communicates that as a nation, we're not ready for this.

Mr. Assante is not an alarmist - far from it. In fact, that's why his word counts for so much in this space. But his vocation and experience put him perpetually on the lookout for issues that bring risk to critical infrastructure systems, and when he sees one, his job is to sound a considered, highly targeted alarm audible to senior decision makers, which is what he just did in Washington.

Here's one of his first points - it sets the high-level stage for some of the more granular suggestions he makes later on:
Developing and implementing effective indicators, defenses, and countermeasures to cyber threats like Stuxnet demands that we look not just to the security community but also to the system designers, planners, engineers, and operators of our essential technology and physical infrastructures. We must take a prudent and proactive approach that enhances our ability to learn and apply knowledge fast enough to manage the dangerous consequences that come with these types of attacks. We can no longer ignore known system weaknesses and simply accept current system limitations. We must admit that our current security strategies are too disjointed and are often, in unintended ways, working against our efforts address the highly-advanced security challenges facing our cyber-dependent critical infrastructures.
That's a lot, a whole lot. Maybe too much to hold in main memory. But then he puts a finer point on it, shining light on operational systems ...
No one should be shocked that cyber exploits can be engineered to successfully compromise and impact control systems. Study after study has identified common vulnerabilities found across control system products and implementations. The exploitation of a hard-coded password design in one vendor’s implementation will not be an uncommon or isolated occurrence.
And finally, towards the close, here's one of several actions he recommends:
Require critical infrastructure asset owners and control system vendors to report industrial control system specific security incidents and the U.S. government must provide up-to-date information to asset owners and operators on observed adversary tactics and techniques, especially when investigations reveal attacker capabilities to side-step or exploit relied upon security technologies.
Not a full solution, mind you, but certainly a firm step in the right direction from where we are now: make more information available to the community so we can more quickly adapt and update our defenses. Today in the energy sector, there's nothing like this. Hence, a gap in knowledge.

Then there's this: we're concerned that Stuxnet's massive attack penetration strategy that defeated most current cyber defenses, armed with more broadly targeted payloads in future versions, and it's definitely getting attention. But less obvious, yet almost as much of a concern. is that a focus on High Impact Low Frequency (HILF) a.k.a., advanced cyber threats, might prompt utilities to take their eyes off more mundane, but nevertheless serious, day-to-day attacks on their systems.

This second gap is the one in setting security priorities ... between preparing for advanced threats as well as ensuring that essential security best practices and defenses are maintained to combat everyday threats from malware, criminals, insiders, etc. There's crawling, walking, then running, and so far on securing the electrical infrastructure, most would say we're crawling.  And then there's walking and chewing gum at the same time: preparing for diverse threats and doing a good-enough job on all of them. This is not a job for wimps, and it's going to take a long time before we see significant progress.

So let's end with David Leeds, alright? When security challenges seem overwhelming it's always helpful, for me anyway, to revisit why we're putting ourselves through all of this in the first place.
[The] U.S. is hardly alone in promoting smart grid as an economic growth engine; virtually every major economy is now either piloting or deploying smart grid technologies, and it’s now understood that you can not run a digital 21st century economy on a 20th century grid.
Maybe we can fuse Leeds' economic drivers with Assante's security cautions and recommendations and come up with a middle-path approach that keeps attackers at bay and keeps the LED lights burning bright.

Click HERE for more on HILF threats and what we might do about them.

Photo credit: Cindy Andrie on Flickr.com

Sunday, November 21, 2010

Massoud Energizes U Minnesota Smart Grid Ad

Nov 22 Update: I'm speaking at the Canadian Smart Grid Summit next week in Toronto, and when I went to check for my time slot, noticed that Massoud is headlining!  See for yourself HERE.


As this Wall Street Journal video points out, the majority of TV ads for colleges shown during football halftime breaks are cookie cutter simple and formulaic.  This spot, though focuses on several recent ones which break the mold. Most notably, from the SGSB's point of view, is the one from University of Minnesota featuring long-time clean tech and Smart Grid security advocate, Dr. Massoud Amin.

Here's the WSJ piece that makes the case:

And for the full 30-second U Minn energy ad they're applauding, click HERE.

Production standards are so high and the content so compelling, you might think you were watching an IBM commercial.

Wednesday, November 17, 2010

A Few Pointed Suggestions for Improving the NERC CIPs, and in so doing, Grid Cyber Security

This short article released on the ControlGlobal site last week addresses technical issues, but defines its terms and acronyms well enough to be understandable to business readers.

Key points are:
  1. Using spot checks on systems to go beyond the current paper chase approach to validating CIP compliance; and,
  2. Acknowledging that attackers and malware will find ways around/through current "outer wall" based network defenses, instituting a less perimeter defense-oriented approach to security controls with guidance on use of DMZs in between internal networks
These guys are aiming for "actual security" versus faux security via a pure compliance choreography. You may not agree with all the guidance. Depending on your orientation, you may think this is too much ... or too little. Or you may find that some of the recommendations would increase costs for stakeholders, but overall, I believe this is potentially helpful stuff.

Monday, November 15, 2010

Upbeat Utility Economics Update

When you're in the trenches with utilities looking at day-to-day challenges with a lot of granularity, it's easy to lose track of the bigger picture trends. For example, we're almost always talking about how many utility folks (internal and contracted) it takes to implement NERC CIP compliance programs. It's a lot of course, especially for orgs who always feel resource constrained ... and of course, are aging by the minute.

And the fourth version of the CIPs with its expanded scope only promise to add to the workload, and the expense.  But guess what?  High above these electric sector security and governance skirmishes float financial analysts.  Picture them as smartly suited genies on flying carpets woven from $100 bills, foretelling the economic future sector by sector.

And what are they saying of our beloved one? Here's a starter from "Utility Stocks Energized" in this past Sunday's WSJ:
"It's funny to say 'growth' and 'utilities' in the same sentence, but it's more of a growth sector than people think," says Jamie Cox, managing partner at Harris Financial. What's powering this growth? A building boom. Some higher-potential utility companies are upgrading their power plants, building out transmission lines or expanding into renewable-energy markets such as solar -- all of which could help boost future profits and dividends.
So how do you like that? As various pundits ponder the lethargic pace of the clean tech revolution and others pronounce it much ado about nothing, those in the rarefied air of the brokerages see what's plainly in front of everyone's noses, and signal that it is good.

Will "energized" investors' new flows of money further spur the infrastructure modernization and build-out of Smart Grid capabilities? How deep into a utility operation might those funds trickle down?  And if the money does come, how soon can it be expected? I might have to leave all of this to my MBA friends, but IMHO anything that communicates confidence in the economic vitality of the sector only serves to embolden the community further.

And what of security? Sounds like there are going to be a lot of new and somewhat complicated systems to protect. And maybe, maybe more so than in the past, it might just feel like there's some money available to afford the necessary protections. We'll see.

Thursday, November 11, 2010

Electric Infrastructure Physical Security "Wrong Way " in Ohio

Not sure even the most robust physical security controls could have prevented this crashing chimney-induced local loss of service. As Chrissie Hynde of the Pretenders put it: "Way to go, Ohio". How did this substation arrive at this sorry state of affairs you may ask?  See for yourself in this short and scary video:

Guess from a security point of view, we'd have to catalog this one under "some things are just out of our control" as energy security policy wonks, right next to city busting asteroids and mid-continent nuclear explosion-generated EMP bursts.

Here's the full page of pictures and the article on MSNBC's photoblog page.

Photo credit: MSNBC

Monday, November 8, 2010

Don't Bully Brave Smart Grid First Movers

Just a short one this week, but with a point I think is well worth airing. A few months ago I wrote a post called "Security isn't the Biggest Threat to the Smart Grid" in which I linked to, and commented on articles taking a previously lauded utility and its partners to task for mistakes that appeared obvious in hindsight.

All I want to say is that we're all in exploratory mode and will be for some time. Much of the technology is new, the standards are still forming and the new business models are embryonic at best. We should profusely thank each and every utility that has the guts to move out early and take a few calculated risks. From them we get early views of what works ... and what doesn't, that can be leveraged by all who follow.

I'm sure that some customers and regulators will disagree, but from this lofty perch, you won't  hear me beat up on any utility for taking the lead on security or other actions that help bring the shape of the future Smart Grid more clearly into view for all of us.

Photo credit: http://www.flickr.com/photos/pointshoot/

Monday, November 1, 2010

Takes Two (or more) to Tango: Building a Foundation for Smart Grid Security with International Allies

Anyone who's pondered the enoromous challenges ahead of us immediately recognizes that Smart Grid security is a team sport. We struggle to get the US's smart grid standards house in order, with a mix of Federal leadership and hopeful cooperation among the 50 state utility commissions and across our dozen or so regions. It remains to be seen how much team spirit emerges from this effort. Yet even if we make good progress, electrical infrastructure security at home is no guarantee of national energy security.

Fossil fuel sourcing and climate change issues aside, US economic (and to a lesser extent, military) well being would be significantly impaired if our key allies and trading partners had their grids knocked out by successful and sustained cyber attacks.

While many may grumble that the NERC CIPS are not nearly robust enough, a scouring of available online documents reveals much less attention is paid to cyber security requirements in E&U project planning. I will be travelling to Europe this week to deliver some training so will attempt to get my own first hand findings from the field, and will report accordingly.

But a look at some of our closest international buddies: Australia, Canada, New Zealand, and the United Kingdom reveals a desire to leverage US resources and lessons learned to the benefit of all. The International Electricity Infrastructure Association (IEIA) recently met in Washington, DC, and from what I heard through the grapevine, these folks are all interested in knowing more about what we're doing, and in some cases, will base their moves on what they see us doing.

Here's what the IEIA lists as its objectives:

  • Founding participants defined the following objectives for the IEIA Forum, as directed by an international Steering Committee representative of participants:
  • Enhance protection of the electric infrastructure of Australia, Canada, New Zealand, the United Kingdom and the United States.
  • Stimulate active involvement of electric sector and government stakeholders and participants
  • Provide a framework for collaboration among represented countries on a government-to-government, industry-to-industry and government-to-industry basis
  • Identify and address infrastructure assurance priorities
  • Align government and industry participant efforts to identify common initiatives and deliverables
  • Share experience, information, solutions and other mutually identified resources
What's not to like on this list? I'd like to see something comparable covering Europe via the EU, and for our friends and allies in East Asia, something similar. Sorry if this is a little too kumbaya for some of you, but that way it goes sometimes. Will get some extra rugged individualism into the blog soon.

Photo credit: http://www.flickr.com/photos/zabara_tango/