The Fruits of Smart Meter Phobia

OK, so you don't want a wireless Smart Meter on the side of your house because you're sure, despite copious scientific evidence to the contrary, that its radio frequency emissions are going to kill you.

Well, after organizing and making your intentions clear, you have won. Congratulations! You can have it your way and keep the darn thing off your house. One small catch, though: you'll cost a lot more money to support so you'll have to pay extra.

We're working on modernizing the grid so it can support greatly increased amounts of intermittent wind and solar energy. We're trying to reduce our use of, and dependence on, fossil fuels, which will make our world a healthier place by far. Smart Meters have an important role to play by giving utilities a better picture of near-real time energy demand, as well as the means to manage demand during periods of peak consumption.

So, about that cell phone you press against your head? And the computer screens you stare at all day. And the wifi router that forms your home network. And the microwave that's running sometimes while you tidy up in the kitchen. You've tolerated, if not embraced, modernization of other sectors of the economy. Please be a bit more consistent with your fears and let us get on with our work.

Image credit: Zazzle.com

Next Gen NERC CIPs Taking Shape in early 2011

Previous posts have tried to give readers a hint at what lies beyond the veil re: versions 4 and 5 of the NERC CIPs. More scuttlebutt has been arriving over the past week or so; heard it through the NERC Standards Development Team (SDT) grapevine. As always, please consume this forward looking stuff with a grain or two of NaCl:

• The SDT has decided to leave the impact levels as they originally were designed based on FERC’s request to do so in version 5 of the CIP rules
• This means there will be high, medium and low impact levels
• Encryption will be a requirement in version 5 for all medium and high impact systems
• Utilities will have a few years to implement new version 5 controls since version 5 won’t go into effect until mid 2013 or so. 
• It is estimated that there will be an additional 20-40 new measurements that the medium and high impact systems will have to incorporate…uncertain on what those are going to be at this point
• And this train has been coming for some time now: the terminology for CIP-002 will change from “Risk Based Assessment Methodology” to “Bright-Line Criteria”

Since January 2008's final ruling by FERC on Order No. 706, the industry has been moving, not necessarily steadily or with great speed, towards a more robust articulation of security standards in each subsequent version of the CIPs. From the cyber security practitioner's point of view, it appears the sector is going to be in a stronger position in a few years. Here's to holding it together until then.

Town Hall Announcement: Obstacles to Energy Sector Information Sharing

Of course, you've already missed Austin's mighty SXSW by the time this event rolls around, but still (not including folks who don't enjoy 110 degree temps sometimes) when is it not a good time to visit Austin?

Besides, a town hall is more intimate and approachable than a conference, right?  Well, there's good news. One of the biggest challenges in our space is getting some attention in April and you're invited to participate. Here's what you need to know:

• Who: our friends at EnergySec are hosting and William Bryan, Dep Asst Secretary of Infrastructure Security & Energy is keynote
• When: 27 April 2011, 8 am - noon
• Where: ERCOT Austin MET Center, 7620 Metro Center Drive, Austin, TX  78744, Room 206
• For more info and to register, click HERE 

Photo credit: Stuart Seeger on Flickr.com

A Creepy Anniversary to Consider

If you think about it, we're here writing and reading about threats and defenses against threats to energy sector networks and software-centric systems because a long time ago, certain smart folks, some just curious and of good intent, others curious and dare I say it, evil, experimented with how they could manipulate computers across a network.

There's been so much heavy duty news lately that the 40th anniversary of the first computer virus is happening below the radar. But if you're curious and not evil, here's a nice short take on the first virus, called Creeper, by the Discovery News with some excellent links to more info.

And BTW, if you're a music buff, I've got a very different Creeper for you here - it's a version from the mighty blues harp master James Cotton on YouTube.

Combating Smart Grid Vulnerabilities ... and Ourselves

In the previous post I attempted to communicate the urgent necessity of setting some performance metrics for ourselves, with the objective of demonstrating to the senior decision makers who sponsor our activities that what we are doing is bearing fruit.

That the sum total of all the money spent on Smart Grid cyber security products and services, plus the monetary and human resources dedicated to the task of formulating solid interoperability and security standards is producing demonstrably more secure utilities and a demonstrably more secure and increasingly smart grid.

Well, the Journal of Energy Security just published an article called "Combating Smart Grid Vulnerabilities" in which my senior colleague, Grid Wise Alliance Chairman emeritus and current Chair of the Global Smart Grid Federation, Guido Bartels makes a case that we seem to be making reasonable progress ... that we're successfully grappling with what we think we know about the security weaknesses in this system under construction. And I can only agree with him.

But he also acknowledges that it's really hard to say for sure. And backs that with the recently published findings of the GAO and the DOE's IG office. A section of the article called "Don't get too comfortable" states:

The [IG report] issued its report on this matter ... in which it found FERC cyber security standards (as implemented by NERC) and the overall approach for regulating the national grid quite lacking, saying: "… even if the standards had been implemented properly, they 'were not adequate to ensure that systems-related risks to the nation’s power grid were mitigated or addressed in a timely manner."

My response to this is: how would the DOE IG, or anyone else for that matter, especially those who aren't working energy and cyber security 24/7 know if and when implemented standards and controls were adequate? We haven't defined adequate and we measure almost nothing because we've told ourselves two things:

1. It's too hard to measure cyber security, especially in the energy sector, and,
2. We can't talk about anything that might be helpful because the info is too sensitive

I agree with Bartels that we are making progress. But how we convince others of that is another matter. There are plenty of MBA's out there and enough Deming disciples to know that we're fooling ourselves if we think that progress is self evident ... that it's obvious to all observers that activity equals efficacy.

Let's admit the emperor is stark naked, get him some decent garb, and build an increasingly secure Smart Grid, the security level of which can be communicated to ordinary folks ... including non-technical senior executives and congressmen.

Smart Grid Security Truth: You Can't Do What You Don't Measure

Are you part of a Smart Grid security task force, working group, support group?  No?  Look to your left and look to your right. Chances are, one of those folks is. It's getting pretty crowded, with many folks and organizations toiling away trying to figure out what a future-state secure Smart Grid should look like layered on top of our largely insecure and aging legacy grid. Two thing's are certain: there are lot of us, and we're awfully busy.

It reminds me of the wood chopping anecdote inside Steven Covey's Seven Habits of Highly Successful People, which goes something like this:

A group of loggers is busy chopping away doing great work under the supervision of the managers and achieving high productivity and throughput. Someone from a mountain overlooking the forests notices something and shouts "hey, you down there ..." Reply: "we are busy, and making great progress" ... and the person on the mountain yells "Wrong forest!"

Which is to say, we can chop all the Smart Grid security wood we want, but if we don't come up with a way to show our mountain top-dwelling managers that we're working in a forest that matters to them, then it's all for naught. We have remember that these are the folks who not only write our paychecks, but also approve the regulations, and who fund the R&D and ultimately purchase the security products and services we present to them as solutions.

You know and I know that increased emphasis on (and competence in) cyber security is an absolute must if this grand initiative called the Smart Grid is going to succeed. Whatever would keep anyone, you might ask, from aggressively funding our activities and the security of this most critical piece of critical national infrastructure? Is robust Smart Grid security not as American as mom and apple pie? (Other countries may have to substitute patriotic food stuffs here ... I'm going to assume reverence for mom is universal).

Well, the answer to why we have to struggle for every last scrap of support is painfully simple: it's because most executives and government leaders perceive no improvement beyond status quo ... no change for the better, from the current level of cyber risk the nation's electric utilities are already carrying.

Put yourself in their shoes for a second. Would you continue to allocate scarce human and financial resources, or prioritize legislation, for activities for which their is no clearly discernible business impact/result/payback?

Look around inside our own tightly knit community and you'll quickly see that even the true Jedi masters have no ready tools for objectively describing the current state or for referencing indicators that reveal improvement  to outsiders.

So, how might we know if our many activities are helping? Why through measurement and reporting, of course. And some folks out there have mentioned this to us in none-too-subtle a fashion. In the recent Government Accountability Office (GAO) report titled: "Electrical Grid Modernization: Progress Being Made on Cybersecurity Guidelines, but Key Challenges Remain to be Addressed" lack of measurement tools was one of the primary findings:

The electricity industry is ... challenged by a lack of cybersecurity metrics, making it difficult to measure the extent to which investments in cybersecurity improve the security of smart grid systems. Experts noted that while such metrics are difficult to develop, they could help compare the effectiveness of competing solutions and determine what mix of solutions combine to make the most secure system.

Furthermore, our experts said that having metrics would help utilities develop a business case for cybersecurity by helping to show the return on a particular investment. Until such metrics are developed, there is increased risk that utilities will not invest in security in a cost-effective manner, or have the information needed to make informed decisions on their cybersecurity investments.

So, to help keep this long post from getting too much longer, I recommend a couple of things to you, dear reader:

1. First, read the recent Gartner Group brief called "Why Communication Fails: Five Reasons the Business Doesn't Get Security's Message" by analyst Jeff Wheaten. It's excellent, and helps map out what's lost in translation when executives try to understand security in their orgs but can't fathom the highly technical, specialized language that's used to describe it. It has some excellent recommendations for improvement, and while it's not energy sector-specific, it doesn't need to be. (Note: unless your org is already a Gartner subscriber, it's going to cost you a bit, but nothing close to what it costs having the funding rug pulled out from under your feet)
2. It's easy to think of reasons why security metrics (or if you'd prefer, measurement) are difficult or impossible to do in our sector. So take that as a challenge and come up with one or two, preferably nice and simple, that'll have people saying "man, that's brilliant". I'd prefer they were high level and didn't require near-realtime sensor readings and massive analytics. Hint: how about something along the lines of Smart Grid and security maturity models?

Still with me? OK, let's do this thing.

Photo credit: tmorkemo on Flickr.com

Night Dragon Reveals Shallow Defense in Depth in Oil & Gas Sector

Click to enlarge

Last month I did an initial post on the Night Dragon attacks, none too pleased that another one of these creatures was on the lose in our industry. Turns out my colleague, security ace Bruce Mayhew's been reading up and pondering on how the oil and gas companies that were targeted could have been caught with their collective guard so down. Here's Bruce and brace up - it gets a little technical:

What ever happened to defense in depth? Look at this modified security stack of defense mechanisms that could have prevented or at least gave earlier warning of the Night Dragon attack. Note: this is not a complete security stack, but a visualization of the many areas that were left unattended that led this to the success of this attack.

This post is only focusing on the portion of Night Dragon that allows the attackers to get RAT installed in the host environment: SQL injection. First off, parameterized database access stops SQL injection cold. And since we're talking about database, let's add in the concept of least privilege for the database functional account. Why was the database account setup to allow ANY access other than reading the database tables? If the application allowed writing of database data, then you would need read/write privileges. While either of these privileges would have potentially allowed for exploiting or corrupting the data, it should not have led to complete system compromise.

OK, let's assume the target application was using a technology that didn't allow for parameterized database access, the next logical defense would have been whitelist validation on the server. While not a fool proof strategy for preventing SQL injections it certainly would have limited the SQL injection attack vector. Now that we have server side whitelist validation, let's add in the exact same validation logic on the client or client-side validation. There is no direct security benefit to adding client-side validation other than I can then detect, on the server side, if the incoming data has been tampered with. If I have client and server-side validation and I receive input that does not validate on the server, the application is under attack. Time to take a defensive action like logging the attack (HTML Entity encoded) and log the user out.

Speaking of logging and logging the user out, was the user ever authenticated in the first place? Did we log that event? Are the logs being monitored? Why was an unauthenticated user given access to a critical asset like the database? There are so many relatively simple mechanisms that would have prevented this attack it makes me want to discuss security (or its complete lack) in the software development life cycle (SDLC). OK, that's another topic for another day.

If you don't completely understand Bruce's comments and guidance, I recommend you find someone on your staff who does and let them see this stat. Seems to me like Night Dragon should have never happened ... we made it far too easy for the attackers to get in and get whatever they wanted. My hopes are that headline news like this, and Stuxnet, Wikileaks and Aurora before it, energize utilities to upend the status quo and reconsider their approaches to cyber security.

Not all Smart Grid FUD is Created Equal

Depends were it comes from. In this case, I'd probably give the Center for Strategic and International Studies (CSIS) and McAfee the benefit of the doubt, pardon the pun. Even if there's the slightest grain of truth in this statement, it is cause for concern for our side:

Because of tight government controls, China's own grid was ranked in the survey as the best protected from cyber attack. A strict regime of compulsory government inspections compares to a third of British critical infrastructure providers who said their network had never been audited by authorities.

Here's the article in Britain's Telegraph, the CSIS link, and the recent McAfee report at the heart of all this.

Remember, even if this info makes you worried, the right thing to do with that anxiety is to channel it into positive action that can enhance the protection of our grid systems through improvements to policy, planning, process, technology, etc. It's a common refrain on this blog but I repeat again, good work rarely gets done in the fetal position.

Image credit: Stephen Brace on Flickr.com

Smart Grid Security East and the Software Security Panel

Today I had the good fortune of being on a small panel, moderated by Matthew Carpenter, and with a representative of embedded software security provider Green Hills Software. We focused on grappling with how utilities and their suppliers are confronting application layer vulnerabilities not just in key systems, but across their entire application portfolios. Here's a summary of what I think are some of the interesting facts and other points we touched on:

• Application (or software) security is one of the newest (i.e., least mature) security sub-domains in every sector, which means utilities are not substantially further behind in this domain than some of their similarly sized, non-electric utility peers
• Large and very large utilities can have anywhere from several hundred to several thousand applications ... that they know of and track. A somewhat unsettling percentage of utilities don't know how many apps they really have. It's an often neglected form of asset management and some are working hard to figure this out. And some aren't.
• These same utilities often have one-to-two hundred developers in their internal development teams, most who have not yet been introduced to secure development principles, and with SDLC's that fail to leverage current tools that can really help
• Many utilities haven't yet formulated an application security policy, meaning, among other things: they haven't yet determined which types of software vulnerabilities add so much potential risk that they simply aren't allowed to exist in operational systems. Again, some are moving out with security policies that drive helpful behaviors in this area, but the majority (IMHO) aren't in motion yet
• I was asked what my Big Blue company is doing to help in the app sec area, and responded that we're working on three levels: (1) providing app sec training, consulting, services and tools to utilities, (2) bringing the same to vendors who supply software and software-intensive system to utilities, and (3) adding secure development processes to the SDLCs of the products we market to utilities, including those that comprise the Solutions Architecture for Energy (SAFE) framework

One point I meant to mention but didn't is that in the spirit of walk-then-run, before trying to develop policies and procedures to harden the entire application portfolio, many of the utilities we've worked with to date start at the project level with AMI and / or Customer Portal implementations. With AMI, we've seen utilities run application security tests on both the internally developed as well as vendor supplied software with good results. So good, in fact, that some of the related meter vendors, seeing the results, have procured our tools for their own internal use in their SDLCs, which again benefits the utilities when they buy these new, more secure products. And ditto for customer portal projects.

As this was a Powerpoint-free zone by design, in today's session we were just guys talking. But I've been building a short slide deck called "Securing Your Smart Grid Customer Portal" and plan to make it available, via the blog, to attendees shortly after the conference concludes. I think (and hope) you will find it helpful.

Smart Grid Security East Going Great, but Where are the SCADA/ICS Companies?

For folks who had the privilege of attending both the first conference in San Jose and this second one in Knoxville, there are several things that jump out at you now that we're more than halfway through:

• Interest is up ... My guess is that there are 2 to 3 or maybe 4 times more attendees overall, and that a much higher percentage are utilities personnel. Also, the conference and exhibit area feels more robust, probably because there are many more sponsors and partner orgs involved
• AMI/meter vendors are getting better and better on security. I was especially moved by Edo Dubrawky's talk on how very thorough he and his team are on software security issues at every stage of the development lifecycle. Definitely seems like solid progress
• Still, after attending Travis Goodspeed's "Embedded Systems Vulnerabilities from the Bottom Up" session I don't think I'll ever trust any electronic device ever again (and that's going to make this job tough). You should see what he can do with toys, toasters, garage door openers and more. All the meter guys (and the rest of us) were paying close attention. So progress is happening, but determined super geniuses still can show we have a long way to go in many departments

But my main issue is that while there's more coverage of Operational Technology (OT) SCADA and ICS security issues, to me it feels like we're still not doing nearly enough. Part of that is that the conference remains skewed heavily towards IT vendors and attendees coming from IT backgrounds. While some of the boutiques who provide OT security services are present, the big OT players should be here telling us how they're responding to Stuxnet's wake up call in their current installed base as well as in their future designs. So, to that end ...

Dear Siemens, ABB and the rest: how about you attend next time and help make this the more meaningful, balanced and productive conference I believe the organizers intend it to be? Apart from the fact that we still haven't figured out, as an industry and a community, how to demonstrate progress to our stakeholders (i.e. measurement/metrics), inadequate consideration of pressing OT security matters is the biggest elephant in the room. An electric-sector security pachyderm we're going to have to deal with one way or another ... and soon.

Photo credit: Namibnat/Vernon Swanepoel on Flickr.com

The Near-Inside Cyber Threat to Utilities

DOE's cyber lead Bill Hunteman  just revealed a security guard once told him "I'm your biggest threat" at 2 am one morning. Guard noted he had keys to every room in that utility and was taking cyber security classes in between shifts. Said he could be inside the network attacking systems all night and nobody would know it.

Food for thought at Smart Grid Security East.

Smart Grid Security East - Underway on Day One with a NISTIR 7628 Progress Report

My but how this conference has grown since its preprocessor in San Jose last year. Hundreds of folks in the hall this morning to hear Erich Gunther's welcome message, and now we've got these folks on stage talking 7628:

• Bill Hunteman, DOE
• Annabelle Lee, EPRI
• Daniel Thanos, GE
• Sandy Bacik, Enernex
• Mike Coop, ThinkSmartGrid (moderator)

Annabelle mentioned she likes Daniel's phrase - thinking about securing the grid from "toasters to turbines."  I'm paraphrasing here, but Daniel, hesitant to put all our security eggs in the NISTIR 7628 (or any other regulatory) basket, got the following across:

Security is a very dynamic space. Regulation can actually degrade security.  It freezes our approaches to a moment in time, while threats continue to change so quickly. Rather we should seek to help folks think better so they can adapt to threats as they evolve.

Then Bill said (my paraphrase again):

I challenge each of you coming to collaborate - let's see if we can reach an agreement, as a community, on what it means to protect the grid. Everyone back in DC still doesn't have a common definition on what this means, and that's really hampering progress.

Someone then asked a question on how we are measuring (and therefore demonstrating) progress to leadership in Washington and elsewhere, while noting that the previous point on not having a common definition to work from is a factor. The answer to that wasn't that completely clear, and my bet is it's likely the question on measurement will be asked again before this conference is through.

To be continued ...