It's Hard for Utilities to Improve Security when Their Business Models are Increasingly Insecure

This one's not about security, unless you consider the well-being of the utilities who own and operate most of the grid to be security related.  In which case this post is completely about security!

Greentech Media (GTM) has just written a short piece highlighting some of the take-aways of a new Edison Electric Institute (EEI) report called "Can the Utility Industry Survive the Energy Transition?" and I'd say both the GTM article and the full EEI report are well worth your attention.

Conference Alert: AGRION Energy & Sustainability

On Feb 19, one of the year's best energy and sustainability conferences will be kicking off in NYC. It's organized  by a great org I've become familiar with recently: AGRION, a global business network for energy, cleantech and corporate sustainability.

On the second day, following a morning keynote by PSE&G CEO Ralph Izzo, I'll be moderating a panel of experts on the topic "Smart Grid Market: Scope and Scale":

• Kevin Genieser, Managing Director & Head of Clean Energy & Renewables, Morgan Stanley
• Joe Callis, Sr. Applied Solutions Engineer, PJM Interconnection
• David Groarke, Smart Grid Senior Analyst, Greentech Media

To be sure, I'll work in an appropriate amount of security substance. After all, you can't deploy a Smart Grid that's easy to disrupt, right?

You can see the full agenda, list of speakers and venue details HERE. Hope some SGSB readers can make it.

ENISA Again: 3rd Time's the Charm re European Grid and Smart Grid Security Policy

8/29 Update:

You still have a few days to register and get your plane or train tickets to Amsterdam. In one fell swoop, the existence of this 10/15 workshop, in itself, fully refutes charges of lack of US-European cooperation, as well as claims that control system security is ignored. Go HERE to learn more and register.

---------------------------------------

While of monologues many great political speech or play are constructed, it's through dialogue we reach understanding and consensus. Wait, who said that?

This blog first posted on the European Network and Information Security Agency (ENISA) and its recent recommendations for EU energy sector security earlier this month.

Perhaps Better Fettered: 2nd Thoughts on ENISA's Cybersecurity Report from this Side of the Pond

Had a number of reader responses to this week's post on the European information security organization's proclamation of intent and recommendations for the electric sector and Smart Grid. 

My post welcomed the attention to the issue by the EU, but expressed, hopefully in a mainly professional way, that this feels, to invoke a common American idiom, a day late and a dollar short.

Here are two additional observations I got:

1. One US respondent says "It contains no call for cooperation with US-CERT, FERC or equivalent body on problems that are clearly of interest to both sides. Compare with various DHS initiatives (such as DHS ICSJWG) which have included foreign participants."

Concur. References to SANS, NIST and DHS in the bibliography notwithstanding, it does appear that explicit calls for trans Atlantic, interagency cooperation are missing, and that this should be rectified in a next version.

2. Another true blue American notes "ENISA reports do not adequately address control systems."

While the bibliography is littered with entries for SCADA and Control Systems-related texts, it doesn't seem like much of that research made it into the final document. Still, while most of the 10 recommendations involve getting ready to get ready to do something, and control system security seems to be largely glossed over, there is, in requirement 6, language that might point to operational systems at some point:

Recommendation 6. Both the EC and the MS competent authorities should promote the development of security certification schemes for components, products and organisational security.

So I'll leave it at that for now. Would welcome an ENISA response. I always try to not be too hard on 1.0 documents because there's always the chance, if not the likelihood, that we'll see them improve in subsequent versions.

I know it doesn't want to be a fetterer, but my sense is that Europe will come to see the wisdom of getting a bit more explicit and comprehensive in these matters.  I know from experience that some of its utilities are looking for more guidance. OK? Back to the Olympics!

Getting Smart at GridWise Global Forum this Week

This just in from the SGSB social media desk - I'll be at the Reagan building in DC starting tomorrow armed with MacBook Air, Twitter and Blogger to both speak at and cover this year's GridWise Global Forum (agenda HERE).

Will be paying particular attention to the opening keynote moderated by IBM Energy & Utilities sector GM Guido Bartels with DOE Secretary Steven Chu and Uzi Landau, who runs Israel's Ministry of National Infrastructures (Tues at 12:45 pm ET), and the following panels:

• "Guarding the Grid: Smart Grid and Grid Vulnerability" (Tues at 4:30 pm)
• "The Technology Horizon: Future Trends and Potential Disruptions" (Wed at 8:30 am)
• "Smart Grid Data: Insights, Privacy, or Both" (Wed at 10:30 am)
• "Smart Grid and the Regulatory Landscape: Evolution or Revolution" (Wed at 1:30 am)

Two of these sessions will be broadcast live (and free) by our friends at Greentech Media. Follow THIS LINK to tune in at the appointed times to "Guarding the Grid" and "Smart Grid Data."

BTW: will using the #IBM@GridWise hashtag for denizens of the Twitterverse.

Conference Alert: Wise up at GridWise Global Forum

This is a big one, and though it's not security focused, security topics will certainly be in the air, and yours truly will be on a privacy panel on Wednesday.

From what I heard of last year's event, this is one of the most high powered Smart Grid conferences on the planet. Note the presence of some senior and very senior international leadership from government and multiple industrial sectors (not just energy).

Details:

• What: GridWise Global Forum
• Where: Washington DC, Ronald Reagan Federal Building
• When: 8-10 November 2011

For more info on speakers, agenda and to register, click HERE

The Normally Strong Grid's Self Inflicted Wounds

So only a few days ago you saw a post here about grid lessons from Hurricane Irene. Now we're back with another major grid event and I'm not sure what to call it other than the recent Arizona, San Diego and Mexico outage ... SanMexiZona outage perhaps?

Investigations are still being conducted, but what do we know so far? Well, a transmission maintenance issue impacted a substation in Arizona, and then:

• Cascading failure reached into California and Mexico, knocking power out to millions
• And caused 2 nuclear facilities to shut down
• Navy and Marine bases turn to back-up diesel generators and kept non-essential personnel home
• And many other types of trouble you'd expect from a black out in a large US city ensued, driving cost estimates into the hundreds of millions.

It's weird. In some ways the grid is a beast, capable of absorbing the worst insults and continuing operations largely unaffected. It virtually scoffs at earthquakes, raging fires, hurricanes, tornadoes ... and across the Pacific, even Godzilla stomping out of Tokyo Bay once in a while. Sure, some outages occur in the areas where equipment is destroyed. But the grid is usually a master of defense and containment.

But then a little thing happens during routine maintenance and a big chunk of the grid unexpectedly swoons. Amory Lovins and others on the 2008 DoD Science Board (DSB) task force on Energy identified the US grid as brittle and a threat to CONUS military readiness. Here's Lovins in 2010:

The US electrical grid ... is very capital-intensive, complex, technologically unforgiving, usually reliable, but inherently brittle. It is responsible for 98–99 percent of U.S. power failures, and occasionally blacking out large areas within seconds—because the grid requires exact synchrony across subcontinental areas … and can be interrupted by a lightning bolt, rifle bullet, malicious computer program, untrimmed branch, or errant squirrel.

Seems like some of the worst behaviors we see in the grid are avoidable. In addition to the many other benefits we often describe to regulators and general public with the Smart Grid build out, improvements to reliability have got to be high on the list, if not #1.

BTW - Try Googling "Errant Squirrel" - it's simply amazing how active (and errant) these critters have been!

Image credit: KUSI News San Diego

A Couple of Closing Thoughts on Hurricane Irene

Hurricane Irene fully cleared my city (Boston) last week, we've had nice weather since, and everyone (or almost everyone) in Massachusetts has their power back at the time of this writing. Folks in some other states aren't quite so lucky.

But before we file away the memory and move on to the next storm or cyber incident, check out this Irene-related online exchange between a residential customer and a utility executive doing his best to keep his customers as informed as possible:

Q: Why am I getting calls to see if my power has been restored when in fact it has not been? I have a 4 year old and 1 year old and you can imagine what it is like being without power. 

A: One of the reasons we perform call backs is because crews have made repairs in the neighborhood and surrounding areas, and we want to ensure that each house has been restored. Without requesting a call back when you report an outage, we wouldn't know the service to your house is still out. Please make sure to report all outages to 1-877-xxx-yyyy.

Sounds like a region ripe and ready for its residential Smart Meter deployments, doesn't it? I'd say it's well worth the extra time and effort cyber professionals need to develop a secure Smart Grid to relegate conversations like this to history.

And the image of the totally chewed up poles (from Nag's Head, North Carolina) really caught my eye. Aren't the poles supposed to be holding up the lines ... and not the other way around? As immigrants to the electric sector quickly learn: cyber risks are one thing; Mother Nature is something else entirely.

Photo credit: Nicholas Kamm of AFP

From the Left Coast comes Big News on Smart Meter Data Privacy Regs

No time to pontificate on this now, but wanted to make sure you saw the news. CPUC's formerly proposed decision has just become a decision. One, the implications of which, could ripple across the US and impact future Smart Meter and Smart Grid deployments. See the Jesse Berst quick take on it HERE.

2nd Smart Grid Security TwitterStorm Spotted

Social media storm chasers have identified this Wednesday afternoon (330 pm ET to be precise) as the likely time the next security related Smart Grid twitter discussion is likely to hit. The previous one, that I was involved in anyway, was last fall, and it was a pretty interesting and educational affair. See announcement HERE.

Subject this time will be the deployment of security controls at a US utility for two primary objectives:

1. To protect itself from potential attacks coming from outside, particularly the Smart Meters and AMI network it's been standing up for customers recently
2. To protect Smart Meter-enabled residential and commercial customers from potential attacks (or accidental, incorrect instructions) originating inside the utility or its systems

Please note, this will be an IBM-centric discussion so I'll be speaking/tweeting from the perspective of my day job using the Twitter ID: @IBMSmartrEnergy and to follow or participate in the conversation folks should use the Twitter hashtag: #IBMSG.

Looking forward to this event: please join in if your schedule allows. BTW I'll be using the TweetDeck app for this event and recommend you give it a try if you haven't already.

How much Smart Grid has been deployed so far?

Not all questions can be answered on the fly.  In fact, not all questions can be answered, period:

• What, for instance, is black matter?
• What is my cat thinking?
• Is there intelligent life on Earth?
• How does Tim Thomas stop so many shots?

Heck, 99% of us can't even agree on what the Smart Grid is, let alone have a clue about when it's going to be here. Nevertheless, after being asked the question in the title above, I pledged to do some digging and post a response here on the SGSB as soon as I thought I had something. This came at the tail end of the recent Virtual Energy Forum (VEF) session called: "Lessons from the Smart Grid Security No FUD Zone." You can try getting to it by clicking HERE, but good luck.

Now without further delay, procrastination or obfuscation, here we go. If you look at this SmartGridNews write-up of a recent IDC Smart Grid market report, the picture may begin to come into view for you. Sometimes you can infer the past by getting a glimpse of the future (a nifty reversal of common wisdom that you can better imagine the future by studying the past).

Around the world, Smart Meters are being deployed in ever increasing rates. Home energy management systems are expected to go through the roof (so to speak). And grid automation is coming on strong. So, question: how much is deployed today vs. what will be ultimately deployed in 5, 10, or 20 years?

Answer: Some of it, not all of it. We're still in the early days. Given the pace of technology change, probably the very early days. It's a good question to keep asking, though, and for some of us to try to keep answering. But I reckon it ain't ever going to be fully answered, because the Smart Grid (if it's still called that in the future) won't ever be fully here.

Photo credit: Radar Communication on Flickr.com

Smart Grid: Good or Bad Idea?

With a hat tip to Ollie Fritz of OSD, here's the fundamental question we security folk caught up in grid modernization activities can't help but ponder:

Are we helping or hurting our nation's overall security posture?

If you persist and continue on to this recent post on Aviation Week's Ares blog, you'll find more smart folks in high places questioning the wisdom of building this thing. That's something you'll sometimes find me doing (though with neither brilliance nor from a lofty perch) over cocktails in semi-private settings, but never directly under the hungry gaze of the press.

You see, whether we think it's net-net a good idea at any one particular point in time, in any one particular geography, it's a moot (some say mute) point to question the value of the Smart Grid. The fact is, notwithstanding Smart Meter resistance movements in California, Maine and Ohio (thanks Andres), we're right now in the construction phase at varying degrees of speed all around the world. And the Smart Grid being built is much much more than those headline grabbing Smart Meters.

The attendant security challenges it brings are monumental. The risks, we hear, are growing daily. But overall, it's all the more worth pondering and tackling because of the central role awaiting a modernized energy grid in our future.

So question though we must (some more than others), the momentum towards a Smarter Grid is inescapable. As Tom Paine said, "Lead, follow, or get out of the way." I'm with him.

Image credit: Stefan Baudy on Flickr.com

Combating Smart Grid Vulnerabilities ... and Ourselves

In the previous post I attempted to communicate the urgent necessity of setting some performance metrics for ourselves, with the objective of demonstrating to the senior decision makers who sponsor our activities that what we are doing is bearing fruit.

That the sum total of all the money spent on Smart Grid cyber security products and services, plus the monetary and human resources dedicated to the task of formulating solid interoperability and security standards is producing demonstrably more secure utilities and a demonstrably more secure and increasingly smart grid.

Well, the Journal of Energy Security just published an article called "Combating Smart Grid Vulnerabilities" in which my senior colleague, Grid Wise Alliance Chairman emeritus and current Chair of the Global Smart Grid Federation, Guido Bartels makes a case that we seem to be making reasonable progress ... that we're successfully grappling with what we think we know about the security weaknesses in this system under construction. And I can only agree with him.

But he also acknowledges that it's really hard to say for sure. And backs that with the recently published findings of the GAO and the DOE's IG office. A section of the article called "Don't get too comfortable" states:

The [IG report] issued its report on this matter ... in which it found FERC cyber security standards (as implemented by NERC) and the overall approach for regulating the national grid quite lacking, saying: "… even if the standards had been implemented properly, they 'were not adequate to ensure that systems-related risks to the nation’s power grid were mitigated or addressed in a timely manner."

My response to this is: how would the DOE IG, or anyone else for that matter, especially those who aren't working energy and cyber security 24/7 know if and when implemented standards and controls were adequate? We haven't defined adequate and we measure almost nothing because we've told ourselves two things:

1. It's too hard to measure cyber security, especially in the energy sector, and,
2. We can't talk about anything that might be helpful because the info is too sensitive

I agree with Bartels that we are making progress. But how we convince others of that is another matter. There are plenty of MBA's out there and enough Deming disciples to know that we're fooling ourselves if we think that progress is self evident ... that it's obvious to all observers that activity equals efficacy.

Let's admit the emperor is stark naked, get him some decent garb, and build an increasingly secure Smart Grid, the security level of which can be communicated to ordinary folks ... including non-technical senior executives and congressmen.

Smart Grid Security East and the Software Security Panel

Today I had the good fortune of being on a small panel, moderated by Matthew Carpenter, and with a representative of embedded software security provider Green Hills Software. We focused on grappling with how utilities and their suppliers are confronting application layer vulnerabilities not just in key systems, but across their entire application portfolios. Here's a summary of what I think are some of the interesting facts and other points we touched on:

• Application (or software) security is one of the newest (i.e., least mature) security sub-domains in every sector, which means utilities are not substantially further behind in this domain than some of their similarly sized, non-electric utility peers
• Large and very large utilities can have anywhere from several hundred to several thousand applications ... that they know of and track. A somewhat unsettling percentage of utilities don't know how many apps they really have. It's an often neglected form of asset management and some are working hard to figure this out. And some aren't.
• These same utilities often have one-to-two hundred developers in their internal development teams, most who have not yet been introduced to secure development principles, and with SDLC's that fail to leverage current tools that can really help
• Many utilities haven't yet formulated an application security policy, meaning, among other things: they haven't yet determined which types of software vulnerabilities add so much potential risk that they simply aren't allowed to exist in operational systems. Again, some are moving out with security policies that drive helpful behaviors in this area, but the majority (IMHO) aren't in motion yet
• I was asked what my Big Blue company is doing to help in the app sec area, and responded that we're working on three levels: (1) providing app sec training, consulting, services and tools to utilities, (2) bringing the same to vendors who supply software and software-intensive system to utilities, and (3) adding secure development processes to the SDLCs of the products we market to utilities, including those that comprise the Solutions Architecture for Energy (SAFE) framework

One point I meant to mention but didn't is that in the spirit of walk-then-run, before trying to develop policies and procedures to harden the entire application portfolio, many of the utilities we've worked with to date start at the project level with AMI and / or Customer Portal implementations. With AMI, we've seen utilities run application security tests on both the internally developed as well as vendor supplied software with good results. So good, in fact, that some of the related meter vendors, seeing the results, have procured our tools for their own internal use in their SDLCs, which again benefits the utilities when they buy these new, more secure products. And ditto for customer portal projects.

As this was a Powerpoint-free zone by design, in today's session we were just guys talking. But I've been building a short slide deck called "Securing Your Smart Grid Customer Portal" and plan to make it available, via the blog, to attendees shortly after the conference concludes. I think (and hope) you will find it helpful.

Day Zero (Pre-Conference Work Shops) News from Smart Grid Security East

The conference hasn't started yet, but it's been a great day here in Knoxville nevertheless, as 3 concurrent workshops are keeping all the early birds busy:

• AMI Security
• NERC CIPs
• Control System Security

While most attendees are getting deep immersion in these subjects from 10 am - 5 pm today, with my short attention span and desire to get the broadest impression, I've jumped from session to session to session. In addition to getting some valuable updates to what's going on in these three domains, I'm getting to put faces to names of people only met online before.

Tomorrow the conference kicks off for real with opening remarks from Enernex's Erich Gunther and a NIST 7628 update from Marianne Swanson and Daniel Thanos.

FYI: Have been doing a little tweeting using the #smartgridsecurityeast tag and plan to continue tomorrow. HERE's the official site for the conference. Stay tuned for more ...

2011 Smart Grid Security Summer School Announced

Summer school this year, so maybe there'll be an Outward Bound Smart Grid adventure camp in 2012? Here are the details:

With support from DOE and DHS, we are proud to present the "Cyber Security for Smart Energy Systems" Summer School organized by the Trustworthy Cyber Infrastructure for the Power Grid (TCIPG) Center. The summer school will be held in the Q Center, St. Charles, Illinois, which is less than an hour away from Chicago's O'Hare Airport, June 13-17. 

An overview of the objectives and topics for the summer school is provided in the attached document. Details on registration, the program, and travel logistics will be available soon HERE.

You may contact Rakesh Bobba (rbobba@illinois.edu) or Scott Pickard (spickard@illinois.edu) if you have questions, comments, or suggestions. We very much hope that you can join us, and we look forward to an exciting summer school.

Smart Grid Security East: Final Reminder ... and an Offer

Here are the details for logisticians:

• Hotel: The Crowne Plaza Knoxville hotel is the site of the Conference, and it's offering discounted room rates of $99 for attendees to the conference. (Remember to specify the “Smart Grid” block or the code “IWM”) 
• Dates: Feb 28 - Training workshops, Mar 1 and 2 - Conference
• Click HERE for conference web site and HERE for $300 off the full price including workshops

And since I think this is a good deal, and nothing of value should be given away for free, I'm going to ask you a question, and the first 5 who answer it correctly can attend Smart Grid Security East for free. Ready? Here you go:

Yesterday, on Valentines evening, an IBM supercomputer named Watson and its two human competitors on Jeopardy were given the following clue by Alex Trebek in the category "Potent Potables Olympic Oddities": "It was the anatomical oddity of US gymnast George Eyser who won a gold medal on the parallel bars in 1904."

What did Watson say? Email your answer to andybochman at gmail dot com and I'll let you know if you were correct ... and fast enough.

FERC Finalizes Agenda for Tech Conference on Smart Grid Interoperability Standards

As noted earlier this week on this blog, FERC has invited its commissioners to an immersive afternoon on Smart Grid interoperability and security standards development, past, present, and future. Now FERC has finalized its agenda and named the panelists who'll be attending.

Following an introduction by NIST's Smart Grid Interoperability Coordinator, George Arnold, will be 2 90-minute sessions:

1. The Smart Grid Interoperability Standards Process for Reviewing and Selecting the First Five Families of Standards, and
2. The Smart Grid Interoperability Standards Development and Identification Process Going Forward

Key logistical details are:

• It's open to the public, so you can go if you want to, and if there's room, attend this event in person at FERC HQ in DC
• If you can't make it or don't want to, a free live webcast will be available here
• Lastly, they indicate that the conference will be archived for 3 months

Here's the latest dispatch from FERC with all the info.

Photo credit: hydroreform on Flickr.com

Teaching the Old Grid New Tricks ...

... will require students versed in the art and science of engineering, including (but not limited to) electrical engineering. We used to say that in the future we'd need these folks. Well, with the recent passing of 2010, the future is beginning to look more and more like the present.

A present in which ...

A great deal depends on whether power companies can find and attract a sufficient number of engineers capable of designing, managing and maintaining the new systems the smart grid demands. And that’s by no means certain. The Center for Energy Workforce Development estimates that by 2015, 51 percent of the power-engineering workforce will need to be replaced because of retirement or attrition. And that’s just to maintain current levels. To drag our aging grid into the 21st century will require power engineers trained in the most sophisticated communications and control concepts.

Seems like the old immovable object about to be whacked by an irresistible force. In a tough job market, this much need can't and won't go unfulfilled for long.

This article quotes a manager at AEP as saying these vacant engineering roles will be filled by new personnel from one of three sources: re-trained internal folks, university programs and vendors. University investment in new teachers and courses has been constrained to say the least. Though the last word may belong to the DOE, which just slapped down a cool $100 million on the counter for Smart Grid training programs.

At the bottom of the article you may notice one reader asks "Just engineers?" The answer, of course, is of course not. Increasingly, folks with training in business and economics are called for as the old business models are poised for a most thorough revision.

And as for cyber security pro's to watch over the systems designed and built by the new crop of inspired engineers and business folks, they're going to likely come from vendors for a while longer, until organizations like SANS and the new NBISE can get a bunch more out the door with the requisite energy sector chops ... like a firm grounding in SCADA/ICS, for instance.

Photo credit: USAFA (my alma mater) graduation by Beverly & Pack on Flickr.com

Looking Back and Looking Forward on Smart Grid Cyber Security at GridWise 2010

As Mark Twain (or Hemingway, Cicero, Voltaire, Blaise Pascal or George Bernard Shaw) once said "If I had more time, I would have made it shorter." That's true of the 25-min audio that accompanies  - feel free to fast forward. But believe you'll find the content here interesting, and depending on your line of work vis a vis the Smart Grid, maybe even helpful.

GridWise 2010 Cyber Security Update

View more webinars from Andy Bochman.

There were several good questions and comments during the Q&A session that followed, but the one I appreciated most was that this wasn't the typical doom and gloom message that typifies many energy sector security presentations.  I count that as good news as that is a design objective. As we've said before, no good work gets done by people in the fetal position. And we've got plenty of work to do.

For more from GridWise here's a LINK to the organization's cyber security resources page. These are great people moving mountains as they advocate for Smart Grid progress. Highly recommend you give them your support and/or get involved if you haven't already.