NERC's Cyber Education Role

Online tech mag Ars Technica recently wrote up the results of two reports on US energy infrastructure, one from the North American Electrical Reliability Corporation (NERC), and the other from a small cyber security company named LogLogic. The sum, for me, was a reminder of how far we have to go on enterprise Smart Grid cyber protection policy and implementation, and how little time we have to get there.

Referenced within the Ars article, is NERC Chief Security Officer (CSO) Michael Assante's April 2009 memo to electrical industry players. His calls for increased attention to cyber risks are still at the basic education level, as many of the targets of his guidance are from operations, and are still relatively new to the IT and cyber security domains:

... as we consider cyber security, a host of new considerations arise. Rather than considering the unexpected failure of a digital protection and control device within a substation, for example, system planners and operators will need to consider the potential for the simultaneous manipulation of all devices in the substation or, worse yet, across multiple substations. I have intentionally used the word “manipulate” here, as it is very important to consider the misuse, not just loss or denial, of a cyber asset and the resulting consequences, to accurately identify CAs under this new “cyber security” paradigm.

Excellent here that Assante keys on manipulation, as cyber attackers oftentimes achieve greater effects through means that at first appear quite subtle ... or aren't visible at all. At some point he's going to have to point out that a precursor to manipulation or outright attack is monitoring, often done by placing apparently benign software agents on target systems to collect data and await further instructions.

Assante also attempts to update industry thinking on the current grid's design that can usually handle large single points of failure. Cyber threats are often targeted less like sniper rifles and more like shotguns:

One of the more significant elements of a cyber threat, contributing to the uniqueness of cyber risk, is the cross-cutting and horizontal nature of networked technology that provides the means for an intelligent cyber attacker to impact multiple assets at once, and from a distance. The majority of reliability risks that challenge the bulk power system today result in probabilistic failures that can be studied and accounted for in planning and operating assumptions. For cyber security, we must recognize the potential for simultaneous loss of assets and common modal failure in scale in identifying what needs to be protected. This is why protection planning requires additional, new thinking on top of sound operating and planning analysis.

Thinking? Excellent. New thinking ... even better !!!

Are We Ready for the -Next- Generation of Security Concerns for the Smart Grid?

Over the past several months, interest in security and the Smart Grid has been growing, and much of the focus has been on trying to raise some baseline interest in ensuring that the millions of meters and interconnection points are specified, deployed, and managed with some sense of urgency around their security.

In re-reading David Leeds' excellent piece on next wave venture investing in the Smart Grid, "Smart Grid 2.0: ‘The Soft Grid’" , I started to look up from our daily security quest to think about how much more complicated all of this is about to get. The article describes the next wave of investment in the Smart Grid, and how, with many of the infrastructure providers well-funded and moving forward, that the investment community is going to start to look for software firms that will build applications to capitalize on all of the new Smart Grid functionality.

I think that in some ways, there is some good news in this piece, mainly that IT-experienced people are beginning to become interested in the Utility space. Jeff St. John had written recently about the movement of Utility execs to Smart Grid startups, but this is more about bringing IT leaders into the Utility market. This is an evolution that Andy and I have been looking for and writing about for a while. Experienced IT and network security people have got to be brought to bear on the challenges of the Smart Grid, but as yet, according to our research, there have not been many folks who have made the jump. In David Leeds' comments, however, he notes that,

... this industry is now an attractive place to hang one's hat, and as such we anticipate that the electric power sector will be inundated with a wave of fresh talent in the next five years.

On the downside however, is the greenfield description of much of the new software that may soon infest the largely untested security byways of the Smart Grid. If Leeds is correct, and venture investing will soon begin to drive a wave of new functionality providers in the software market riding the Smart Grid, then the real impacts of any underlying insecurity within the Smart Grid infrastructure are shortly going to multiply manyfold.

Internet Co's will Embrace Smart Grid, but will Energy Co's Embrace Internet?

This piece in MIT's Technology Review describes a few of the economic incentives for Internet companies like Akamai to investigate and invest in energy market-aware hardware, software and networking gear.

The ability to throttle back energy consumption could have another benefit for massive Internet companies, the researchers say. If an energy company were struggling to meet demand, it could negotiate for computation to be moved elsewhere; the researchers say that the market mechanisms needed to make this possible are already in place.

Expect much more of this in the near future from companies well versed in rapid adaptation via flexible, well managed IT operations. But what to expect of utilities and other energy ecosystem players? One of the patterns that's emerged from conversations we've had with industry is that most utilities have succeeded until now by purposefully avoiding aggressive IT innovation. The logic being that energy generation and delivery need to be 99.99% reliable, whereas IT and the Internet have a not undeserved aura of instability (see "blue screen of death" and the "three fingered salute" as well as recent pervasive troubles in Twitter-dom.

How a history and culture of IT skepticism will affect future energy co. adaptation to Smart Grid technologies remains to be seen ... but we'll be watching.

Empire State: Building with Smart Grid Grants, Hold the Security

There was a release today from Governor Patterson's office in Albany about the creation of a new Smart Grid Consortium in New York. Feel free to read the release, as it exhibits strong exothermic properties. Within it, however, is a reference to the new NY State Smart Grid Consortium Smart Grid Vision and and Technical Plan Report (Draft).

I would encourage you to use it as a resource, there are some nice charts, and it is truly a tutorial on merging real energy thinking with real politicking around grant dollars. Jobs, dollars, energy, etc., like political Prego, "It's in there."

As all of us in the Smart Grid community know, there is a double-edged sword in the hands of government these days, and it is called the Smart Grid Investment Grant Program, and we have written of it here before. A real boon, for all of the incentive it provides, and a recipe for long-term disaster as it drives substantial investment long before the community has matured in its understanding of need and security.

Most unfortunate in this report is its complete lack of focus on security, except for sprinkling the word into the document, hoping perhaps to ward off any real requirement of substantiated activity. I urge the NYS Consortium to remember that there are few areas of the country where blackouts have caused such chaos, and where potential blackouts would have such a devastating effect on the enormous financial, media, and technical communities that populate it. This graph, from page 44, says it all:


Someone in a New York utility has got to be doing something...

Lockheed Throws its Hat in the Smart Grid Ring

Looking for business ... and perhaps a handout. Like many other large co's who have recently made their intentions known re: Smart Grid stimulus monies, Lockheed brings modest energy credentials to the table, so partnering with an expert is the way to go. More importantly, though, from this blog's point of view, is Lockheed's comparatively deep background in cyber security. Let's see if they choose to play a leadership role or not.

Here's the announcement.

5 Years and How Many Devices?

We were working yesterday on some background for our continuing research on Smart Grid device security, and I found an absolutely prescient piece by an associate professor at CMU, in the Department of Electrical and Computer Engineering, named Philip Koopman. The article was carried in July, 2004 by Embedded Computing Magazine. You can find it here.

I'd recommend you give it a read, because it provides some non-Apocalyptic views of the dangers of insufficiently secured micro-controlling devices, just the kind that we have been worrying about as we watch Smart Grid pilots, roll-outs, and meter buys over the past year. The Smart Grid wasn't yet in vogue, and the interactive power management that empowers it was not evident, but Professor Koopman does an excellent job of painting some non-tragic but disturbing scenarios in an even less connected energy market.

We at the Smart Grid Security Blog continue to plead, on street corners, conferences, and on Capitol Hill, that people take a closer look at their new interactive power infrastructure before we find ourselves in too deep. Would that we knew Professor Koopman in 2004, because he shines a light years in advance of our current road to risk:

Many embedded systems are created by small development teams or even lone engineers. Organizations that write only a few kilobytes of code per year usually can’t afford a security specialist and often don’t realize they need one. However, even seemingly trivial programs may need to provide some level of security assurance. Until standard development practice includes rigorous security analysis, developers may overlook even the solutions already available.

You are a man ahead of your time, Koopman.

Temporarily Mismanaging Demand Management in Atlanta

We've been having a heat wave up in Boston this week, which makes us more empathetic about this recent snafu in Atlanta. With Smart Grid-like configurations, technology can be a force multiplier to achieve great new capabilities ... as well as amplify the negative consequences of human error. In this case, it was a relatively short loss of household AC. But I'm sure you get the point.

And how did folks happen to part of this demand management program in the first place? They responded to the following very reasonable Duke Energy entreaty to save some dough:

Why Sign Up - Depending on which Power Manager option you choose, you will receive a one-time credit of $25 or $35 on your bill just for signing up. You will receive a credit on your electric bill whenever we use the Power Manager device to turn your air conditioning unit off and then automatically back on. You are helping to preserve the environment and keep electric costs low by reducing the demand for electricity and delaying the need to build additional power plants in our region.

How the Program Works - Duke Energy will install a free load management switch next to your air conditioner on the outside of your home. This radio-controlled device will cycle your air conditioner off and on when demand is especially high. Depending upon the option you choose, your air conditioner is cycled off and then back on approximately one time each half hour, for the length of the cycling event. Cycling events will not normally exceed a four to six hour time frame and will not occur on weekends or holidays (except in a system emergency). To help keep you comfortable, the indoor fan continues to run to circulate air throughout your home.

Hopefully we'll get to make and learn from lots of small, relatively benign mistakes before the consequences become much greater. And become much greater they surely will ...

The Smart Grid Needs an Identity

Actually, eventually, millions or billions of mobile device identities. Shidan Gouran helps us imagine the enormous scope of the beast we're coaxing into life, including some of its basic identity-related security challenges:

Grid authorization, authentication and accounting mechanisms, together with security infrastructure, would be needed to ensure a device’s proper identity, the network's integrity, and proper accountability. The accounting process would also have to include sophisticated and decentralized clearing-house services so consumers could charge their PEVs with roaming utilities. Of course all these requirements would extend to every mobile/portable electric device including laptops and other consumer electronics.

Read the whole piece here.

Brave or Foolish - San Diego Set to Deploy Millions of Smart Meters

While this earth2tech article focuses most of its attention on the wireless options and plans being considered by San Diego Gas & Electric, it's the number of smart meters they plan to install over the next 2 years that caught my eye:

1.4 million

Pilots need to get larger, there's no doubt about it. But that's a lot of money in devices and installation services fees. And a heck of a lot of dough down the drain if the technology choices are found less than prescient a few years down the road. Here's the whole piece.

FERC Sounds Smart Grid Alarm at GovEnergy 2009

Joe McClelland, Director of FERC's Office of Electric Reliability, said something that echoed through the otherwise upbeat annual conference focused on new energy technology and policy. Near the close of the conference, McClelland, a lifelong energy vet, said:

The Smart Grid scares the hell out of me. It's not the new capabilities; it's the amount of interconnectedness it brings.

As Jack and I have maintained, it's way too early in the evolution of the Smart Grid to think we have all the problems worked out. In fact, we only know a fraction about what this complex creation is ultimately going to look like. The best we can hope for short term is to be finding and asking the right questions.

But folks like McClelland know what has kept the grid relatively safe and reliable until now, and they sense instinctively that something very new and hard to plan for, let alone control, is coming on fast.

So, What About that Smart Grid Investment Grant Program?

Having mentioned the Smart Grid Investment Grant Program in a post over the weekend, I have gotten a couple of questions from people interested in the actual requirements. Again, you can find the document at www.fedconnect.net, search for DE-FOA-0000058. To save all of you the effort of fetching and searching, here is the relevant text, from Page 24, at the bottom, in the section, "5. Technical Approach to Interoperability and Cyber Security" (Emphasis added)

Submitted Project Plans are also required to include a section on the technical approach to cyber security. Cyber security should be addressed in every phase of the engineering lifecycle of the project, including design and procurement, installation and commissioning, and the ability to provide ongoing maintenance and support. Cyber security solutions should be comprehensive and capable of being extended or upgraded in response to changes to the threat or technological environment. The technical approach to cyber security should include:

A summary of the cyber security risks and how they will be mitigated at each stage of the lifecycle (focusing on vulnerabilities and impact).

A summary of the cyber security criteria utilized for vendor and device selection.

A summary of the relevant cyber security standards and/or best practices that will be followed.

A summary of how the project will support emerging smart grid cyber security standards.

DOE intends to work with those selected for award but may not make an award to an otherwise meritorious application if that applicant can not provide reasonable assurance that their cyber security will provide protection against broad based systemic failures in the electric grid in the event of a cyber security breach.

Hmmmm...Looks like time to find out how these requirements are going to be verified, and to find out exactly how much weight should should have. More to come...

Security and the Smart Grid Investment Act

From South Jersey to You: The Courier Post Online reports that PSE&G is applying for federal stimulus dollars as a source of funding for their grid upgrade to make it Smart Grid capable. As they look for approximately $76M ( 50% of the expected costs of the improvements ) in tax dollars, the utility is highlighting the project's effects on job creation and advancement of Smart Grid goals. They are applying for a grant through the Smart Grid Investment Grant Program, which has made $3.9B available for Smart Grid Technologies and Grid Infrastructure.

One interesting note within the Grant program ( which you can look for at FedConnect ) is located in Part IV, Section B.3.b. It is a requirement for a project plan within the application for grant to contain a:

technical approach that describes how the project will address interoperability and cyber-security;

On request of the SGSBlog, PSE&G Investor Relations is going to see if they can produce a public copy of that section of the application. More data as it becomes available.

White House Underpowered for Smart Grid Security?

With the recent departure of the talented Melissa Hathaway from the White House, the blurry view of security for the Smart Grid is unlikely to clear anytime soon. Following Ms. Hathaway's analysis of the nation's cyber weaknesses and infrastructure needs, it seemed likely that underlying requirements for additional protection and reinforcement would only be an act of Congress and a pen-swipe away. With her exit, leaders on the Hill and experts in the market are going to have to create their own momentum, and bring that awareness to 1600 Pennsyvania Avenue.

Stay tuned, and keep your eyes open to legislation like HR 2165, SR 946, and others. The stimulus money is going to continue to drive the supply side of SmartGrid energy, and tax-advantaged investments by homeowners and businesses are likely to drive some non-trivial rates of adoption. In the middle will be the infrastructure, and right now, it looks like no-one at the top is really minding the store.

2nd Gen Zigbee Hack Method Revealed at Black Hat 09

Zigbee is an enabling technology for energy management products in the home. Assuming security analyst Travis Goodspeed's work is solid, you don't have to be a security expert to get the gist of his conclusions re: a popular Zigbee device: :

This paper has shown that all Chipcon radios at the time of publication are vulnerable to key theft because of unprotected Data memory. Further, as all popular 8051 compilers place even constants in Data memory for performance reasons, it can be assumed that all products which were shipped prior to the authorship of this paper are vulnerable. Extracting a key is as simple as connecting a debugger, erasing the chip, then freely reading the contents of RAM. Further, as the competing radios from Ember offer even less security, the tamper resistance of wireless sensors should perhaps be considered forfeit by default.

Here's the paper he just presented in Vegas.

Cyber-Energy Security at USEA

I like this SAIC presentation from last month, in particular where it addresses which part of an energy org is the best one for addressing looming cyber-related security issues. The enterprise IT shop is recommended, though of course, many if not most energy co. IT operations are not prepared for the scope or complexity of this task as it relates to the Smart Grid. (more on this to follow).

It also does a nice job of characterizing cyber challenges that seems right for this moment, but also would have been on target well before the dawn of the Smart Grid:

• There is a growing focus on the impacts of risks and vulnerabilities to critical energy infrastructure
• System complexity is growing through expanding interconnectivity of systems
• Digital systems are proliferating, extending the electronic perimeter to new system components and participants
• Lots of legacy investments that need to be secured alongside newer, unproven technologies
• What level of investment and focus on securing infrastructure will be enough?

Black Hat Smart Grid Security Paper Contends Security-Baked-In Window Already Closed

This paper, presented at hacker conference Black Hat this week by security consultant Tony Flick, lambastes government and industry for not bringing more rigorous security policy standards and enforcement regimes to the Smart Grid in the formative stages. And puts the onus on NIST to get things back on track stat:

The opportunity to integrate security into the smart grid from the very beginning has already passed; however, most of the implementations have been small. Before larger implementations continue, such as the smart grid rollout in Miami, the security frameworks and initiatives surrounding the smart grid technology should be allowed to mature. While NIST is the proper organization to issue the security requirements, more granular requirements need to be addressed. Technology companies should not be left to determine which authentication mechanism to implement or what encryption key size to use. NIST should be responsible for determining these requirements.

Not sure that can happen ... or can happen fast (or well) enough. My wish is that we'd use hundreds or thousands of more manageable microgrids as Smart Grid security pilots. Find out what works and what doesn't on a smaller scale, before bringing inadequate policies and technologies to bear on entire cities and regions. This would mitigate risk of large scale trouble, generate a heaping helping of new data, and give NIST more time to develop necessarily "more granular" standards in coordination with industry partners.