Old Reliable: A New Grid Needs New Virtues

On March 24nd, the US House of Representatives Subcommittee on Energy and the Environment considered and approved a new piece of legislation, the "Grid Reliability and Infrastructure Defense (GRID) Act ", moving it to the full Energy and Commerce Committee for debate. It was co-authored by Massachusetts' own Ed Markey, and is intended to raise awareness and responsibility for protecting the next generation of electrical grid infrastructure, the one that gets improved with Internet-style technologies.

It would be hard to argue with the premise of this bill, that our own national defense relies on power, and that power is largely expected to be derived, or backed, by the national Grid. Simple logic tells us that:

Threat to Our Grid = Threat to Our National Security

Y2K Redux: Smart Grid's Unforeseen Benefits for Utilities

A long time ago I left the Air Force and joined the business world. An exciting job as a technology analyst at Aberdeen Group called me back home to Boston. But after one year in the ivory tower, with the opportunity to meet and in some cases interview some of the major IT movers and shakers of the day (name drop alert: Sam Palmisano, Craig Mundie, Scott McNealy), I chose to dive into the IT trenches and found a spot in a local boutique IT services firm named Primeon. What was Primeon's claim to fame back in 1999? Y2K of course.

I soon learned my job was to stress to potential partners and customers that it was time to take stock. To sort out, if they hadn't already, what hardware and software their enterprises depended upon most. The idea being: you can't remediate what you don't even know you have.

Well, what do you get when you do a real inventory (also referred to as the practice of "asset management")? First of all, lots and lots of grunt work if you're being thorough. But beyond that there's plenty to learn about your own operations and efficiencies:

• How much shelf-ware (software your org bought but doesn't use) you're paying for in yearly maintenance/support
• As with the US Navy, how many redundant applications you have, and how much money you might save through consolidation (i.e., pulling the plug on the older, less functional, harder to maintain ones)
• How well your org adheres to policies that matter, like industry interoperability standards, or how many digits are required in the year field

In short, perceived Y2K threats and remediation costs were used to justify the development or purchase of newer apps and the shuttering of older apps and systems. It became a catalyst for modernization and efficiency that continues to confer benefits to the more aggressive organizations today.

How's this apply to Smart Grid security? Much of the work to be done to get ready for AMI and Smart Grid capabilities involves linking and integrating systems that were previously isolated from each other - that wasn't a Y2K survival requirement. Of course there are other big differences between preparing for Y2K and roll-out of the Smart Grid. With few exceptions, the Y2K window opened and closed in a 24 hour period, while new Smart Grid applications and equipment have been rolling out in fits and starts for several years, and will continue to arrive for the foreseeable future. And the threats to Smart Grid systems are infinitely more varied and complex than the year date problem was to computers more than a decade ago.

Jack and I maintain that you can't secure (or demonstrate compliance with) what you don't even know you have. You can't understand the most vulnerable junction points between your IT and SCADA systems if you're not really sure how one or both is secured on its own. It's hard to prepare to roll out needed enterprise access control or single sign-on capabilities when you have no idea how current users are granted or denied access to key systems pre-Smart Grid.

As more utilities turn to asset and portfolio management processes and systems as a precursor to doing Smart Grid right, there's reason to believe a resurgence of taking stock a la Y2K is at hand. And beyond being better prepared to operate in the highly interconnected world of the Smart Grid, there are additional benefits to be had for utilities seeking greater self knowledge.

Inviting Smart Grid Consumers to the Dance

I may miss a few, but the list of the biggest threats to the success of the emerging Smart Grid includes:

• Complex technology
• Well equipped, sophisticated attackers and other bad actors
• Pressure to deploy ahead of still-forming standards
• Immature or hastily conceived business plans/model
• Aging equipment and aging workforces
• Organizational and cultural rifts inside utility companies (e.g., IT vs. operational)
• Inter-state legal and other jurisdiction challenges

But perhaps the greatest is also the simplest to understand and articulate: achieving real two-way communications between utilities and their customers. And I'm not talking about bi-directional digital networks; I'm talking old school ... meaning starting from zero and taking deliberate steps to forge and maintain real working relationships between providers and customers.

Maybe a little bit on the late-side (considering the recent, less-than-optimal experiences of PG&E, Xcel and Oncor customers) but better late than never, this article in Smart Grid News announces the formation of the non-profit Smart Grid Consumer Collaborative (SGCC).

It's going to take more than this to get the word out. Many worldly and well educated peers in other sectors still draw a blank when they hear the term. Others have heard of the Smart Grid, but don't have the foggiest notion of what it is or why it's coming. I know because because, to their chagrin, I ask folks all the time. The formation of the SGCC isn't a full solution to the Smart Grid customer communications challenge by any means, but it sure smacks of a move in the right direction.

You can visit the SGCC site here and we recommend you do.

Photo Credit: The Seattle Municipal Archives on Flickr

Grid Cascade Report: Trap or Training?

As the grid grows more complicated and more confusing, many of us are spending time thinking about the ways in which we can hopefully make it more secure, or at least more reliable, in the face of a new wave of threats and dangers. An article in the March 20th issue of the New York Times, "Academic Paper in China Sets Off Alarms in U.S." describes a new twist on an old distraction: state-sponsored attacks, in this case from China.

First off, I am not going to make any judgments about whether or not we are in the cyber-gunsites of any nations. I always assume that cyber-warfare/defense is now a common discipline in most technologically developed countries, some of which like the United States a lot, and some of which may like us a little less. If you are interested in some relatively comprehensive discussion on the topic of China's capabilities, you can take a browse at a Northrup Grumman Corporation report done for the US-China Economic and Security Review Commission, entitled, "Capability of the People’s Republic of China to Conduct Cyber Warfare and Computer Network Exploitation." There is a ton of information there, and a detailed analysis of practices, training, and competencies, but that is not really my issue here.

Cyber-attacks, their origins, purposes, etc. have always been notoriously difficult to divine. Once somebody is caught, there are occasional revelations; the ever-popular "disgruntled former employee", the "group of (pick a nationality) extremists", the "hackers associated with (pick a cause)". In general though, understanding the ultimate source of an attack or the mindset of the attacker is more like reading tea leaves than reading a bio. It even happens to the US, as is the case here in recent news from Iran, "Iran arrests 30 accused of U.S.-backed cyber war". That lack of real conclusive correlations in so many attacks has always led me to focus on the vulnerability, or the exploit, or the damage. What can we learn, what can we do, how can we help?

In this case, the Times' John Markoff and David Barboza are writing about the testimony given by Larry M. Wortzel, Commissioner to the afore-mentioned U.S.-China Economic and Security Review Commission, on March 10, 2010. In that testimony, there is mention made of a paper issued by two academics in China on:

"...how to attack a small U.S. power grid sub-network in a way that would cause a cascading failure of the entire U.S. west-coast power grid."

Now that sounds serious.

I am not going to pretend that I have taken the time to review the mathematics that underpin the researcher's report, entitled, "Cascade-based attack vulnerability on the US power grid", and I will assert up front that the formula they use in their abstract is enough to give me flashback memories of long mornings spent contemplating another vocation while in Troy, New York, but I have read it. And anyone can understand that even in their abstract, they are letting the cascade cat out of the bag, because they state that their research produced a "counterintuitive finding", that an attack on the lowest load nodes of a system would be more damaging than attacks on the highest load nodes. Who knew?

Giving away this kind of revelation seems to fly in the face of the sort of tone of the remarks that this article was a blueprint for attacks. This was a report on a surprising aspect of grid vulnerability, and for those who will actually read the report, it closes with a straightforward note on the writers' hope that these results described may "...have practical implications for protecting the key nodes selected effectively and avoid cascading-failure-induced disasters in the real world." To me that looks like well-meaning advice, not like a plot.

Back in 1982, Amory Lovins and L. Hunter Lovins published a book on cascading failures and more, entitled: "Brittle Power: Energy Strategy for National Security". It is rich in information on threats to US energy sources, and even offers relatively detailed anecdotes about the sources of risk in our national energy infrastructure. Much more recently, Amory has again written of the risks with a modern DoD-oriented view, in an NDU article entitled, "DOD’s Energy Challenge as Strategic Opportunity" where he relates that:

"the U.S. electric grid can be interrupted by a lightning bolt, rifle bullet, malicious computer program, untrimmed branch, or errant squirrel."

It would be difficult to find someone who has worked as long to elevate the discussion of energy security or its national importance, and yet many of his messages are also about inherent vulnerabilities that can topple our grid. Lovins helps us to see ways in which we are at risk, and to think about different ways to arrive at resolution.

While picking up the cited article on cascading failures, I browsed around to see what other related topics could be found there, particularly from China. There were plenty. The way I figure it, there is probably a ton of power needed in an industrializing economy growing as quickly as China's, and so they are probably investing a ton in understanding how to make that power reliable. There are a couple of other articles focused on attack strategies to exercise and understand the grid, and another about using power flow entropy as an early indicator of impending failure.

I am not so innocent as to believe that cyber warfare is not planned and practiced by nations all over the world, but there is also research and science that can be leveraged. I hope that our legislators, lobbyists, and scientists use these papers to inform the security of the Grid with at least the same enthusiasm that they present them to us as indicators of international threat.

Smiling Chinese Outlet Photo Courtesy of:

http://www.flickr.com/photos/fdecomite/ / CC BY 2.0

Smart Grid Confidence Game Part 2: Earning Operator Trust

Back in December, in a post called "Smart Grid Security Confidence Game" we considered how the drum beat of media messages about potential Smart Grid vulnerabilities and risks might affect the public's enthusiasm for the Smart Grid, and enjoined Smart Grid community advocates to add their security and privacy strategy accounts to the mix and bring a little balance.

But there's another group whose trust is just as important in this transition, and that's the folks who actually run the grid on a day to day basis. In this recent article about "Self Healing" capabilities, tech journalist Alyssa Danigelis gets to the heart of the matter from a user confidence point of view:

The major challenges to adopting advanced self-healing grid applications don’t necessarily stem from a lack of available technology.... The problem seems to be getting to the point where utilities can integrate them into intelligent systems and then comfortably rely on the processes to work smoothly and safely. To put advanced self-healing grid systems in perspective, last year some utilities still were using paper maps, says Gary Rackliffe, VP of ABB in North America. “Going from paper maps to saying, ‘I’m going to sit here and watch the computer make the decision and start throwing switches’—that’s a significant leap forward,” he says.

Automated systems make thousands of decisions per second. But still, imagine getting pilots to take their hands off the controls and trust early autopilot systems for the first time? What about autopilot and terrain following radar for high speed, low level flying? Yikes! At some point, though, you've simply got to let go because the system is far too complex to fly manually. And once they see it working, initial skeptics become the strongest advocates:

Progress Energy’s Harrison expresses certainty that when workers are trained on a self-healing system and understand how it will react, they will adapt. She recalls when the utility’s distribution control center first started telling linemen to go to a location because the outage would be in the vicinity. At first, line workers doubted control room operators’ ability to pinpoint the cause of an outage, and they’d say, “I’m going to have to ride this line out.” Then the accuracy improved so much that the linemen stopped questioning it. “Today if for some reason that information isn’t available,” Harrison says, “they get upset. ‘What do you mean you can’t tell me?’”

We're still quite a few years away from being hands-off on the grid. But the momentum taking us towards more automated grid operations is now unstoppable. Imagine a future where energy system engineers and executives review outage "near misses" detected and prevented by ultra-fast reacting, self-healing systems. When the public gets the word that the Smart Grid has earned its operators' trust, it'll be easier for them sit back, relax, and enjoy a more flexible and reliable energy system.

Photo Credit: Global Jet/Bill @ Flickr

Getting Started and Smarter

“There are two mistakes one can make along the road to truth...not going all the way, and not starting."

- Prince Gautama Siddhartha, 563-483 B.C.

It is clear that the Smart Grid has developed a form of its own momentum, and it is a momentum expressed in dollars (planned, if not spent). Many of the projects are just beginning, and much of the funding is yet to be disbursed, but there are important steps that can be taken now. These are steps that will be much harder to apply once the projects hit full speed in their deployment, and there is a need for some thought before urgency begins to trump security.
Whether one looks at the security components of the SGIG planned investments totalling $3.4B, or at the results from Pike Research which call for $21B to be spent on cybersecurity over the next 5 years, it is obvious that the context for these decisions will be broader than any single initiative, and that security will need to hit the ground running when these funds begin to flow. Forward-thinking Smart Grid security planners are looking for things that they can begin working on now, in the relative calm before increased funds and expectations accelerate and super-heat any plans for securing their new efforts in the Smart Grid.

As 2010 was getting started, I was asked by Forbes.com to create a prescription for better security in the new year. My advice was geared to a general market request, and not focused here, at the Smart Grid, but some of the same recommendations that will help a bank or a retailer to be better protected can channel this new wave of investment in secure directions. Because some of this is going to take some space to describe, I'm going to break this up across two entries. This first will focus on the "Why and What", and the second will address the follow-through.

Understanding Motivations

There are not a great many people who will argue that the Smart Grid does not have to be secure, but there are multiple underlying reasons what that security is or is not going to be a priority. The very first step in thinking about how to secure new Smart Grid projects is to understand why that security is important to your organization. It is important, at this stage of planning, to remain focused on the core question of "Why", and not get wrapped up in what it means to be secure. Motivation may be a need to fulfill all security deliverables as specified in a grant request, or it could be a recognition that rate payers in a region are particularly sensitive to privacy concerns. As an example, we wrote back in September of 2009 about NISTIR 7628, and its emphasis on the integrity of data and services. A utility would consider this direction an important driver of security, but it would only rise to the top if compliance with likely NIST guidelines was going to be the primary measure of security success within that organization.

This is not a distinction without a difference, because the coming investment and crisis of time and resources will force some pretty hard decisions, and internalizing the organization's motivation (compliance vs. profitability, adoption rate vs. energy savings, etc.) will prevent whipsaw decisions in the face of conflict.

Another area to examine is the group of individuals who will be driving Smart Grid initiatives, those that will be looking to measure successes and challenges. In some cases, the motivation can be defined simply by a sense for the downward facing pressure that is applied. At some point, however, someone has been driven by a concrete need that they are looking to fulfill, and that organizational dynamic will have much to do with ensuring sufficient support, resources, and visibility. Understanding where this critical connection is made will help to inform frequency and style of reporting, champions necessary for planning and budgeting, and the right place to go when things change or slip. In this way it becomes clear that motivation has three faces: the motivation behind the program, the motivations of the individuals supporting it, and the motivation to prioritize security among a variety of competing areas.

Determining what needs to be secured
It will be impossible to have any sense for the state of security in the new Smart Grid environment unless time is taken to inventory and understand its many components. As with any type of security, the first step that will generate a reusable artifact is this inventory. There are different approaches to take. In the actual practice of improving security, all of these approaches must be balanced, but in performing the unweighted analysis of areas requiring protection, it is helpful to limit the view to a single perspective. While fleshing out the actual plan, priorities, balance, and integration of approaches will be critical, but this is more about identifying areas, and less about articulating security strategies for those areas. Here are the three of most common lenses:

• Data Oriented
  In a data oriented view, the security approach is driven by consideration of the types of information that will flow into and through the system. The inventory will contain elemental-level identification of data from customer meters, from other providers, rate settings, internal financial systems, customer interaction portals, etc. Each of these elements is then tagged with security characteristics. These include privacy, lifespan, destruction methodology, communications required, online storage capability, and any other security-impinging implication of internal or external requirement.
• Function or Service Oriented
  A different view, and one that can be more suited to integration efforts of existing systems, focuses on the action functions or services that the new systems are intended to provide. In this model, the security of new systems that are to be developed are first understood through their specifications. Items to include are lists of all existing systems that are touched, and all platforms that will be integrated in the planned infrastructure. Each of these connection points should be assessed for existing of appropriate security characteristics such as authorization, encryption of data in transit and storage, auditing and logging requirements, and expectations of platform stability and security.
• Threat Oriented
  Longtime security devotees and practitioners have traditionally favored a threat-based approach. This involves an inventory of the active risks that will confront the system once it is deployed. At a high level, these include areas of exposure to external hacking, internal attack and data theft, malicious code, data corruption, and the breaches of ancillary but connected systems. Each of these risks is noted, as is the area of the system that is likely to be impacted. This inventory is later balanced by likelihood of actual breach, and appropriate mitigating controls can be applied.
  
  I like this methodology least among the starting techniques because it tends to be a limitless list, expanding with each creative turn of the listmaker's mind. Given the newness of IT security in the pantheon of concerns around the Smart Grid, and given the disparate skill sets that describe Utility security professionals and their IT-soaked cousins, this list will naturally be incomplete. A second negative about this approach is that it is one step removed from actual knowledge of what to do to address the issues. Any threat can typically be addressed from multiple perspectives, and divining which approach to take for each vulnerable area is always more time consuming than starting with a good asset (data or service) inventory security model, and only then applying a threat-based technique to look for holes.

Understanding the extent to which security is going to be a priority, aligning effort to the underlying motivation for that direction, and then mapping out areas of necessary consideration and investment, will help to frame a successful security strategy. There is significant exploration and investigation work here, but it can create a much more comprehensive backdrop for security decisions, and will absolutely improve the manageability of the project once in action.

So far, we have been looking at the initial exposition stages of the security plan for Smart Grid initiatives. There has not been consideration of the next steps to take in drilling down to forms of action, that will be done in a later entry to the blog. This does not mean that completing these tasks in not valuable in their own right. The artifacts that will emerge will be a living list of areas to watch, and can be used as anchors for new efforts, consistent reporting, and measures of progress. The discussions that take place during their creation will serve to raise awareness and sophistication of understanding among many parts of the organization, and that will grow organically as these efforts mature.

Photos courtesy of:

http://www.flickr.com/photos/monana7 / CC BY 2.0

http://www.flickr.com/photos/robinsoncaruso/ / CC BY 2.0

http://www.flickr.com/photos/49024304@N00/ / CC BY 2.0

SmartGridCity Competition: Infrastructure vs. Applications

You've got to deploy new meters and AMI infrastructure just to get in the Smart Grid game. But what if the costs go beyond your projections and you've agitated your rate payers well before delivering to them pledged new benefits and capabilities? At this point you're ready to buy or build the customer-side software applications that can begin to deliver on the many and several promises of the Smart Grid, but your hands are tied by tightening budgetary strings. As energy journalist Stephen Monroe puts it:

There is plenty of precedent for utility-scale subsidization of such "behind the meter" costs as programmable thermostats, compact fluorescent bulbs and high-efficiency furnaces. But with sunk costs for SmartGridCity already in the thousands of dollars per meter, regulators and ratepayers this year must decide how much more the system can bear before the project crosses from a forward-looking investment to one of never-present value.

Two risks come to mind in considering Boulder's current dilemma. The first is that the AMI/Smart Grid build out falters due to loss of regulator and ratepayer confidence. We need these advanced capabilities badly and failure to deploy the new grid, and its enabling applications, is unacceptable ... but possible, if we don't learn and adapt from the experiences of first movers.

The second risk is that in situations of substantial financial stress, security requirements are sometimes tossed out the window. Whether buying or building, deploying secure applications takes time and money, and the impulse to "deploy now and secure later" will be very strong.

To do so would be to put off failure for another day, in another form. Winning hearts and minds via powerful applications is a winning formula for the Smart Gird. Unless its merely prologue for widespread disappointment or anger from breaches involving loss of private data and/or system outages.