Is International Collaboration in the Cards for the Smart Grid?

There are currently Smart Grid conferences, planning committees and pilot deployments happening on every continent except maybe Antarctica. Yet most everything I've read to date concerns work being done in the US. I can tell you, however, that many of the readers of the Smart Grid Security Blog are from Europe and Asia. I can also relate that after moderating a Smart Grid panel at a recent clean tech conference in Boston, I was approached by a gentleman who wanted to ensure I knew about a big RFP coming out soon to build a Smart Grid for the city-state of Singapore. (Here's a link to a conference that just took place there.)

So, with that said, here's a short post on the international angle: le Smart Grid. Warning: if you favor answers, this post is light on them and chock-a-block full of questions. Here's a few starters to get us started:

• Will the fully deployed Smart Grid have borders?
• In North America, will the Smart Grid eventually transcend the current regional topology of Regional Transmission Operators (RTO's) and Independent Systems Operators (ISO's)?
• While the electrons that constitute my emails transit the continent (heck, most of the globe) with ease, the same cannot be said for the electrons currently bringing my monitor to life. Will the Smart Grid change this?
• Is there anything the US can learn from early international efforts in Europe, where Germany was a first mover?

According to this recent article from Smart Grid News, seems like current thinking, in the US anyway, may not be very collaborative ... at least not as far as security is concerned. Here's a recent statement from a Canadian Electricity Association (CEA) VP on how current Smart Grid security legislation and standards make no mention of working as a team with our partners in the Great White North:

[The US has] got to realize that the North American grid is international, it's interconnected, it's integrated. Consultations, cooperation between governmental authorities on both sides of the border is going to be imperative, otherwise you won't be able to ensure system reliability and you'll probably undermine system reliability.

I realize my understanding of these issues is likely simplistic. Yet the ability to quickly "island off" healthy portions of the grid from unhealthy ones is key functionality every region and every nation is shooting for. But islanding should be an emergency response, not the square one status quo inside the US or among close allies.

Smart Grid Privacy Before Smart Grid Security

Can we have a little privacy, please? The question of how to secure a system isn't fully relevant until you've figured out what needs securing, and that often begins with policy decisions on how to manage sensitive customer data.

Here at the SGS Blog, our relentless quest for more and better info re: the state of security policy and technology for the Smart Grid sometimes has us overlooking things of a less technical, but no less impact-full nature, like privacy. As Katie Fehrenbacher of earth2tech puts it:

"Smart Grid security” is most often discussed in the terms of national security — a hacker develops a worm that can jump across smart meters and black out neighborhoods, for example, or can make a generator blow up remotely. Privacy — keeping personal information in the hands of the consumer and away from advertisers, the utility or any other third party — is an entirely different concern that utilities have to be prepared for with the build-out of the Smart Grid.

Yup. From the maltreated customers' point of view, be they large commercial or industrial enterprise or a simple household, it matters little whether their data is divulged via hacking or poor privacy controls. The simple fact that someone or some organization in a trusted position was less than fully responsible with their financial, health, behavioral (or other) info is more than enough cause to trigger a call to Attorney911. And media reports of privacy debacles will serve to greatly reduce confidence and enthusiasm for wider Smart Grid deployments.

So much depends on the customers' first experience with the Smart Grid and the amount of control over privacy decisions they are given. Here's draft privacy standard verbiage from Rebecca Herold, who in addition to being "The Privacy Professor," doubles as an energetic volunteer on NIST's Smart Grid Privacy Group:

Consent and Choice: The organization must describe the choices available to individuals and obtain explicit consent if possible, or implied consent when this is not feasible, with respect to the collection, use and disclosure of their personal information.

That sounds like a great way to begin a new relationship. Mutual consent. The freedom to say "I do" or "I do not." And why do I say "new" relationship? One thing we've learned in our recent travels through the Smart Grid universe is that the typical US utility has a less than stellar understanding of its customers. And the adverse is true: many utility customers cannot even name the company that supplies their electricity. If Smart Grid dreams do indeed come true, both parties will soon be on a first name basis. They're going to learn things they never knew about each other before. And if it's done right, they will come to trust each other with some very important information and services.

Photo Credit: NotionsCapital on Flickr

Smarter Grid. Struggling SCADA?

In June of this year, the FBI arrested a hacker named Jesse McGraw (aka "GhostExodus") for installing malicious software on a couple of systems at a hospital in Texas. He didn't crack some protocol or breach a server, he allegedly walked around in his security guard uniform and a "hoodie" with a USB drive carrying malware. An ultimate insider.

The entire episode can be found in a very readable account at the website of the somewhat eponymously named Wesley McGrew, who actually located and identified McGraw after a relatively short period of social network mashing, Googling, and just good, old-fashioned rational thinking. ( For those of you with eye-strain from concentrating on the Smart Grid Security Blog, there is also a very good podcast interview with McGrew by Michael Farnum at An Information Security Place.)

The story has been told in multiple places, and was widely covered in local media at the time, but in doing some research today on SCADA vulnerability and exploitability, there were items in the complaint, in the write-up, and in the comments (some of them quite scathing) from the hacker's cohorts to McGrew's account of the events, that made me think of the SCADA security challenges associated with the new Smart Grid environment in some different and more urgent ways.

What Once was Old is Old Again
It is not news that components of SCADA systems can be older and have been designed for reliability and stability on mainly protected networks populated with trusted people. In discussing his motivations for researching the attacker, and for calling the authorities, McGrew cites his current doctoral research in information security, particularly in SCADA security. When he discovered that the attacker had installed botnet software on a hospital HVAC system, his level of urgency shot up. He feared that even modest corruption of that system could cause real danger to patients, at one point referring to SCADA systems of the type as a sort of "rickety ensemble" of old and new pieces, which could not be expected to withstand much tinkering.

He is not alone in this expectation. In a presentation back in 2007, delivered at HITBSecConf2007 Malaysia, called "Hacking Scada", other statements supported this fear, including the fact that ordinary anti-virus software could be expected to crash many SCADA systems due to the increased load, and that simple utlities like "ping" had been shown to bring those assets down.

As an IT person coming to utilities, I had expected vulnerability, but did not expect the real fragility in these important systems.

HMI by DIY
I was also surprised to learn that many of the front-ends ( HMI or Human-Machine Interface systems) of these newer SCADA implementations are actually created on-site. Think of it as a Do-It-Yourself graphical user interface. This is necessary, in as much as most of them are actually doing extremely custom things. The presence of different sensors, different arrangements, different control structures, demand that the interface itself be created in a way that is very much tailored to the environment that is actually going to be managed.

I learned this while researching the new importance of the internet protocol and even web-oriented interfaces, as components in the HMI interfaces of these systems. Packages actually ship with IDEs (Integrated Development Environments) containing libraries and widgets necessary to create useful, functional, and hopefully intuitive representations of the complex system of sensors, RTU's, PLC's, and more. It is not clear how seriously security is regarded in the creation of these custom interfaces, or how simple it can be to enable security controls available through the IDE's. It appears that there exist few standards and fewer tools relating to their certification.

Getting Warm in Here?
As it was with attacks and breaches in the early days of the Internet, the facts surrounding the means of identifying the actual attack and attacker are discouraging.

Based on the reporting from the hospital...which existed in Texas...in summer...it's hot there...the air conditioning system had failed multiple times, and they didn't check for, or find, the remote control software on the HVAC system. Instead, a researcher hundreds of miles away had gotten an unrelated message from a hacker, did some research, and discovered from pictures of the HMI screens that the system had been corrupted.

Admittedly, information security may be relatively new to the traditional SCADA user, but there needs to be better tooling, or better integrity assurance, or just better education and awareness to make some information security analysis more standard.

IT Hacking Ignorance
It could be that the most dangerous reality of this article could be summed up in the uninformed actions of the attacker, and the reactions of others to his arrest. The malicious software that was delivered through a USB drive into an exposed USB port, was a botnet, remote control software, and the attacker was planning a "massive" denial of service attack from all of his controlled machines.

I think it is pretty clear that this guy did not know how unstable this system would become, or how important HVAC is in a hospital in Texas. Operating room environmentals, pharmaceutical storage temperatures, patient recuperation, are all intimately connected to those systems. It is literally life and death. It is hard to imagine from the descriptions of the attacker and his attack that he construed his incursion as being as dangerous as it was. Similarly, the ignorance of many of the comments on his arrest miss this entirely, presenting their view of the attack as being that he "hacked an air conditioner or something".

Whether it be in the minds of the internal resources who do not think about information security and an HVAC system, or external attackers who do not understand the complexity, seriousness, and importance of these newly interconnected SCADA systems, the fundamental disconnect on action and effect need to be made much more visible.

The reliance of SCADA-enabled systems like HVAC on their actual software, and the reliance of the utilities and customers on these SCADA systems is a connection that is becoming obvious as the Smart Grid expands the number and the exposure of these systems to all.

Images Courtesy:

http://www.flickr.com/photos/borkurdotnet/ / CC BY 2.0

and

Wesley McGrew

Seeking a Balanced Perspective: How Cyber Risks to Grid are and are not MAD

As you may suspect by now, Jack and I are not fans of alarmist language. You won't hear us using terms like "Cyber Pearl Harbor" or "Cyber 9/11" unless our purpose is to debunk them, as Jack did quite thoroughly on his former blog, Suitable Security, here. We find that hysteria is not a particularly promising state of mind to be in when one is attempting to make the world better, safer and more secure. And that's the lead-in to this second post re: the recent 60 Minutes feature on ominous trouble in Cyberland.

Oh, one more thing before the post really starts -- I should explain the kitten. This kitten is here to help you relax. OK? Let's begin.

MAD, or Mutually Assured Destruction, is a Cold War-era term which neatly describes why nuclear deterrence works and has so far kept our planet from being reduced to a glowing ember from a massive thermonuclear exchange. You are still relaxed I see ... that's good.

Last week we posted a link to, and a couple comments on, an alarming 60 Minutes episode on cyber security risks to critical US infrastructure. It described how vulnerable the US is to computer hackers and used examples from DOD, the financial sector and the electrical grid. An additional level of disturbing detail was provided by former Director of National Intelligence (DNI) Mike McConnell, who said he's certain that foreign code is resident on national grid systems. Our own anecdotal experience with critical systems in other industries corroborates this. In hacker lingo: we are "owned."

Still relaxed? You should be, because there's ample evidence, in the 60 Minutes material and elsewhere, that even as we are heavily targeted, we also have substantial penetration of our potential adversaries' systems. Hence, the resemblance to MAD. I'm making this comparison preemptively before some journalist or K Street analyst does, because I think it's worth laying a few of the cards on the table and thinking about this in a non-alarmist fashion. Here's a short list of attributes to compare and contrast:

Nuclear characteristics:

• Once underway, nuclear war is for keeps: you're either launching nukes or you're not
• Though some once believed in it, "limited nuclear war" is generally considered unlikely
• While we work to make missile defense a reality, our best defense against nuclear attack has been a good offense (see: deterrence)
• Damage from nuclear exchanges is usually believed to be catastrophic
• With missiles and bombers heading our way, it's fairly easy to discern the origin of attack, and hence, the attacker
• There are currently 9 countries listed as nuclear nations. Others seek to join this group, but it's expensive, complicated and time consuming, not to mention dangerous and sometimes destabilizing

Grid Cyber characteristics:

• Probes and attacks are happening all the time by multiple parties and damage of various degrees is being absorbed by all involved
• All cyber war is, by definition, limited
• Our best defenses are multi-layered, resilient and constantly evolving
• Damage is infinitely variable in severity and often hard to detect
• Often cannot identify attack origin or attacker
• Any country, organization or individual with access to the Internet can be an attacker

So the Cyber wars are already well underway and yet you are still able to read this post on your computer or smart phone. This is because given the degree of inter-dependency of the global economy, most industrialized nations have little desire to wreak massive cyber havoc on their neighbors, who, while they compete in many domains, are also full time partners. Though you'll sometimes hear speculation to this effect, especially as it concerns the Smart Grid as a "hackers' paradise", it's unlikely (though possible) that catastrophic harm can befall the diverse US national grid from cyber attack alone. But that doesn't mean major localized or regional damage couldn't be wrought.

Take aways:

• Unlike with nukes, where deterrence between nuclear nations has worked so far, no one is fully deterred from experimenting with and sometimes wielding cyber weapons against our grid or other critical US infrastructure systems. Most nations do, however, seem deterred from launching massive cyber attacks on us and others ... and life and commerce go on
• International crime gangs and other non-state bad actors abide by completely different rule sets from those described above. Deterrence means much less to them, so we've got to continue to bring our cyber security "A game" to the Smart Grid build out as well as to the rest of our critical national infrastructure
• Understanding and accepting that all sides "own" other systems conjures up the alternative title to the Cold War classic "Dr. Strangelove," which was "How I Learned to Stop Worrying and Love the Bomb." I'm not suggesting you begin loving cyber risks to the grid or Smart Grid; just want you to worry a little less if the 60 Minutes piece has rendered you sleepless or immobile. Clearly we’ve got work to do, but as NASA and the NY Times said today, we’re not going to die tomorrow or the day after tomorrow
• For a somewhat more detailed, balanced examination of cyber risks to the grid, see University of Minnesota's Dr. Massoud Amin's short paper "Electricity Infrastructure Security", PDF downloadable here.

So, if you've made it this far, I've got a question for you: did the kitten help?

Photo Credit: Catherine Caf on Flickr

Smart Meter Increases "Suit" Pacific Gas and Electric

On November 16, 2006, at a lucky customer's home in Bakersfield, CA, PG&E launched its SmartMeter program, designed to alleviate costs for customers, costs for supporting the power grid, and the cost of generating so much energy in the area. Even the commissioners were optimistic, as reported in a PG&E press release:

"I am pleased to witness today the installation of the first smart meter for a PG&E customer," said Michael R. Peevey, president of the California Public Utilities Commission. "This technology will link the prices energy consumers pay to the costs of that energy in the wholesale market, empowering consumers with the information necessary to make sound energy choices. Research suggests that even modest levels of price sensitivity in the retail market can yield substantial benefits as customers decrease or shift their energy usage. These types of demand response programs are one of the best ways to meet the energy needs of California's growing population, as outlined in our Energy Action Plan."

It is hard to know exactly when the honeymoon ended, whether it was when Bakersfield.com reported on a customer who found his power usage had tripled during a six-hour blackout, or at the town meeting in Fresno on October 20th which quickly became a unanimous indictment of Smart Meter-ing, or now in November, as a class-action suit has been filed against PG&E, asserting a variety of mistakes and misrepresentations. For those of us who have spent a fair amount of time researching the potential for advances derived from Smart Metering, these developments are disconcerting.

From a security perspective, there are two very important areas of guidance to take from these developments, and from the likely continuing negative perception of Smart Metering in some areas.

Integrity and Availability of Data

As we wrote here, and as others opined elsewhere, there is likely an abundance of information about to flood utilities. Some have rejected, or at least resisted, the idea that anything like high volume sampling would happen, and that aggregated data would be the more probable artifacts that utilities would store for billing and management. This suit and the ongoing outcry for justification of higher bills are exactly the reason why more detailed and regular metering information will need to be gathered and stored.

See, it is likely that these bills are actually accurate. As the commissioner stated at the outset, "modest levels of price sensitivity in the retail market can yield substantial benefits". Ok, so maybe the hot tar and chicken feathers are not necessarily a benefit, but they highlight a new awareness on the part of the consumers. It is surprising that this message of usage and contention for power has not been better absorbed by the public. Take an average citizen. They use power, like everybody else, from 8-6. Enter the Smart Grid, and the smart meters. In an attempt to incent off-peak usage, and to compensate for the increased cost of peak generation, power is more expensive from 8-6, and so the average consumer's bill, if they do not change their behaviors, is going to be higher. The smart meter only becomes an engine of positive financial impact for consumers when they figure out ways in which to really alter their power use to advantage the off-hour charges.

Until that happens, expect that there will be continuing challenges to the veracity of the smart meter data, and continuing scrutiny of the systems that collect and store it. This equals what we described in earlier posts, a need for lots of data, lots of governance of that data, and good security from authenticating the user to authorizing the billing.

Actual Smart Meter Opponents

Any publicly-perceived inequitable grab for cash by a business or utility can spawn a grass-roots movement in opposition. Ignoring the more fringe folks who bring you the youtube videos of jack-booted thugs monitoring your hot-tub to charge you with profligate energy spending, there are others who are more credibly mobilizing around this issue. An example is San Francisco-based TURN (Toward Utility Rate Normalization). With a 35 year history in utility consumer advocacy and activism, the have a new focus on the perceived inequity of a smart metering infrastructure that saves costs for utilities (better management, less truck-rolls, easier disconnects) while increasing the actual bills for consumers.

With group action, and organized effort, there comes increasing visibility and controversy around the issues, and there are likely to be more critical assessments made of Smart Metering infrastructures. This will naturally splash as well onto the overall Smart Grid approach of which smart meters are such an important part. With any such increase in visibility and controversy, individuals outside the credible groups may well begin to conspire to take more aggressive action, potentially creating a new wave of "hacktivism", with the focus in this cycle being the Grid. This will change the nature of the threat to the Smart Grid enormously, making it much more likely to experience the types of attacks that more typically plague governmental and military infrastructures.

Some of the Solution is in the Data

Many of the same constituencies who are actively opposing the Smart Meter evolution are also very much interested and involved in the promotion of more efficient energy usage and more integration of alternative sources. It is now the responsibility of the utilities to educate their customers about the actual dynamics of power and power pricing, to help them to better understand the choices that they will need to make.

For those utilities who have not yet begun to alter the finances of their customers through higher peak pricing, there is a cautionary tale here. It seems that it might well be worth 3-6 months of reporting on usage, with simulated billing and recommendations for changes, prior to actually instituting those changes. It would better showcase the insight provided by Smart Metering, would provide a sense of empowerment for the users, and would certainly eliminate some of what seems to be a sense of blindsiding on the part of the consumer.

Image thanks to the whimsical stylings of Roger Wood

Smart Meters as Rough Yardsticks

In reading through the successful Grant recipients from the Smart Grid Investment Grant Program, it was interesting to make a couple of notes:

1. Smart Meter Roll-out
   

   In the FERC's Demand-Response Paper from September of 2009, the number of Smart Meters currently implemented is roughly 8 million. Looking at the total of the specifically identified smart meters implemented as a result of successful SGIG requests, that number is now funded to get to a total of 18 million with the SGIG funding. That means that the SGIG will carry smart meter deployment to more than 20% of the FERC demand response projection of 80 million meters by 2019. Let's hope that the meters are chosen correctly.

   Per-Meter Costs
   There is enormous variability in the costs of the smart-meter roll-outs as described by the various grants. This is understandable in that the number of meters is only one criteria of many of these proposals. For some, these are an initial effort, for others they are scaling existing investment up. The meters, though, do loosely equate to the public involvement (connected by meters) that the SGIG is attempting to accelerate. As such the range and variety are worth noting.

   • 79% of grants expect associated costs of < $500/meter
   • 18% of grants expect associated costs of $500-$1000/meter
     
   • 2% of grants expect associated costs of $1000-$2000/meter
     
   • 1% of grants expect associated costs of >$2000/meter

So what does this tell us?
The information is pretty scant in the released SGIG award documents, but there are some insights, if not actual conclusions, that can be drawn from it.

1. Its about Usage
   According to the rudimentary data that is provided, Smart Meter-related projects are consuming by far the largest section of SGIG funding, and at least 85% of the total investment (SGIG and Utility/Vendor) expected for these projects. There are mentions of accommodating other energy sources, but the projects seem pretty focused on how power is consumed, and how that consumption is measured, as opposed to how it will be generated and distributed.


2. There is No Clear Standardization of Direction
   While these grants are providing the impetus for some organizations to begin work on Smart Grid infrastructure, the sheer size of them make the investment much more about rapidly scaling that adoption. Given that, and given the need to maintain stability in power, the projects themselves seem to be surprisingly one-off's, each intending to validate or optimize one organization's view of the new generation of Grid. As an example of this, take a look at the wording provided for two projects in North Carolina, from Duke Energy and Progress Energy, respectively
   

   [Duke Energy] Comprehensive grid modernization for Duke Energy’s Midwest electric system encompassing Ohio, Indiana, and Kentucky. Includes installing open, interoperable, two-way communications networks, deploying smart meters for 1.4 million customers, automating advanced distribution applications, developing dynamic pricing programs, and supporting the deployment of plug-in electric vehicles. Will also benefit customers in IN and OH. ($200,000,000 SGIG/$851,700,000 Total)

   and
   

   [Progress Energy]Build a green Smart Grid virtual power plant through conservation, efficiency and advanced load shaping technologies, including installation of over 160,000 meters across its multi-state service area. Will also benefit customers in SC. ($200,000,000 SGIG/$520,000,000 Total)

   It is hard to think of projects of this magnitude as test beds


3. Ready or Not, Here We Come
   From a security perspective, this is a massive investment in expanding the exposed surface of the grid, and it will impact a new generation of underlying communications infrastructure. Most of the synopsis data includes things like two-way communications, interactivity, new networking infrastructure, etc. That is a wholesale shift for millions of customers, and we continue to hope that people are putting hard thought into it, because those dollars will be spent, and we will need to reconcile the security one way or another.
   

I guess that last conclusion that I draw is that this program also tells us that even in these small-ish numbers, the costs are huge. Through either market forces or another wave of government investment, getting to the FERC's "partial adoption" could easily cost another $15B of government funding on this route, and another $20-30B in private investment. The numbers to get to a fuller adoption are far higher. From a security perspective, all of this continues to point back to understanding what is necessary within the new infrastructure, and what acquisition guidelines should drive these enormous purchases, because it will be impossible to unwind this once it gets moving.

The SGIG has put fuel into a very powerful and creative technical engine within the energy industry, and like an automobile, that power is generating speed. As that speed builds up, we need to see similar emphasis on keeping the headlights on so we don't crash on these unfamiliar roads.

60 Minutes Sounds Grid Security Alarm

Hat tip to my classmate and former Discovery Channel Powrtalk colleague Chris Davis for alerting me to the show that aired tonight. The popular news journal interviews former Director of National Intelligence (DNI) Mike McConnell, FBI Cyber Division Assistant Director Shawn Henry and others. It begins with cyber crime in the DOD world, goes through some real-world financial services industry examples, and concludes with conviction that the computers that run the Grid have been seriously compromised and that there's little the US government has been able to do to make private operators close out their vulnerabilities.

Remember, the subject here is the current Grid, the pre-cursor to the future Smart Grid, which will bring with it new types of additional abilities but also better ways of isolating some of them when necessary. The segment is called "Sabotaging the System" and you can watch it in its entirety right here, right now ... after a brief commercial, that is.


Watch CBS News Videos Online

Smart Grid Intro for CSO's

Having come to the Smart Grid Security discussion from the Security side of the equation, I have for years spoken at the highlight events, whether RSA, Gartner ITExpo, etc. This spring, when asked to present at CSI, I thought it would be a good opportunity that we could use to begin to bridge that IT and Utility security gap that Andy has written a fair amount on.

As such, last week I presented the following deck at the CSI IT show at the Gaylord National conference center, and it was meant to give just a taste of the Smart Grid to traditional IT security professionals, and to give some security information and guideposts to any utility folks that were there.

It turned out that we had representatives of both groups in the audience, and I have had several requests for the materials, mainly because these people wanted to begin the process of informing their own colleagues and managers. Be aware that it is intentionally light, it touches a few of the areas that are important, but it is by no means supposed to be an education on Smart Grid Security. It is more like the free chapter you would get if a book existed on the topic. Hopefully it was enough to energize some of these people who self-selected into the room and who are at least aware that there is a grid that is Smart, and there are security issues that may plague it.

Here is the deck. Please feel free to share it, and to generate a more aware population wherever you are. Andy and I expect to launch a version with voice-over in the next few weeks, so stay tuned for a truly simple way to get people to understand more about the nature of some of the challenges of securing the Smart Grid.

NERC Grid Security Update: On the Lookout for a New Order

This article is a bit jumbled, but it does communicate the gravity with which NERC CSO Michael Assante approaches cyber threats to the national grid. Quite simply, he views the threats to the grid and emerging Smart Grid to be something beyond what we've ever faced before. This from a recent panel appearance:

There was a known security rule set in the Cold War. We knew and expected behaviors. We could calculate escalation. We took this into any account when we planned any action. When cyber defenses and communications entered the military, it was a force multiplier. We appreciated what it gave us. What we didn't realize was that cyber would be the thing that destroyed the rules of order.

That last line really got my attention. We are just beginning to learn the new rules. But you have to be careful and alert. So many experts from other domains giving advice about how to secure the Smart Grid these days, pretending they understand what it's ultimately going to look like. When in fact, these are still the early days and, given the pace of technological change we've witnessed in recent years and decades, the Smart Grid of 2020 will look quite different than we imagine it today. Like Assante and NERC, all of us "good guys" need to make ourselves ready for what's coming.

Photo Credit: US Army on Flickr

Seriously - A Surge

A couple of weeks ago, I took a look at the data provided by the teams at PGE and Austin Energy, combined it with data provided by DOE, and I arrived at the conclusion that the Smart Grid will create a glut of information that the utilities had best begin planning for, because it could easily swamp both the utility and the networks that are expected to carry it.

Unsurprisingly, there was a fair amount of interest in both the conclusions I had reached and in the substantiation of the data I had used. Some of the inquiries were pretty straightforward. My thanks to Editor Katie Fehrenbacher from Earth2Tech for her thoughtful questioning and for introducing me to some equally reasonable experts from the IEEE.

Others were less open to the concept, and there were two main objections to the data. The first was based in existing utility practices. This line of questioning had within it the expectation that a meter read would only contain basic information about the identity of the power meter, the timestamp, and the meter reading itself. Were that the case, it would be possible that the data would be in a paltry range, around 14 bytes per read, resulting in a belief that such a small amount of data would never amount to anything like the avalanche I had described in the piece. The second objection was that there was little likelihood that such data was going to be stored for long, meaning, I guess, that we could design the system as though it had never arrived at all. Many of the questions came from individuals with strong/long histories in utilities, so I felt it my responsibility to validate, again, my data.

While I consider myself to be relatively well-versed on the core of these topics, it is the nature of this blog to focus on my expectations of the future based on information provided elsewhere, by others more directly in the path of the Smart Grid. That said, credibility is a big deal for us, and I decided to go back to Austin Energy, and understand better the reality of the situation from the folks who are actually doing the job, and who are considering these concerns as fundamental parts of their planning for successfully serving their clients on the new grid in the years to come. Andy and I called Andres Carvallo and Karl R. Rábago at Austin Energy, and they generously agreed to help us understand the world and the Smart Grid that they are planning for.

Smarter Grid versus Simpler Meter-Reading

One of the first things I learned was the richness of information gathering and interactivity that these gentlemen expect to coax from the new grid infrastructure. While time, location, and power used are at the heart of a meter read, there is much more to be learned. Investment in the Smart Grid would have a maximum return when the savings were more than a human reader's footwear and gasoline. Some examples are:

Device Health Information

By watching for varying temperature, periods since outage, battery power, heartbeat, and other meter variables, it is possible to better predict and recover from any failures that may happen.

Real Time Monotoring

As has happened historically with most new technologies, it can be expected that people yearning for more data will only be satisfied by that which is most current. It is unlikely to happen in the general population immediately, but history shows us that it is likely that such a real time monitoring feed may be in demand almost immediately, as customers recognize that there is now more information through which they can better manage their energy.

Energy Services Provision trumps Energy Provision Services

There are doubtless going to be additional requirements from the newly informed and empowered customer base for functionality that is logically delivered by the provider. This was a real eye opener for me, that Power Providers are now actively thinking about services that they can offer over the new and smarter infrastructure. Things like profiled energy use: "I am going away, manage my power." or "There is a spike in prices, manage me down by 10%", or "I only want to use power that is generated from renewable resources." These all require data, new interfaces, and a channel overwhich all of the control and monitoring information can be passed. Winners in the new market will be finding ways to capitalize on the need for energy-related services, and will not limit their investment to further driving down the costs of simply providing energy.

Networking Overhead

Given the complexity, regularity, and importance of this data, it is clear that a protocol (Like IP) will probably be adopted to package up and send all of this information in a payload to central systems for analysis, aggregation, storage, and action. Protocols carry their own overhead in terms of describing their content, sources, destinations, etc. None of this is free from the perspective of the systems carrying or storing the data.

Other Factors

We are only just beginning to see the potential for Smart Grid and Soft Grid enablers, leading me to believe that even my estimates are very likely to be low, particularly as we clamor for realtime monitoring and data analysis.

Based on all of this, it looks like the numbers are far from a simple 14 Byte read, and are more likely in the range given by Andres of 4K to 16K per reading. If we estimate the maximum case, the numbers are even higher than I had referenced in the earlier article. Let's not think about real-time (the numbers are mind-numbing), but instead look at a simple check every 5 minutes. 12 (reads/hr) X 24 (hrs/day) X (365 days/yr) X 16K (Bytes/read) yields roughly 1.7GB/meter/year. Multiply that by the number of meters (pick your own scope), and I think the challenge is clear. For more reality, take that number and multiply by 5 for readings every minute, or by 300 for readings every second. That's big.

So, is this a problem because the data going to cause the Smart Grid to explode like a flawed radiator hose in July? I don't think so. I think that time has proven that technical advancement has always helped us stay ahead of crushing data or processing burdens by decreasing computing and memory costs. This has allowed us to paper over our excesses with iron and silicon.

No, this is a problem because rushed, tactical, and incremental hardware adds will not make that data secure. It has to be expected that as organizations run out of room for data, they will simply rush to add more. Caught in a flood of data, the pressures for survival and successful operation will naturally trump any meaningful consideration of rearchitecting data storage for adequate and appropriate security.

This planning (and budgeting) needs to happen now. As Andres said on our call, "You cannot simply build an airplane for passengers who are 5'6" tall and weigh 140, because you can guess that your average passenger, much less your larger passengers, will simply not fit, because they are not that small." In other words, you need to plan for what you can reasonably expect, not for what will make your life, your business, or your CFO, ecstatic.

I think that this is the final insight. For firms that are seeing the Smart Grid as an enabler for cost-savings by transferring operations onto an IP infrastructure, or a wireless metering system, there is little reason to be concerned with a data glut.

For those who recognize that the Smart Grid and the coming Soft Grid will need data, and will need security, and will likely grow to fill whatever space is available, the call is clear. Plan for an avalanche, for a flood. Create systems and segregations that will allow for managing these flows reliably. Characterize what must come through, and what can be dropped, along the way to the back end. Do all of those things and the current systems will be fine, the next systems will not choke, and the ultimate end state will be similar enough to what has been planned to ensure stability, quality, and cost-effective services to all who connect to the grid.

The data surge is coming, and you can either surf it, or be pounded by it. You certainly will not be able to ignore it.

Image Thanks to:

http://www.flickr.com/photos/ndave/ / CC BY 2.0

Notes from 2009 Control Systems Cyber Security Conference

We first posted on Joe Weiss's work back in July following a presentation he gave to the Air Force. Now here's a great review of a significant annual conference, one that focuses not on IT or internet security in a Smart Grid context, but rather on the security issues related to the millions of control systems that automate the Grid. This is Joe's summary:

The Ninth Control Systems Cyber Security Conference was hosted by Applied Control Solutions (ACS) the week of October 19 in Bethesda, MD. The festivities started Monday morning with parallel activities. A tour was arranged of Washington Suburban Sanitary Commission’s Rock Creek water treatment facility. In parallel, the initial meeting of the ISA Nuclear Plant Cyber Security Joint Working Group was held.

The ACS Conference started Monday afternoon with two introductory sessions: Control Systems for the non-Control System Engineer and IT for the Control Systems Engineer.The Conference began in earnest Tuesday with approximately 110 attendees from US and international electric and water utilities, chemical and oil/gas companies, IT and control system suppliers and consultants, universities, and US and international government agencies. The Conference is called Control Systems Cyber Security is that industrial control systems are common across multiple industries. The agenda can be found at www.realtimeacs.com.

There were two hacking demonstrations of control systems and several discussions on control system cyber vulnerabilities. There was also a discussion on the need for technical control system cyber security curriculum (policy programs exist). There were two keynotes: the Honorable Yvette Clarke (D-NY), Chairwoman of the Subcommittee on Emerging Threats, Cybersecurity, Science and Technology and member of the Intelligence, Information Sharing and Terrorism Risk Assessment Subcommittee provided the lunch keynote. Whitfield Diffie gave the evening keynote and discussed control system cyber security issues from the Tuesday’s session.

There were four different sessions on actual control system cyber incidents – none of which was public! In one session, two control system engineers from two different utilities that have control systems from every major supplier discussed their recent control system cyber incidents – one had his plant shutdown. A couple interesting side notes were that existing control system logging are adequate to identify control system incidents and their control system suppliers weren’t of much help when it came to providing control system cyber security support. Both engineers felt it was so important to share information they attended the Conference on their own nickel. This is in marked contrast to the utility and industry leadership who didn’t think this conference was important enough to attend even though many were based in Washington. Wednesday evening, the Honorable James Langevin gave the evening keynote. Congressman Langevin felt this was so important he spent 30-45 minutes after his presentation answering questions and talking to the attendees.

We received a summary of government activities including legislative efforts on cyber security, cyber security activities by the Nuclear Regulatory Commission, efforts on-going at the Bonneville Power Administration using the NIST Framework, and non-governmental activities in certification and cyber incident collection. Also got a very interesting presentation on cyber security legal issues and a discussion of the Russian cyber attack on Estonia.

On the last day, NIST held training sessions on two very relevant NIST standards:

-- SP 800-53 - Recommended Security Controls for Federal Information Systems - including those for the Bulk Power System
-- SP 800-82 - Guide to Industrial Control Systems (ICS) Security provides guidance on securing Industrial Control Systems (ICS), including Supervisory Control and Data Acquisition (SCADA) systems, Distributed Control Systems (DCS), and other control system configurations