The Harsh (Security) Reality of Old Software in the Current and Future Grid

You know I'm always advocating for getting better security awareness and process into everything we do with Smart Grid software. Whether it's developing policy, getting tools and building secure processes into the software development lifecycle (SDLC), and/or educating and arming orgs on what questions to ask software vendors re: the security rigor they include (or fail to include) in the development and integration of the products they market.

Sounds like a pretty good idea. Could make the Smart Grid and utilities' worlds and the North American grid infrastructure a whole lot safer and better, right?  

But then someone comes along and rains on my parade. And not just anyone, but a card carrying grid guru with more experience in this field than just about any other mortal. And what does he say?  This:

One must keep in mind that there will be far more poorly coded, totally untrustworthy firmware and software in the field for decades (the existing installed base) than new, more secure systems following sound development practices installed over the same time period. Dealing with this reality and the fact that the old stuff will not be ripped out should be a priority.

"Thanks" to Erich Gunther of Enernex. So, sports fans, while I and others keep beating the drum for more-secure new software, would a few of you mind getting on the challenge Erich's pointing out? Like, right away please.

New SGSB Webcast is Live

SGSB Webcast 5: Smart Grid Software Security

View more webinars from Andy Bochman.

While it's fun to think of all the great new gadgets and devices that are enabled by the Smart Grid (and that the Smart Grid enables), none of them could even begin to work without the "invisible glue" out of which the entire enterprise is being constructed: software.

As we rush to deploy Smart Meters by the millions, consumer portals, HANs and iPad applications that can communicate with them, meter data management systems (MDMS) to handle the tons of data that's generated, electric vehicles (EVs) to push local electric infrastructures to the limit, and synchrophasers across the continent to give us a better view of "the greatest engineering achievement of the 20th century", it's important to not forget about software just because we often can't see it.

Misconfiguration of software assets and (usually unintentional) vulnerabilities in code are the primary pathways hackers use to breach systems, alter their behavior and reach sensitive data. This presentation is more of a "why to" than a "how to" manual. There are plenty of the latter and I'd be happy to point you to some. But the reasons for taking on this challenge are compelling, and IMHO, need to get out.  

Enough already, here you go. It's about 17 minutes long, and you'll like it better if you make it bigger (click on "Full" icon in the lower righthand corner).

Smart Grid Intro for CSO's

Having come to the Smart Grid Security discussion from the Security side of the equation, I have for years spoken at the highlight events, whether RSA, Gartner ITExpo, etc. This spring, when asked to present at CSI, I thought it would be a good opportunity that we could use to begin to bridge that IT and Utility security gap that Andy has written a fair amount on.

As such, last week I presented the following deck at the CSI IT show at the Gaylord National conference center, and it was meant to give just a taste of the Smart Grid to traditional IT security professionals, and to give some security information and guideposts to any utility folks that were there.

It turned out that we had representatives of both groups in the audience, and I have had several requests for the materials, mainly because these people wanted to begin the process of informing their own colleagues and managers. Be aware that it is intentionally light, it touches a few of the areas that are important, but it is by no means supposed to be an education on Smart Grid Security. It is more like the free chapter you would get if a book existed on the topic. Hopefully it was enough to energize some of these people who self-selected into the room and who are at least aware that there is a grid that is Smart, and there are security issues that may plague it.

Here is the deck. Please feel free to share it, and to generate a more aware population wherever you are. Andy and I expect to launch a version with voice-over in the next few weeks, so stay tuned for a truly simple way to get people to understand more about the nature of some of the challenges of securing the Smart Grid.

Seriously - A Surge

A couple of weeks ago, I took a look at the data provided by the teams at PGE and Austin Energy, combined it with data provided by DOE, and I arrived at the conclusion that the Smart Grid will create a glut of information that the utilities had best begin planning for, because it could easily swamp both the utility and the networks that are expected to carry it.

Unsurprisingly, there was a fair amount of interest in both the conclusions I had reached and in the substantiation of the data I had used. Some of the inquiries were pretty straightforward. My thanks to Editor Katie Fehrenbacher from Earth2Tech for her thoughtful questioning and for introducing me to some equally reasonable experts from the IEEE.

Others were less open to the concept, and there were two main objections to the data. The first was based in existing utility practices. This line of questioning had within it the expectation that a meter read would only contain basic information about the identity of the power meter, the timestamp, and the meter reading itself. Were that the case, it would be possible that the data would be in a paltry range, around 14 bytes per read, resulting in a belief that such a small amount of data would never amount to anything like the avalanche I had described in the piece. The second objection was that there was little likelihood that such data was going to be stored for long, meaning, I guess, that we could design the system as though it had never arrived at all. Many of the questions came from individuals with strong/long histories in utilities, so I felt it my responsibility to validate, again, my data.

While I consider myself to be relatively well-versed on the core of these topics, it is the nature of this blog to focus on my expectations of the future based on information provided elsewhere, by others more directly in the path of the Smart Grid. That said, credibility is a big deal for us, and I decided to go back to Austin Energy, and understand better the reality of the situation from the folks who are actually doing the job, and who are considering these concerns as fundamental parts of their planning for successfully serving their clients on the new grid in the years to come. Andy and I called Andres Carvallo and Karl R. Rábago at Austin Energy, and they generously agreed to help us understand the world and the Smart Grid that they are planning for.

Smarter Grid versus Simpler Meter-Reading

One of the first things I learned was the richness of information gathering and interactivity that these gentlemen expect to coax from the new grid infrastructure. While time, location, and power used are at the heart of a meter read, there is much more to be learned. Investment in the Smart Grid would have a maximum return when the savings were more than a human reader's footwear and gasoline. Some examples are:

Device Health Information

By watching for varying temperature, periods since outage, battery power, heartbeat, and other meter variables, it is possible to better predict and recover from any failures that may happen.

Real Time Monotoring

As has happened historically with most new technologies, it can be expected that people yearning for more data will only be satisfied by that which is most current. It is unlikely to happen in the general population immediately, but history shows us that it is likely that such a real time monitoring feed may be in demand almost immediately, as customers recognize that there is now more information through which they can better manage their energy.

Energy Services Provision trumps Energy Provision Services

There are doubtless going to be additional requirements from the newly informed and empowered customer base for functionality that is logically delivered by the provider. This was a real eye opener for me, that Power Providers are now actively thinking about services that they can offer over the new and smarter infrastructure. Things like profiled energy use: "I am going away, manage my power." or "There is a spike in prices, manage me down by 10%", or "I only want to use power that is generated from renewable resources." These all require data, new interfaces, and a channel overwhich all of the control and monitoring information can be passed. Winners in the new market will be finding ways to capitalize on the need for energy-related services, and will not limit their investment to further driving down the costs of simply providing energy.

Networking Overhead

Given the complexity, regularity, and importance of this data, it is clear that a protocol (Like IP) will probably be adopted to package up and send all of this information in a payload to central systems for analysis, aggregation, storage, and action. Protocols carry their own overhead in terms of describing their content, sources, destinations, etc. None of this is free from the perspective of the systems carrying or storing the data.

Other Factors

We are only just beginning to see the potential for Smart Grid and Soft Grid enablers, leading me to believe that even my estimates are very likely to be low, particularly as we clamor for realtime monitoring and data analysis.

Based on all of this, it looks like the numbers are far from a simple 14 Byte read, and are more likely in the range given by Andres of 4K to 16K per reading. If we estimate the maximum case, the numbers are even higher than I had referenced in the earlier article. Let's not think about real-time (the numbers are mind-numbing), but instead look at a simple check every 5 minutes. 12 (reads/hr) X 24 (hrs/day) X (365 days/yr) X 16K (Bytes/read) yields roughly 1.7GB/meter/year. Multiply that by the number of meters (pick your own scope), and I think the challenge is clear. For more reality, take that number and multiply by 5 for readings every minute, or by 300 for readings every second. That's big.

So, is this a problem because the data going to cause the Smart Grid to explode like a flawed radiator hose in July? I don't think so. I think that time has proven that technical advancement has always helped us stay ahead of crushing data or processing burdens by decreasing computing and memory costs. This has allowed us to paper over our excesses with iron and silicon.

No, this is a problem because rushed, tactical, and incremental hardware adds will not make that data secure. It has to be expected that as organizations run out of room for data, they will simply rush to add more. Caught in a flood of data, the pressures for survival and successful operation will naturally trump any meaningful consideration of rearchitecting data storage for adequate and appropriate security.

This planning (and budgeting) needs to happen now. As Andres said on our call, "You cannot simply build an airplane for passengers who are 5'6" tall and weigh 140, because you can guess that your average passenger, much less your larger passengers, will simply not fit, because they are not that small." In other words, you need to plan for what you can reasonably expect, not for what will make your life, your business, or your CFO, ecstatic.

I think that this is the final insight. For firms that are seeing the Smart Grid as an enabler for cost-savings by transferring operations onto an IP infrastructure, or a wireless metering system, there is little reason to be concerned with a data glut.

For those who recognize that the Smart Grid and the coming Soft Grid will need data, and will need security, and will likely grow to fill whatever space is available, the call is clear. Plan for an avalanche, for a flood. Create systems and segregations that will allow for managing these flows reliably. Characterize what must come through, and what can be dropped, along the way to the back end. Do all of those things and the current systems will be fine, the next systems will not choke, and the ultimate end state will be similar enough to what has been planned to ensure stability, quality, and cost-effective services to all who connect to the grid.

The data surge is coming, and you can either surf it, or be pounded by it. You certainly will not be able to ignore it.

Image Thanks to:

http://www.flickr.com/photos/ndave/ / CC BY 2.0

Smart Grid Security: Answers in Questions

Over the past year, Andy and I have written about the risks and opportunities in the growing software sector of the Smart Grid Marketplace. We have described the space, some of the firms, the investment, and what we are seeing for security in those organizations we speak with. In response, and I think with genuine interest, we've been asked what we are worried about, and in turn, what recommendations would we specifically make to individuals who are either investing in these solutions, or who are actually building them.

In the recent NIST strategy and requirement recommendations release, there was a substantial body of information to be reviewed, and this post is not meant to summarize or to supplant those results (obviously). This is a relatively lightweight view of heavy duty and high-level considerations in software as a critical element in the development of the Smart Grid. It is a practical list of questions that organizations should be able to answer before they commit to software that will either replace or broker their interactions with the Smart Grid.

What is the software's provenance?

Provenance is a term that gets thrown around a lot, but I use it to express the idea of origin. Where did the software come from? Who made it? What was it made from? While absolute provenance is difficult or impossible to ascertain, these answers can help to guide risk awareness and management. Is it new software built for me? Is it existing software that has run similar systems elsewhere? Is it a new solution from an existing partner, or revision 0.9 from a start-up? Is it built from the ground up, or does it contain elements of legacy applications, particularly those that my have been written with a different security mindset? By understanding more about the roots of software, the strategy to secure its use will be better informed.

Why ask the question? Unless you know about the origins of software, it is very hard to put together a plan to ascertain its security. Knowing who built it provides a resource to ask about the way in which it was built. Knowing about its components provides information to use in testing it or researching testing done by others.

What is the plan for ongoing governance?

Governance, similarly, has a variety of depths of detail and application, particularly in IT. For our purposes, the questions can be limited. How will the software be updated? Who will make those decisions? What is the process to initiate or approve a change? New software in any environment, and even established software in a dynamic environment, will face frequent opportunities and requirements to change. Understanding the models through which those changes are considered, approved, and delivered enables organizations to measure and manage their own risk from flux in the software, and in any collateral instability introduced to dependent systems.

Why ask the question? Instability = Insecurity. Haphazard or non-existent governance leads to more frequent changes, less testing time for the solution in place, and to inevitable discontinuities if the software is a component of a larger system. Weak governance also increases the opportunities and likelihood of malicious coding behavior by simply increasing the chaos during the software delivery process.

What does the software do with data?

Data is at the root of almost every application's function and purpose. Whether it exists to generate data, to gather it, or to analyze it, data is not only central to the application, it is often the prime target for an attacker. For that reason, there are multiple facets to consider. What kinds of data does the application gather, where does it come from, and how does it enter the system? Once the data has entered the system, does it get stored, and is it stored with appropriate protection of privacy and integrity? If the data ever moves between components of the system or between multiple systems, is it appropriately protected by the software for privacy and integrity? Does the system restrict access to the data, and is access control sufficiently granular to permit only authorized individuals to enter into the system? Each of these questions naturally results in a series of more technical and specific questions about the behavior of the application, but requiring answers to these high-level queries will mean that these will not be ignored.

Why ask the question? Data is central to the smartness of the Smart Grid, and its protection is expected by subscribers, is in many cases mandated by regulation, and is certainly necessary to ensure reliable operation of the Smart Grid.

How has the software been tested?

The testing of software, particularly for security issues, is still a developing field. There are a variety of approaches and mechanisms, each with their own strengths and deficiencies. What testing has been done, and on what components? What approaches were used, and with what results? Have all components been considered for security issues prior to their inclusion, and how were they vetted prior to selection?

Why ask the question? Understanding the testing process for the software can uncover blindspots to some sets of security issues, and can also identify weaknesses in methodology that can indicate systemic problems from the provider. If the testing ignores a specific area, like data storage or access control, then that lack of attention raises the likelihood that there could have been a similar lack of focus during its construction. Testing has many facets, and security must be among them.

These questions are intended to be a very brief introduction to some of the underlying and quite concrete issues that must be considered during the Grid's evolution to a Smart Grid. In time, each of these areas must be expanded into multiple levels of detail, but for now, this is a start. It is the start of generating more informed awareness, and of describing the types and amount of data that is required to feel secure during the adoption of new Smart Grid technologies.

In return, though, having those answers will certainly bring more confidence, more security, and more opportunity for success in the new Smart Grid.

Image kudos to flickr ( http://www.flickr.com/photos/marcobellucci/ / CC BY 2.0 )

And Then There was None

News from the Smart Grid Investment Grant program

Early Birds win again! Looks like the interest and enthusiasm for Smart Grid Programs has rapidly outstripped even the Government's own $3.4B largess. In an amendment dated September 21, the DOE announced that:

The Department of Energy has received a significant number of high quality applications and our review continues. The dollar value of applications far exceeds the funding available under this Funding Opportunity Announcement. As a result, Phase III is canceled.

and

Given the facts cited above, the Department may decide to cancel Phase II following final selection decisions made on applications currently under review.

So, what was intended to be a three phase investment program in new approaches to energy and grid management has become at best a two-phase program, and likely a single shot of stimulus into the Grid. Taking the amendment on its face, that the dollar value of applications already received far exceeds the funding available, we can conclude:

In the planned Phase I application period, running from the initial solicitation date of June 25th, 2009, to August 6th, 2009, there were requests for grants FAR EXCEEDING $3.4B. This means that, on average, the DOE received grant requests FAR EXCEEDING $113M every business day of the Phase I application period.

Each of these applications was expected to include many things, not least among them a well-articulated security plan. You will remember, from the cyber security requirements description:

Submitted Project Plans are also required to include a section on the technical approach to cyber security. Cyber security should be addressed in every phase of the engineering lifecycle of the project, including design and procurement, installation and commissioning, and the ability to provide ongoing maintenance and support. Cyber security solutions should be comprehensive and capable of being extended or upgraded in response to changes to the threat or technological environment.

Yikes. And more specifically must include:

• A summary of the cyber security risks and how they will be mitigated at each stage of the lifecycle (focusing on vulnerabilities and impact).
• A summary of the cyber security criteria utilized for vendor and device selection.
• A summary of the relevant cyber security standards and/or best practices that will be followed.
• A summary of how the project will support emerging smart grid cyber security standards.

In 20ish years of working in security, I have seldom found an organization that could create this level of cyber security detail within six months for an existing system, much less create it in 30 business days for a brand new project.

The infusion of SGIG capital has definitely gotten things moving, but we should all hang on. This looks to be a bumpy ride.

( Photo thanks to http://www.flickr.com/photos/pleasantpointinn/ / CC BY 2.0 )

Surge Protection: The New Smart Grid Data Challenge

As has often been written, the advancements of the Smart Grid are founded in information. Data is used to inform consumption, to make rates more dynamic, and to enable the next-generation power prosumer. In reading a recent piece on potentially mandated Smart Metering in the UK, the Telegraph raises the issue of data handling relative to today's data management. In short strokes, 44 million homes were typically measured twice a year, making for 88 million entries for data. In the new system, every home is measured twice a day, meaning that those 88 million entries have now become over 32 billion. Now this sounds like a lot, and let's quickly look at the new challenges that arise for organizations seeing this kind of increase:

Data Center Expansion

The types and volume of data associated with Smart Grid use will mean a new need to bring Internet-style data centers into the complex mesh of Utility control systems

Data Organization and Retention

With Time of Use pricing and user charge recovery for power generated, a sizable subset of this data will no longer be simply transient and used in the aggregate. Individual elements will need to be captured and tagged for later retrieval over whatever period is chosen by regulators as appropriate for looking back.

Data Privacy

While there may be dubious benefit to stealing the private data from individual citizen's Smart Meters, it is naive to think that privacy concerns will not find their way into regulation, meaning that data will need as well, to be partitioned when needed longer term, destroyed when transient, and never left in an unknown state.

I led with the UK piece, because it does a relatively non-threatening analysis of data gathering trends from a Smarter Grid.

The US Smart Grid, however, has a series of challenges that expand on this by many times. Back in May, Beth Pariseau did a piece on Smart Grid storage for SearchStorageChannel.com where she interviewed a variety of players, including Austin Energy's CIO, Andres Carvallo. The data usage trends described are nothing short of mind-boggling.

In the Austin Energy data, for phase one of the roll-out which included 500,000 meters, the increase in yearly data storage went from 20TB to 200TB, with disaster recovery redundancy. This is for 15 minute sampling, and first stage (appears to be largely home-oriented) integration. Ignoring smaller sampling frequencies (resulting in much higher data storage) necessary for some Smart Grid functionality, this presents a model of about 400 MB per meter per year. ( 200,000,000,000,000/500,000 ).

While this sounds mind-numbing, there is substantiation (and a reasonably close ratio) in the same piece, this from Pacific Gas and Electric, who added 1.2PB of memory (and growing) to support 700,000 meters, or over 170MB per meter per year. (This was sampling only twice per day).

What conclusions can we draw from all of this?

Massive Data is about to swamp existing infrastructure, requiring some hard thinking about how to architect, secure, segment, and deploy, the data centers that will accommodate it.

There is striking variability in the amount of data that organizations are expecting, seeing, and preparing for. Work is needed on what information should be gathered, what needs to be stored long-term, what needs to be tagged with user information, and what needs to be treated as private.

This is a new area for providers. The storage, record keeping, and maintenance of all of this data, particularly that which needs to be help for longer regulated periods, is unlikely to be a current function of the provider budget and functional organization. The steps to rationalize this area financially is critically important. Any plan to advance smart metering should include these costs in justification or grant request.

Every new idea for the Smart Grid, particularly those in the soft grid investment space must detail the additional burden they are likely to place on providers from a data acquisition, data management perspective.

Like so much of our economy, these advancements are changing the Grid from a Power economy to a Data and Power economy. To survive and thrive these new requirements must be considered. In the medium and long term, those organizations which consider, and then capitalize on, all of this data acquisition, will find themselves in a much better position to add services, ensure satisfaction levels, and find new ways to make the Smart Grid even Smarter.

[ And by the Way: In their August 2009 report on "Assessment of Demand Response and Advanced Metering", FERC presented a partial scenario (80M meters) and a full deployment scenario (140M meters) by 2019. Assuming that we feel comfortable in the midrange of the data descriptions used earlier, this would imply the need for the creation of infrastructures necessary to organize and manage roughly 100PB of information within the next ten years. Good luck to us all. ]

(SmartGrid diagram courtesy of US D.O.E).