60 Minutes Sounds Grid Security Alarm

Hat tip to my classmate and former Discovery Channel Powrtalk colleague Chris Davis for alerting me to the show that aired tonight. The popular news journal interviews former Director of National Intelligence (DNI) Mike McConnell, FBI Cyber Division Assistant Director Shawn Henry and others. It begins with cyber crime in the DOD world, goes through some real-world financial services industry examples, and concludes with conviction that the computers that run the Grid have been seriously compromised and that there's little the US government has been able to do to make private operators close out their vulnerabilities.

Remember, the subject here is the current Grid, the pre-cursor to the future Smart Grid, which will bring with it new types of additional abilities but also better ways of isolating some of them when necessary. The segment is called "Sabotaging the System" and you can watch it in its entirety right here, right now ... after a brief commercial, that is.


Watch CBS News Videos Online

Smart Grid Intro for CSO's

Having come to the Smart Grid Security discussion from the Security side of the equation, I have for years spoken at the highlight events, whether RSA, Gartner ITExpo, etc. This spring, when asked to present at CSI, I thought it would be a good opportunity that we could use to begin to bridge that IT and Utility security gap that Andy has written a fair amount on.

As such, last week I presented the following deck at the CSI IT show at the Gaylord National conference center, and it was meant to give just a taste of the Smart Grid to traditional IT security professionals, and to give some security information and guideposts to any utility folks that were there.

It turned out that we had representatives of both groups in the audience, and I have had several requests for the materials, mainly because these people wanted to begin the process of informing their own colleagues and managers. Be aware that it is intentionally light, it touches a few of the areas that are important, but it is by no means supposed to be an education on Smart Grid Security. It is more like the free chapter you would get if a book existed on the topic. Hopefully it was enough to energize some of these people who self-selected into the room and who are at least aware that there is a grid that is Smart, and there are security issues that may plague it.

Here is the deck. Please feel free to share it, and to generate a more aware population wherever you are. Andy and I expect to launch a version with voice-over in the next few weeks, so stay tuned for a truly simple way to get people to understand more about the nature of some of the challenges of securing the Smart Grid.

NERC Grid Security Update: On the Lookout for a New Order

This article is a bit jumbled, but it does communicate the gravity with which NERC CSO Michael Assante approaches cyber threats to the national grid. Quite simply, he views the threats to the grid and emerging Smart Grid to be something beyond what we've ever faced before. This from a recent panel appearance:

There was a known security rule set in the Cold War. We knew and expected behaviors. We could calculate escalation. We took this into any account when we planned any action. When cyber defenses and communications entered the military, it was a force multiplier. We appreciated what it gave us. What we didn't realize was that cyber would be the thing that destroyed the rules of order.

That last line really got my attention. We are just beginning to learn the new rules. But you have to be careful and alert. So many experts from other domains giving advice about how to secure the Smart Grid these days, pretending they understand what it's ultimately going to look like. When in fact, these are still the early days and, given the pace of technological change we've witnessed in recent years and decades, the Smart Grid of 2020 will look quite different than we imagine it today. Like Assante and NERC, all of us "good guys" need to make ourselves ready for what's coming.

Photo Credit: US Army on Flickr

Seriously - A Surge

A couple of weeks ago, I took a look at the data provided by the teams at PGE and Austin Energy, combined it with data provided by DOE, and I arrived at the conclusion that the Smart Grid will create a glut of information that the utilities had best begin planning for, because it could easily swamp both the utility and the networks that are expected to carry it.

Unsurprisingly, there was a fair amount of interest in both the conclusions I had reached and in the substantiation of the data I had used. Some of the inquiries were pretty straightforward. My thanks to Editor Katie Fehrenbacher from Earth2Tech for her thoughtful questioning and for introducing me to some equally reasonable experts from the IEEE.

Others were less open to the concept, and there were two main objections to the data. The first was based in existing utility practices. This line of questioning had within it the expectation that a meter read would only contain basic information about the identity of the power meter, the timestamp, and the meter reading itself. Were that the case, it would be possible that the data would be in a paltry range, around 14 bytes per read, resulting in a belief that such a small amount of data would never amount to anything like the avalanche I had described in the piece. The second objection was that there was little likelihood that such data was going to be stored for long, meaning, I guess, that we could design the system as though it had never arrived at all. Many of the questions came from individuals with strong/long histories in utilities, so I felt it my responsibility to validate, again, my data.

While I consider myself to be relatively well-versed on the core of these topics, it is the nature of this blog to focus on my expectations of the future based on information provided elsewhere, by others more directly in the path of the Smart Grid. That said, credibility is a big deal for us, and I decided to go back to Austin Energy, and understand better the reality of the situation from the folks who are actually doing the job, and who are considering these concerns as fundamental parts of their planning for successfully serving their clients on the new grid in the years to come. Andy and I called Andres Carvallo and Karl R. Rábago at Austin Energy, and they generously agreed to help us understand the world and the Smart Grid that they are planning for.

Smarter Grid versus Simpler Meter-Reading

One of the first things I learned was the richness of information gathering and interactivity that these gentlemen expect to coax from the new grid infrastructure. While time, location, and power used are at the heart of a meter read, there is much more to be learned. Investment in the Smart Grid would have a maximum return when the savings were more than a human reader's footwear and gasoline. Some examples are:

Device Health Information

By watching for varying temperature, periods since outage, battery power, heartbeat, and other meter variables, it is possible to better predict and recover from any failures that may happen.

Real Time Monotoring

As has happened historically with most new technologies, it can be expected that people yearning for more data will only be satisfied by that which is most current. It is unlikely to happen in the general population immediately, but history shows us that it is likely that such a real time monitoring feed may be in demand almost immediately, as customers recognize that there is now more information through which they can better manage their energy.

Energy Services Provision trumps Energy Provision Services

There are doubtless going to be additional requirements from the newly informed and empowered customer base for functionality that is logically delivered by the provider. This was a real eye opener for me, that Power Providers are now actively thinking about services that they can offer over the new and smarter infrastructure. Things like profiled energy use: "I am going away, manage my power." or "There is a spike in prices, manage me down by 10%", or "I only want to use power that is generated from renewable resources." These all require data, new interfaces, and a channel overwhich all of the control and monitoring information can be passed. Winners in the new market will be finding ways to capitalize on the need for energy-related services, and will not limit their investment to further driving down the costs of simply providing energy.

Networking Overhead

Given the complexity, regularity, and importance of this data, it is clear that a protocol (Like IP) will probably be adopted to package up and send all of this information in a payload to central systems for analysis, aggregation, storage, and action. Protocols carry their own overhead in terms of describing their content, sources, destinations, etc. None of this is free from the perspective of the systems carrying or storing the data.

Other Factors

We are only just beginning to see the potential for Smart Grid and Soft Grid enablers, leading me to believe that even my estimates are very likely to be low, particularly as we clamor for realtime monitoring and data analysis.

Based on all of this, it looks like the numbers are far from a simple 14 Byte read, and are more likely in the range given by Andres of 4K to 16K per reading. If we estimate the maximum case, the numbers are even higher than I had referenced in the earlier article. Let's not think about real-time (the numbers are mind-numbing), but instead look at a simple check every 5 minutes. 12 (reads/hr) X 24 (hrs/day) X (365 days/yr) X 16K (Bytes/read) yields roughly 1.7GB/meter/year. Multiply that by the number of meters (pick your own scope), and I think the challenge is clear. For more reality, take that number and multiply by 5 for readings every minute, or by 300 for readings every second. That's big.

So, is this a problem because the data going to cause the Smart Grid to explode like a flawed radiator hose in July? I don't think so. I think that time has proven that technical advancement has always helped us stay ahead of crushing data or processing burdens by decreasing computing and memory costs. This has allowed us to paper over our excesses with iron and silicon.

No, this is a problem because rushed, tactical, and incremental hardware adds will not make that data secure. It has to be expected that as organizations run out of room for data, they will simply rush to add more. Caught in a flood of data, the pressures for survival and successful operation will naturally trump any meaningful consideration of rearchitecting data storage for adequate and appropriate security.

This planning (and budgeting) needs to happen now. As Andres said on our call, "You cannot simply build an airplane for passengers who are 5'6" tall and weigh 140, because you can guess that your average passenger, much less your larger passengers, will simply not fit, because they are not that small." In other words, you need to plan for what you can reasonably expect, not for what will make your life, your business, or your CFO, ecstatic.

I think that this is the final insight. For firms that are seeing the Smart Grid as an enabler for cost-savings by transferring operations onto an IP infrastructure, or a wireless metering system, there is little reason to be concerned with a data glut.

For those who recognize that the Smart Grid and the coming Soft Grid will need data, and will need security, and will likely grow to fill whatever space is available, the call is clear. Plan for an avalanche, for a flood. Create systems and segregations that will allow for managing these flows reliably. Characterize what must come through, and what can be dropped, along the way to the back end. Do all of those things and the current systems will be fine, the next systems will not choke, and the ultimate end state will be similar enough to what has been planned to ensure stability, quality, and cost-effective services to all who connect to the grid.

The data surge is coming, and you can either surf it, or be pounded by it. You certainly will not be able to ignore it.

Image Thanks to:

http://www.flickr.com/photos/ndave/ / CC BY 2.0

Notes from 2009 Control Systems Cyber Security Conference

We first posted on Joe Weiss's work back in July following a presentation he gave to the Air Force. Now here's a great review of a significant annual conference, one that focuses not on IT or internet security in a Smart Grid context, but rather on the security issues related to the millions of control systems that automate the Grid. This is Joe's summary:

The Ninth Control Systems Cyber Security Conference was hosted by Applied Control Solutions (ACS) the week of October 19 in Bethesda, MD. The festivities started Monday morning with parallel activities. A tour was arranged of Washington Suburban Sanitary Commission’s Rock Creek water treatment facility. In parallel, the initial meeting of the ISA Nuclear Plant Cyber Security Joint Working Group was held.

The ACS Conference started Monday afternoon with two introductory sessions: Control Systems for the non-Control System Engineer and IT for the Control Systems Engineer.The Conference began in earnest Tuesday with approximately 110 attendees from US and international electric and water utilities, chemical and oil/gas companies, IT and control system suppliers and consultants, universities, and US and international government agencies. The Conference is called Control Systems Cyber Security is that industrial control systems are common across multiple industries. The agenda can be found at www.realtimeacs.com.

There were two hacking demonstrations of control systems and several discussions on control system cyber vulnerabilities. There was also a discussion on the need for technical control system cyber security curriculum (policy programs exist). There were two keynotes: the Honorable Yvette Clarke (D-NY), Chairwoman of the Subcommittee on Emerging Threats, Cybersecurity, Science and Technology and member of the Intelligence, Information Sharing and Terrorism Risk Assessment Subcommittee provided the lunch keynote. Whitfield Diffie gave the evening keynote and discussed control system cyber security issues from the Tuesday’s session.

There were four different sessions on actual control system cyber incidents – none of which was public! In one session, two control system engineers from two different utilities that have control systems from every major supplier discussed their recent control system cyber incidents – one had his plant shutdown. A couple interesting side notes were that existing control system logging are adequate to identify control system incidents and their control system suppliers weren’t of much help when it came to providing control system cyber security support. Both engineers felt it was so important to share information they attended the Conference on their own nickel. This is in marked contrast to the utility and industry leadership who didn’t think this conference was important enough to attend even though many were based in Washington. Wednesday evening, the Honorable James Langevin gave the evening keynote. Congressman Langevin felt this was so important he spent 30-45 minutes after his presentation answering questions and talking to the attendees.

We received a summary of government activities including legislative efforts on cyber security, cyber security activities by the Nuclear Regulatory Commission, efforts on-going at the Bonneville Power Administration using the NIST Framework, and non-governmental activities in certification and cyber incident collection. Also got a very interesting presentation on cyber security legal issues and a discussion of the Russian cyber attack on Estonia.

On the last day, NIST held training sessions on two very relevant NIST standards:

-- SP 800-53 - Recommended Security Controls for Federal Information Systems - including those for the Bulk Power System
-- SP 800-82 - Guide to Industrial Control Systems (ICS) Security provides guidance on securing Industrial Control Systems (ICS), including Supervisory Control and Data Acquisition (SCADA) systems, Distributed Control Systems (DCS), and other control system configurations

Electric Car Conundrum: V2G a Smart Grid Blessing or Curse?

Initially arriving in the US in low volume in late 2010, the addition of thousands and later millions of cars with 5-10 KW battery packs drawing power from (and sometimes giving back to) the grid is cast as both a positive and a negative, depending on your point of view.

On the positive side, as this article says, high performance, deep cycle lithium ion and lithium air batteries en mass may be the energy storage solution the industry has been searching for. Here's an example starring Duke:

Duke Energy committed to an electric vehicle future when it committed with the FPL Group to buy 10,000 electric vehicles and plug-in hybrids in the coming decade, as they upgrade their fleets. The energy storage in these vehicles could eliminate the need for peaking plants and enable the expanded use of renewable energy. Duke Energy’s electric vehicle future may save billions in future power plant investments.

Sounds good, but others worry, here, that local electrical infrastructure can barely handle the additional iPods and iPhones it's had to deal with lately. Adding clusters of electric cars charging at approximately the same time each evening might break the camel's back in many neighborhoods. According to Peter Darbee, the CEO of Pacific Gas & Electric:

A high concentration of plug-in electric vehicles poses a serious challenge to utilities. Plug-in electric cars could draw electricity equivalent the amount needed to run one home, or up to three homes in certain places. You can see if you have three or five electric cars arrive in a neighborhood, you're going to overload the local circuits, and that will lead to blackouts. So we see it as an opportunity but we also see it as a challenge of significant proportions.

We all know how neighbors like to mimic and compete with each other (have you seen the Halloween decorations next door !?). One electric car will beget two will beget ten or twenty. Scheduling software will help, but much depends on fast this goes, and how close to edge local circuit gear is at the outset.

Nissan Leaf photo credit: Wikimedia Commons

Why Smart Grid Security is about so much more than Smart Grid Security

Frankly, after having worked in the security industry for ten years now, there are days when I feel like I've had my fill. At a recent Smart Grid conference I sometimes wished I could focus solely on cool new functionality like Vehicles to Grid (V2G) for instance.

But then I remember that what got me into energy was a passion for renewable technologies. A passion which was refreshed last week when by futurist Ray Kurzweil, speaking at MIT's Enterprise Forum, reminded us that solar energy technology is now on an exponential growth curve, just seven evolutionary steps away from reaching price/performance parity with the cheapest fossil fuels: coal.

Well guess what? If solar was ready for prime time today the grid couldn't handle it. Wouldn't that be depressing? We've got a few more years to get the grid ready by making it smarter, more flexible and able to handle the intermittent aspect of solar and wind.

So we need this Smart Grid to be up, running and well along its nation-wide implementation in the next 5-10 years. During this period, security consultancies like IOActive and Wurldtech will continue to tell us that the Smart Grid is a house of cards, ready to be blown over by casual hackers, let alone organized criminal gangs, non-state combatants, and nation states determined to harm the USA. There will be times when we'll second guess what we're doing, when we'll question whether NERC's vigilance and NIST's Smart Grid security standards are up to the task, whether key industry players are putting enough thought and effort into the security elements of their solutions, or are simply trying to sell us what was "secure enough" in the past.

Ultimately, the Smart Grid must both appear to be secure (so we continue to invest in and deploy it) and actually be secure, so it doesn't suffer a knock-out blow in its formative days. All this security stuff, while potentially tedious to some, is an acknowledgement that a secure Smart Grid is a mandatory prerequisite to our nation's energy future, nothing less.

Photo: Wikimedia Commons

Military Planning For Prolongued Outages via Smart/Micro Grid Technologies

While the US Department of Defense has many unique tasks and requirements, many of its concerns and challenges re: the current grid, Smart Grid and Smart Grid security are common to all enterprises. Much of what motivates DOD motivates others, including:

• Desire for continuous operation and continuous service to customers by keeping core systems running during (possibly prolonged) power outages impacting local communities
• Energy efficiency savings via reduction in electricity and fossil fuel usage
• Demonstrating proactive/compliance measures vis-a-vis climate change and the increased use of renewable energy sources
• Maintaining confidentiality/privacy of data and doing all of the above is a safe and secure manner

So along those lines, here's an excerpt from a recent post on the DOD Energy Blog on the so-called "brittle grid" problem I believe you'll find interesting:

Eighteen months have now passed since the public release the "Defense Science Board Task Force Report on Energy" This is from the section called "Managing Risks to Installations":

For various reasons, the grid has far less margin today than in earlier years between capacity and demand. The level of spare parts kept in inventory has declined, and spare parts are often co-located with their operational counterparts putting both at risk from a single act. In some cases, industrial capacity to produce critical spares is extremely limited, available only from overseas sources and very slow and difficult to transport due to physical size.

In many cases, installations have not distinguished between critical and non-critical loads when configuring backup power systems, leaving critical missions competing with non-essential loads for power. The Task Force finds that separating critical from noncritical loads is an important first step toward improving the resilience of critical missions using existing backup sources in the event of commercial power outage. The confluence of these trends, namely increased critical load demand, decreased resilience of commercial power, inadequacy of backup generators, and lack of transformer spares in sufficient numbers to enable quick repair, create an unacceptably high risk to our national security from a long-term interruption of commercial power.

Granted, DOD's not the only organization with these concerns ... and the obligation to plan accordingly. Hospitals, police & fire, essential services, etc. all have to think this way. DOD is exploring campus microgrid strategies (including on-site power generation, energy management and energy storage systems, and more) to allow bases to "island" themselves away from commercial grid infrastructure.

The technology is getting to the point where this approach is becoming just as feasible for industry. We'll be investigating further and will post the results right here.

Photo Credit: Kristen Holden on Flickr

Smart Grid Security: Answers in Questions

Over the past year, Andy and I have written about the risks and opportunities in the growing software sector of the Smart Grid Marketplace. We have described the space, some of the firms, the investment, and what we are seeing for security in those organizations we speak with. In response, and I think with genuine interest, we've been asked what we are worried about, and in turn, what recommendations would we specifically make to individuals who are either investing in these solutions, or who are actually building them.

In the recent NIST strategy and requirement recommendations release, there was a substantial body of information to be reviewed, and this post is not meant to summarize or to supplant those results (obviously). This is a relatively lightweight view of heavy duty and high-level considerations in software as a critical element in the development of the Smart Grid. It is a practical list of questions that organizations should be able to answer before they commit to software that will either replace or broker their interactions with the Smart Grid.

What is the software's provenance?

Provenance is a term that gets thrown around a lot, but I use it to express the idea of origin. Where did the software come from? Who made it? What was it made from? While absolute provenance is difficult or impossible to ascertain, these answers can help to guide risk awareness and management. Is it new software built for me? Is it existing software that has run similar systems elsewhere? Is it a new solution from an existing partner, or revision 0.9 from a start-up? Is it built from the ground up, or does it contain elements of legacy applications, particularly those that my have been written with a different security mindset? By understanding more about the roots of software, the strategy to secure its use will be better informed.

Why ask the question? Unless you know about the origins of software, it is very hard to put together a plan to ascertain its security. Knowing who built it provides a resource to ask about the way in which it was built. Knowing about its components provides information to use in testing it or researching testing done by others.

What is the plan for ongoing governance?

Governance, similarly, has a variety of depths of detail and application, particularly in IT. For our purposes, the questions can be limited. How will the software be updated? Who will make those decisions? What is the process to initiate or approve a change? New software in any environment, and even established software in a dynamic environment, will face frequent opportunities and requirements to change. Understanding the models through which those changes are considered, approved, and delivered enables organizations to measure and manage their own risk from flux in the software, and in any collateral instability introduced to dependent systems.

Why ask the question? Instability = Insecurity. Haphazard or non-existent governance leads to more frequent changes, less testing time for the solution in place, and to inevitable discontinuities if the software is a component of a larger system. Weak governance also increases the opportunities and likelihood of malicious coding behavior by simply increasing the chaos during the software delivery process.

What does the software do with data?

Data is at the root of almost every application's function and purpose. Whether it exists to generate data, to gather it, or to analyze it, data is not only central to the application, it is often the prime target for an attacker. For that reason, there are multiple facets to consider. What kinds of data does the application gather, where does it come from, and how does it enter the system? Once the data has entered the system, does it get stored, and is it stored with appropriate protection of privacy and integrity? If the data ever moves between components of the system or between multiple systems, is it appropriately protected by the software for privacy and integrity? Does the system restrict access to the data, and is access control sufficiently granular to permit only authorized individuals to enter into the system? Each of these questions naturally results in a series of more technical and specific questions about the behavior of the application, but requiring answers to these high-level queries will mean that these will not be ignored.

Why ask the question? Data is central to the smartness of the Smart Grid, and its protection is expected by subscribers, is in many cases mandated by regulation, and is certainly necessary to ensure reliable operation of the Smart Grid.

How has the software been tested?

The testing of software, particularly for security issues, is still a developing field. There are a variety of approaches and mechanisms, each with their own strengths and deficiencies. What testing has been done, and on what components? What approaches were used, and with what results? Have all components been considered for security issues prior to their inclusion, and how were they vetted prior to selection?

Why ask the question? Understanding the testing process for the software can uncover blindspots to some sets of security issues, and can also identify weaknesses in methodology that can indicate systemic problems from the provider. If the testing ignores a specific area, like data storage or access control, then that lack of attention raises the likelihood that there could have been a similar lack of focus during its construction. Testing has many facets, and security must be among them.

These questions are intended to be a very brief introduction to some of the underlying and quite concrete issues that must be considered during the Grid's evolution to a Smart Grid. In time, each of these areas must be expanded into multiple levels of detail, but for now, this is a start. It is the start of generating more informed awareness, and of describing the types and amount of data that is required to feel secure during the adoption of new Smart Grid technologies.

In return, though, having those answers will certainly bring more confidence, more security, and more opportunity for success in the new Smart Grid.

Image kudos to flickr ( http://www.flickr.com/photos/marcobellucci/ / CC BY 2.0 )

Islands No More

In a bracing report from Australia, we learn from the Sydney Morning Herald that Integral Energy was inundated with a virus on non-critical systems, but at such a penetration level that they chose to rebuild 1000 desktop machines to eliminate the problem before it "spreads to the machines controlling the power grid."

The security consultant interviewed in the piece, Chris Gatford from HackLabs mentions that in his experience there is ample evidence that the networks may well have been connected despite the efforts of the utility to separate them. This is particularly problematic, I am sure, because there are not only power control systems to worry about, but also online payment, user account management, and other relatively advanced functions at Integral Energy.

His comments seemed familiar to me, so I went back through my notes, all the way to a report from the team at Riptech in 2001 ( Bought by Symantec) called " Understanding SCADA System Security Vulnerabilities ", where the authors describe a very similar disconnect between assumptions and reality in these internal networks:

MISCONCEPTION #1 – “The SCADA system resides on a physically separate, standalone network.”

Most SCADA systems were originally built before and often separate from other corporate networks. As a result, IT managers typically operate on the assumption that these systems cannot be accessed through corporate networks or from remote access points. Unfortunately, this belief is usually fallacious.

In reality, SCADA networks and corporate IT systems are often bridged as a result of two key changes in information management practices. First, the demand for remote access computing has encouraged many utilities to establish connections to the SCADA system that enable SCADA engineers to monitor and control the system from points on the corporate network. Second, many utilities have added connections between corporate networks and SCADA networks in order to allow corporate decision makers to obtain instant access to critical data about the status of their operational systems. Often, these connections are implemented without a full understanding of the corresponding security risks. In fact, the security strategy for utility corporate network infrastructures rarely accounts for the fact that access to these systems might allow unauthorized access and control of SCADA systems.

MISCONCEPTION #2 – “Connections between SCADA systems and other corporate networks are protected by strong access controls.”

Many of the interconnections between corporate networks and SCADA systems require the integration of systems with different communications standards. The result is often an infrastructure that is engineered to move data successfully between two unique systems. Due to the complexity of integrating disparate systems, network engineers often fail to address the added burden of accounting for security risks.

As a result, access controls designed to protect SCADA systems from unauthorized access through corporate networks are usually minimal, which is largely attributable to the fact that network managers often overlook key access points connecting these networks. Although the strategic use of internal firewalls and intrusion detection systems (IDS), coupled with strong password policies, is highly recommended, few utilities protect all entry points to the SCADA system in this manner.

I think that the team at Integral Energy knows this as well. Their actions show that they felt it necessary to take serious and disruptive measures to eradicate a virus outbreak before it jeopardized the entire infrastructure. Their willingness to speak of it publicly also provides a real service to those of us who are considering the impacts of the introduction of multitudes of new systems and new access points into those same networks.

One sees allusions to the concept of separate networks, with various properties, in existing regulation, CIP descriptions, etc. If we can agree that there are likely to be unintended cross-overs between these systems and their populations, then we must also agree to stop considering the artifice of disjoint networks as being anything but an anachronism, and treat the security of each network with the same rigor and protective approaches, regardless of our faith in its isolation from sources of corruption.

( Image Thanks to Flickr: http://www.flickr.com/photos/cloudsoup/ / CC BY 2.0 )

CSOs and the Smart Grid

Setting the Stage
So you're an executive in charge of security at a medium, large or very large organization. You might be called Chief Security Officer (CSO) or Chief Information Security Officer (CISO) or maybe VP or Director of Security. You most likely report to the Corporate CIO, or you're in a business division and you and your boss plug into a General Manager. You decide, with blessing from above for the big stuff, the following:

• Where you'll get the biggest risk reduction (or compliance) bang for your limited budget buck
• Which technologies get purchased and implemented
• Which vendors will augment your in-house security team, and,
• Corporate security policies, and how to best promulgate them to other parts of the co. for whom security is at best an annoyance, and at worst, something to be openly resisted

Yours is a world of risk management as you oversee the wellness (e.g., integrity, reliability, performance, compliance) of your IT, networking and communications systems (and true CSOs own physical security as well). In addition to managing for threats coming from those directions, in recent years, new threat vectors from service oriented architectures (SOA), Web 2.0 and cloud computing have kept you busy.

Hey, Have you Heard of Smart Grid?
So how much time do you spend on future threats? If you have heard of the Smart Grid, and if you've been reading up on it, then you probably don't need to read further here. You're in the top 10% of your class and get a star on your forehead. If however, you're like some CSOs I've talked with who claim to have never heard the term, then this is your wake up call. There has been little written to guide CSOs through the early stages of preparing to protect their organizations in a world where the power systems they rely on look increasingly like the Internet (and in some cases are the Internet!).

How is it different from today's electrical grid? For starters, it's a 2 x 2-way system. Thanks to advanced metering infrastructure (AMI) and net metering, electricity and usage information will flow from generators to consumers and back again. The total amount of information, which in the beginning will be substantial, will quickly become enormous. Data protection will be crucial, and demand management strategies which could save your organization significant money, could also get you in trouble fast. Water and other services will also be impacted for better and worse. In short, for each benefit a Smarter Grid will bring an organization, there is a commensurate risk to mitigate. And it's your job to know (and plan for) this.

Only CSOs at utilities see this world first hand, and even in the energy and utilities vertical, many of those CSOs work in a balkanized world where their policies touch only IT, and the "rubber meets the road" part of their company, field operations, doesn't want to anything to do with them.

So most CSOs are left to infer what they need to know from a mountain of Smart Grid articles and a multiplicity of Smart Grid conferences. My guess is once they've poked a toe into these confusing waters one time, they soon find their time better spent working on present challenges. The appropriate information has not yet been boiled down for this most important enterprise leadership function ... one that could and would do the right things, proactively, if it had the right knowledge to work with.

CSO Info Resources Not Too Helpful Yet
Where do CSOs turn for expert guidance and to learn from what their successful peers are doing? Why, the journals and other news sources that serve them. Yet from the looks of these two articles from CSO Online and the CSO Roundtable, all they're getting is high level introductory material that in no way considers how Smart Grid trends intersect with CSOs' particular responsibilities. I would advise these orgs to get on the ball: it's their job to see over the horizon and around corners to give their readers the info they need to protect their companies ... and their jobs.

No Answers Yet, But Here are a Few Starter Questions
NIST and other standards bodies are working around the clock to bring appropriate and helpful security standards to this new domain and you don't have to know them yet (however, for a sneak peak, here's the most recent draft edition of Smart Grid Cyber Security Strategy and Requirements from NIST). So much is still in flux that doing too much at present might be as bad as doing too little. But that doesn't mean you shouldn't start getting your head around this challenge and thinking through some of the scenarios. Here's a handful:

1. Supply Chain - Similar to Y2K preparation in some respects, even if you get your house in order for the arrival of the Smart Grid, if the companies yours depends on are not prepared it may affect you. It's time to talk about this with them.
2. Vehicle Fleet - More choices are coming, including hybrid electric, full electric, natural gas, etc. Are you thinking about the challenges and opportunities that present themselves in beginning to move away from gasoline and diesel? What are the security implications of your enterprise depending on these new transportation technologies?
3. Local utilities - All utilities are under guidance to prepare for Smart Grid standards and technologies. What are your providers doing in your different locations and how soon will their actions begin to affect you? What do you need to do to not get blind sided?
4. Smart Grid pilots - With stimulus help from the Fed Gov, pilots are springing up everywhere. Related to number 3 above, are there any pilots going on you could participate in? While this might take resources away from more proximate concerns, the education might more than pay for the time invested.
5. Centralized policy and control - If yours is a geographically distributed operation, to what extent will you attempt to define and enforce Smart Grid-related security policy in a uniform way, versus allowing disparate facilities and offices to determine their own best approaches?

That's all for now, but on each of these and many more there's a ton of thinking and planning to be done. While in most cases it's too early to implement, it's certainly not too early to imagine.

And there there was none...

News from the Smart Grid Investment Grant program

Early Birds win again! Looks like the interest and enthusiasm for Smart Grid Programs has rapidly outstripped even the Government's own $3.4B largess. In an amendment dated September 21, the DOE announced that:

The Department of Energy has received a significant number of high quality applications and our review continues. The dollar value of applications far exceeds the funding available under this Funding Opportunity Announcement. As a result, Phase III is canceled.

and

Given the facts cited above, the Department may decide to cancel Phase II following final selection decisions made on applications currently under review.

So, what was intended to be a three phase investment program in new approaches to energy and grid management has become at best a two-phase program, and likely a single shot of stimulus into the Grid. Taking the amendment on its face, that the dollar value of applications already received far exceeds the funding available, we can conclude:

In the planned Phase I application period, running from the initial solicitation date of June 25th, 2009, to August 6th, 2009, there were requests for grants FAR EXCEEDING $3.4B. This means that, on average, the DOE received grant requests FAR EXCEEDING $113M every business day of the Phase I application period.

Each of these applications was expected to include many things, not least among them a well-articulated security plan. You will remember, from the cyber security requirements description:

Submitted Project Plans are also required to include a section on the technical approach to cyber security. Cyber security should be addressed in every phase of the engineering lifecycle of the project, including design and procurement, installation and commissioning, and the ability to provide ongoing maintenance and support. Cyber security solutions should be comprehensive and capable of being extended or upgraded in response to changes to the threat or technological environment.

Yikes. And more specifically must include:

• A summary of the cyber security risks and how they will be mitigated at each stage of the lifecycle (focusing on vulnerabilities and impact).


• A summary of the cyber security criteria utilized for vendor and device selection.


• A summary of the relevant cyber security standards and/or best practices that will be followed.


• A summary of how the project will support emerging smart grid cyber security standards.

In 20ish years of working in security, I have seldom found an organization that could create this level of cyber security detail within six months for an existing system, much less create it in 30 business days for a brand new project.

The infusion of SGIG capital has definitely gotten things moving, but we should all hang on. This looks to be a bumpy ride.

( Photo thanks to http://www.flickr.com/photos/pleasantpointinn/ / CC BY 2.0 )

Conference Alert: SCADA and Control Systems Security Summit

Just the facts, M'am:

• What: a gathering of like minded individuals intent on learning more about threats posed by systems not well known or understood by IT and Internet security crowd. Similar to mainframes in that they were originally conceived to run in an utterly disconnected world, early SCADA implementations (many still performing critical roles today) were designed with little thought to access control and authentication. Yet SCADA and other types of electronic control systems are as much a part of the emerging Smart Grid as will be the latest hardware and software offerings from CISCO, GE and SilverSpring. Because they have remained relatively obscure outside the operational utilities domain, developing strategies to secure them is now the order of the day as development of the Smart Grid leaps ahead.
• When: 7-9 December 2009
• Who: DHS, DOE, NERC and NIST will be there, joined by others from government and industry
• Where: Washington DC (venue to be named)
• How: For more info and to register, click here

Preparatory Resources

• The Open SCADA Security Project
• The Center for SCADA Security - Sandia National Labs
• Wikipedia's SCADA entry
• Blackhat Powerpoint: SCADA Security and Terrorism (PDF)

Photo courtesy of: Ian David Blum on Flickr

Surge Protection: The New Smart Grid Data Challenge

As has often been written, the advancements of the Smart Grid are founded in information. Data is used to inform consumption, to make rates more dynamic, and to enable the next-generation power prosumer. In reading a recent piece on potentially mandated Smart Metering in the UK, the Telegraph raises the issue of data handling relative to today's data management. In short strokes, 44 million homes were typically measured twice a year, making for 88 million entries for data. In the new system, every home is measured twice a day, meaning that those 88 million entries have now become over 32 billion. Now this sounds like a lot, and let's quickly look at the new challenges that arise for organizations seeing this kind of increase:

Data Center Expansion

The types and volume of data associated with Smart Grid use will mean a new need to bring Internet-style data centers into the complex mesh of Utility control systems

Data Organization and Retention

With Time of Use pricing and user charge recovery for power generated, a sizable subset of this data will no longer be simply transient and used in the aggregate. Individual elements will need to be captured and tagged for later retrieval over whatever period is chosen by regulators as appropriate for looking back.

Data Privacy

While there may be dubious benefit to stealing the private data from individual citizen's Smart Meters, it is naive to think that privacy concerns will not find their way into regulation, meaning that data will need as well, to be partitioned when needed longer term, destroyed when transient, and never left in an unknown state.

I led with the UK piece, because it does a relatively non-threatening analysis of data gathering trends from a Smarter Grid.

The US Smart Grid, however, has a series of challenges that expand on this by many times. Back in May, Beth Pariseau did a piece on Smart Grid storage for SearchStorageChannel.com where she interviewed a variety of players, including Austin Energy's CIO, Andres Carvallo. The data usage trends described are nothing short of mind-boggling.

In the Austin Energy data, for phase one of the roll-out which included 500,000 meters, the increase in yearly data storage went from 20TB to 200TB, with disaster recovery redundancy. This is for 15 minute sampling, and first stage (appears to be largely home-oriented) integration. Ignoring smaller sampling frequencies (resulting in much higher data storage) necessary for some Smart Grid functionality, this presents a model of about 400 MB per meter per year. ( 200,000,000,000,000/500,000 ).

While this sounds mind-numbing, there is substantiation (and a reasonably close ratio) in the same piece, this from Pacific Gas and Electric, who added 1.2PB of memory (and growing) to support 700,000 meters, or over 170MB per meter per year. (This was sampling only twice per day).

What conclusions can we draw from all of this?

Massive Data is about to swamp existing infrastructure, requiring some hard thinking about how to architect, secure, segment, and deploy, the data centers that will accommodate it.

There is striking variability in the amount of data that organizations are expecting, seeing, and preparing for. Work is needed on what information should be gathered, what needs to be stored long-term, what needs to be tagged with user information, and what needs to be treated as private.

This is a new area for providers. The storage, record keeping, and maintenance of all of this data, particularly that which needs to be help for longer regulated periods, is unlikely to be a current function of the provider budget and functional organization. The steps to rationalize this area financially is critically important. Any plan to advance smart metering should include these costs in justification or grant request.

Every new idea for the Smart Grid, particularly those in the soft grid investment space must detail the additional burden they are likely to place on providers from a data acquisition, data management perspective.

Like so much of our economy, these advancements are changing the Grid from a Power economy to a Data and Power economy. To survive and thrive these new requirements must be considered. In the medium and long term, those organizations which consider, and then capitalize on, all of this data acquisition, will find themselves in a much better position to add services, ensure satisfaction levels, and find new ways to make the Smart Grid even Smarter.

[ And by the Way: In their August 2009 report on "Assessment of Demand Response and Advanced Metering", FERC presented a partial scenario (80M meters) and a full deployment scenario (140M meters) by 2019. Assuming that we feel comfortable in the midrange of the data descriptions used earlier, this would imply the need for the creation of infrastructures necessary to organize and manage roughly 100PB of information within the next ten years. Good luck to us all. ]

(SmartGrid diagram courtesy of US D.O.E).

Smart Money on the Smart Grid?

The Venture Capital business is a brutal one. The process can appear to be like Darwinian Natural Selection on speed, as venture dollars drive multiple entrants into an emerging space in hopes that as the weak are weeded out, their own investments will survive and thrive. At worst, there is cold comfort in the fact that the compressed timeframes will help them to identify their own latent failures more quickly so that they can cut their losses.

I was discussing this mechanism of investment acceleration yesterday with a colleague who does some later stage (profitable stable companies) cleantech investing, and he was remarking on the Klondike Gold Rush-like movement by some Venture firms into cleantech, and into Smart Grid startups particularly. The Smart Grid boom, in his view, is the first and closest child of the Internet boom. Biotech (another area of large investment) has been a very different model, with its long lead-times and eight or nine digit price tags. I had to agree. So much of the Smart Grid is looking like Soft Grid, and successful startups are bringing in management software, efficiency software, upgraded infrastructure and communications. It really does feel like the early days of the Internet, where technology startups faced relatively low costs to enter into a new market, where the existing infrastructure needed evolutionary enhancements pretty regularly, and where the established players were unlikely to step outside of the box to make those changes happen. In the Internet era it was telecommunications companies who provided both the enabling backbone and the lack of groundbreaking higher-level innovation that created the opportunity for entrepreneurs. Now it is the utilities' turn.

In sheer numbers, the investment is amazing. The Cleantech Group reported yesterday that the cleantech sector accounted for 27% of venture investing in the second quarter, which shows how enormous this wave is, totaling over $1.5B for that period. They also reported that many of the largest investments went to firms which were also leveraging Government funding dollars. So, what does this foretell?

It foretells a glut of new technologies, advancements, approaches, and failures. Larger organizations will be able to invest their own time and money on comprehending and capitalizing on the meaty part of the wave, while these new entrants stay at the crest, and either find the ride or the rocks as the industry approaches the first winnowing stages. Ordinarily, this kind of furious growth yields rapid progress, and markets and nations benefit from the rapid determination of good and stable solutions. Whether this will work for the Smart Grid is yet to be seen. The nature of power, and the economics of traditional utility finances can make this tumult and its turbulence a disaster.

Venture investors expect to see failures, their models assume them. The Government investors expect to see, well, whatever. The government is funding policy through technology.

Power providers and customers, however, can not be tolerant of too much instability, and so we hope that adoption of these technologies will remain proactive but prudent, regardless of the "energy" that all this investment may put into the grid.


Image courtesy of flickr :

http://www.flickr.com/photos/flashpackinglife/ / CC BY 2.0

Smart Grid Startups and Security: Round 2 from GridWeek

This post picks up up where we left off last week during GridWeek 2009, examining patterns that emerged from our talks with Smart Grid startup booth reps. Jack and I noticed that few of the startups are staffed with a dedicated security professional, and had tasked an existing player (CTO, Application Engineer, etc.) with the responsibility. Other exhibiting companies (Cap Gemini, Cisco, GE, ABB, Siemens, etc.) had booths too, but it seemed crazy to ask them if they employed dedicated security pro's, because of course they do, both for their internal operations as well as for their client-facing products and services.

Back to the startups. As you know, we like to pose questions ... so here are a few:

• In a domain where security rigor is universally regarded as essential, how much security thinking is going on within these start-ups, and how long will the present level be enough?
• Put another way, when you're a small but growing company in the Smart Grid software or hardware space, how long can you hold out before adding a full time security professional to your team?
• Do you hire a security staffer once your development team reaches a certain size, say a headcount of ten, or should you put the security pro in place up front to help define the development process before you start writing real code?
• Given the amount of innovation required in most of these companies, how reasonable is it to expect that the CTO can juggle all the technology balls he/she is responsible for, and do a good job on security tasks (which will often seem like a distraction) at the same time?

I liken this to the situation that faced large and medium companies approximately ten years ago, when it was becoming clear that as they embraced the Internet for new capabilities, they were inadvertently bringing a whole host of new risks and vulnerabilities on board. This is from CSO Magazine in 2001 on why to hire Corporate Security Officer and what he or she can do for you:

... a core responsibility of the CSO will be vulnerability assessment and risk management. Therefore the CSO should report to the COO or CEO. After all, the CSO will evaluate the technology environment and audit the security measures implemented by the CIO. It is in the company's and the CIO's best interest to have the CSO perceived as an impartial assessor of the technology environment instead of a possible rubber stamp .... Think of the CSO as the head of quality assurance for security.

In startup-land, there is no real need for C-level titles beyond CEO. But ignoring the titles, the functional benefits of a dedicated security staffer are clear, no matter what they're called. In other markets we have seen them labeled: Security Architects, Information Security Officers, Security Managers, Security Officers, Information Security Managers, etc. Depending on the offering and the market strategy, there's a mix of roles that these folks may fill, including ensuring the security of the company (its systems, processes and people) and the security characteristics of its products; hardware, software or both.

Hyperbole aside, we all know that the Smart Grid is an area of growing and inevitable security risk. If I'm a utility, and as such am a prospective new customer for a startup, and I'm held accountable to the highest security standards by those who regulate me, I'm going to be damned sure that I put prospective vendors through the ringer before bringing their technology in house. And if I'm a startup, while having a qualified security person on my staff is no silver bullet, our guess is they'll be more than worth their salary as the regulators press their security cases and the utilities/customers get more and more savvy about risk.

Smart Grid: Greener but no Greenfield

It is good to see the attention that the new NIST draft directives for the Smart Grid are getting in the press. Ordinarily, this type of draft release is not interesting enough to the general public to merit any real press, and ends up being a conversational target to the few who arrive interested in the space. Any mainstream attention comes much later in the cycle, as affected parties either applaud or complain.

One impression that I would like to correct is that the Smart Grid itself, and therefore, the challenges of Smart Grid security, is something being developed from scratch.

In Federal Computer Week, Bill Jackson calls out the following:

Deployment of a Smart Grid offers a greenfield opportunity because the existing grid, parts of which are 50 years old or older, was not designed to support alternative energy sources such as wind and solar power, and the two-way flow of energy and data. But this wholesale upgrade also makes it imperative that security be built in now, because the grid lifecycle is measured in decades rather than years, as it is for much of the rest of our information infrastructure. Equipment being designed for deployment now might not be replaced for decades.

There are so many capabilities within the Smart Grid that are new, and there is so much investment going into it, that it is completely understandable to conceive of the Smart Grid as the "new" grid, as opposed to the evolution of the "old" grid. The Smart Grid as a replacement is a misperception that we have seen often in our work on evangelizing smart grid security. The Smart Grid is not a greenfield, not a replacement infrastructure, and most definitely not a new grid. We always have to remember that the Smart Grid is a new way of leveraging, stabilizing, advancing, and enhancing, the OLD Grid.

The billions that have been made available through the Smart Grid Investment Grant Program, the additional billions that are pouring into development of renewables, transmission and distribution advancements, PEV's, and storage, are only a small fraction of the total picture when the nation's power infrastructure is viewed in its eventual entirety. As a result, when we are considering the security of the Smart Grid, we must always consider (as the NIST work does ) the existing grid. Whether we work to create more secure means to connect to it, or to actual revisit the older technologies and improve their protections, those challenges will likely be the most pressing, and the most complicated, that we need to solve.

What's on First: Insights in NIST's 1st Draft

Never will one mistake the complexities of the Smart Grid, and of undertaking the improvement of its protections, for a straightforward task in security and engineering. It presents an Augean stable of issues, and NIST has waded in with a legion of contributors, to first make sense of it all, and then to start handing out shovels.

In the first draft of their analysis, announced during Grid Week, Annabelle Lee and team have created a dense, but readable tome, numbering some 236 pages at present, entitled, Smart Grid Cyber Security Strategy and Requirements. I encourage you to read it, either on its own, or as an adjunct to the more general draft of NIST's Smart Grid guidance on interoperability. In the event that you are interested in some sense of where the emphasis was put, and are more engaged by the higher level issues of focus and risk, I did a bit of data reduction and reached some pretty interesting, if unintended ( and definitely scientifically questionable ) conclusions.

One of the techniques that NIST uses in creating a better means of discussing cyber security for the grid is to categorize the areas of likely risk and their impacts. This is very helpful, as there are myriad instances of connection between systems within the Smart Grid and some higher level abstraction helps to make the issues digestible. These 15 categories are defined within the document, as are the potential impacts to them ( Confidentiality, Integrity, Availablity ), and their levels ( High, Medium, Low ) using established definitions from the venerable FIPS Publication 199. This exercise, and the tables contained within the draft, permits a reader with a spreadsheet (me) to draw two conclusions about priorities in Smart Grid Security.

Conclusion 1: Integrity is the most important attribute

In reviewing the definitions of the categories, and the impact that was most highly rated, the answer was unanimous. Integrity, as opposed to confidentiality or availability, was rated as a "High", in every single instance. (NB: In categories 10-12, there is a range of impact level, but each included "High" for Integrity ) Whether because corrupted data could degrade the operation of the grid, or because it could be used to defraud customers, suppliers, or the market, integrity showed up as the Number 1 concern, with no exceptions, according to the NIST results.

Conclusion 2: B2B and control system connections are Riskiest

There were only two categories which ranked with "Highs" across the board, for Confidentiality, Integrity, and Availability, and both could be described as connections between different kinds of systems. The categories are numbers 6 and 7, relating to B2B and control/non-control systems respectively. This feels right intuitively, but it also represents a potential area of rapid growth in both members and risk for the Smart Grid. It describes the connections that are both most likely to be leveraged by new entrants and which are most likely to use either IP, or actual Internet-based, networking. As we have written about before, the Soft Grid is probably the next big area of investment and expansion, as organizations form to leverage the new infrastructure and public enthusiasm to deliver more interesting and likely complicated applications.

In the remarkable depth and detail of the NIST report, it is very possible to become discouraged by the references to "hundreds of standards" and by the complexity of the diagrams it contains. It is important to have a sense for where to start, as the NIST process will necessarily be a lengthy one, and time ( and Smart Grid Investment Grants ) are waiting for no-one. If, as contributors to the Smart Grid, or as advisors to organizations which seek to connect, we can help them to focus on these few issues from the start, it is possible that they will be far better prepared for the new documents, threats, and requirements that are certain to follow.

New Smart Grid Standards are Out - Complexity is In

Earth2tech, as usual, does a great job of reducing complexity into consumable pieces. In this case, the subject is the new NIST Smart Grid standards draft released today (PDF here). Far from appearing as an afterthought or not at all, Cyber Security issues are front and center in the executive summary and are described in some detail on pages 71-79 of the document. Also significant is that control system security, which some feel is getting short shrift in this process, is given substantial attention and weighting, with a list of applicable security-related standards on page 79.

As the diagram above illustrates, however, complexity itself may ultimately become the biggest security challenge. The best human minds, augmented with the most sophisticated tools, will have a monumental task keeping track of the myriad threat vectors and security controls deployed to defend against them. As one of the GridWeek conference panelists said on Tuesday, acknowledging complexity's potential risks, "we hope that we can move towards simplicity at some time in the future." Yeah, that'll be easy.

Diagram: NIST

GridWeek:Startups and Security

We are dealing with some raw data here, but one thing jumps out after speaking with a dozen or so Smart Grid start-ups in the Exhibition area: few of the new startups employ a security professional. Some are flatfooted when asked about how and if their product is secured, some are more assured. But even in the latter case the answer tends to be that "the CTO handles security."

There is little doubt that the CTO's of these organizations are highly skilled and technically very deep. But, given the nature of many of these cutting edge providers, they are much more likely to be schooled, and buried, in issues directly related to the functionality that they are attempting to provide. Security will necessarily be put relatively low on the priority list, particularly in the absence of any specific requirements or breaches as identified by others external to the company.

One phenomenon we noticed was that the impetus for people even having a name to assign to security is derived from more consistent utility behaviors in the area. Almost to a person, the interviews which we performed resulted in a statement about how the security resource was identified because the utilities demanded that there be a person with security responsibility in the vendor providers. Kudos to the utilities, and here's hoping that the security person in name will grow into a security resource in fact, as the requirements of their position be more fully articulated going forward.

This blog maintains that the great Smart Grid project could fail, or fail to thrive, largely based on its ability to get security reasonably right, and because adoption will be partially determined by industry and public perception of its safety. The finding that young Smart Grid companies, as represented here, have not prioritized security action, versus titling and responsibility, is a concern. Some of the firms like Itron and Gridpoint have taken time to articulate their security strategy, and that is definitely a step forward, but there is much work to be done by all, in describing, and demanding, a consistent security emphasis going forward.

We will continue to reach out to the CTO's in the coming weeks to better understand their familiarity and efforts in security, and will bring that to you here.