As we stated in a previous post called the Smart Grid Security Confidence Game, the large-scale Smart Grid build-out that waits just beyond the lessons learned in the SGIG pilots isn't going to happen if the utilities, the regulators and the users don't trust the security controls.
Tuesday, December 29, 2009
As we stated in a previous post called the Smart Grid Security Confidence Game, the large-scale Smart Grid build-out that waits just beyond the lessons learned in the SGIG pilots isn't going to happen if the utilities, the regulators and the users don't trust the security controls.
Tuesday, December 22, 2009
Such is the case with a new DOE organization, the Advanced Research Projects Agency-Energy (ARPA-E), which came to life just this year and has been given a $400 million boost to get itself and its first bunch of projects off the ground. ARPA-E is not about incremental improvements in energy science; no, it focuses exclusively on high risk, bet the farm, swing for the fences, change the world energy technologies.
One thing you can say for sure: whether ARPA-E advances technologies that benefit the grid directly or finds ways to greatly increase the capabilities of renewable power generation or storage, it all grows the Smart Grid one way or another. By the way, Majumdar came across as warm, brilliant, determined and 100% sincere. I for one am rooting big time for him and his world changers.
Photo Credit: Lawrence Berkeley National Lab
Wednesday, December 16, 2009
For my own bootcamp/bootstrap education, I have been consuming first, "Securing SCADA Systems", by Kurtz, and then "Cybersecurity for Scada Systems", by Shaw. But these are probably more dense than is neccessary for those who are looking for a more readily consumable description of challenges and recommendations. In trying to find that level of content for you, our valued readers, I stumbled upon course material from some extremely helpful folk at Idaho National Labs. Don't let the nuclear tone and front page announcement of graphite testing fool you, there is a four hour course and an eight hour course here, and they have a raft of good content inside.
One of the slides was especially excellent, and I present it here by way of both introduction to our newer readers, and as validation for those who have, with us, been working to highlight and hopefully increase the level of IT/Cyber security discussions that are surrounding the Smart Grid. Here it is:
It is hard for anyone to deny that the worlds of modern internetworked information technology and of the existing SCADA-driven grid are merging. That said, this diagram, which while using information derived in 2007, shows the manifest disconnect in security practices and priorities between the two communities as they operate today. This data is directly in support of much of what we are seeing, and clearly reinforces some recent feedback we have gotten. In moderating a panel at last week's IQPC Scada and Control System Security Summit, Andy and I got a question relating to the new burdens that the Smart Grid was placing on the existing grid for things such as Antivirus/Anti-malware software, Intrusion Detection/Protection, and more. It became clear that these arguably baseline technologies were not yet deployed broadly within the utility community, and that the introduction of the Smart Grid was causing people to finally start to view them as important, if not required. This was not to say that they wanted it, or that they felt comfortable that they could accommodate the additional load on their systems, but the perceived connectivity of the Smart Grid is causing them to consider this, for the first time, as a priority.
Coming from an IT perspective, this was surprising. According to members of the audience, the Windows XP Service Pack 2 BIOS security change that occurred years ago had disrupted multiple SCADA systems, as have more recent instances of corruption and malware, as reported in the media. Considering that, it is almost unthinkable that basic security technologies have not been deployed, even if only in response to the unacceptable vulnerability conditions. Unthinkable or not, we need to start thinking hard about it, because clearly it is happening.
Some of the reasons for this lack of progress are well-known. The overtaxed nature of both the systems and the individuals charged with their operation, the proprietary nature of some of this infrastructure, and the cost-averse nature of many utility commissions all conspire to a preference for the pretense that these are isolated, and therefore inviolable networks.
This slide points out, with vivid clarity drawn from analysis of these control systems, how far there is to go, and how different the drivers and fears of the organizations are from those who typically and aggressively pursue security at a proactive or holistic level.
We are just now beginning to recognize and recommend the need for a balanced approach to IT and Cyber security in the new and existing Grids. The work done at INL is extremely helpful in creating a bridge between the existing and incoming Grid and Smart Grid communities, and I recommend that you take the time to examine it to the purpose of expanding the group that can speak in, and be concerned with, the colliding challenges of internetworked computing, security, expertise, stability, and staffing.
Sunday, December 13, 2009
That NIST is involved there is no doubt. See this from the Energy Independence and Security Act of 2007: NIST has "primary responsibility to coordinate development of a framework that includes protocols and model standards for information management to achieve interoperability of smart grid devices and systems…". In point of fact, NIST's role in the process is to be the honest broker between warring tribes of standards bodies, lobbyists and advocates of all stripes. As the above slide makes plain, each home is a bloody standards battleground. This is not easy work for NIST, or any of the innumerable stakeholders.
But to repeat: NIST is not making the standards. It's an open process and that's a job for all of us. Just so you know.
Slide Credit: "Repowering the Nation: Setting Standards for the Smart Grid" presented at MIT
Monday, December 7, 2009
- Cutting electricity to homes and businesses
- Overburdening the grid
- Causing brown-outs
- Having smart-grid devices attack the grid itself
- Getting free service
- Undermining confidence
It seems to me like there's a vacuum out there where only pundits dwell. Wouldn't it be excellent if some of the utilities could be more forward leaning and get out in front of this issue?!? Messages on the thorough measures they're taking to protect the grid and their customers might help. But so far they're not saying much, and until they do, folks like Winkler are the ones exhibiting their confident predictions that rough days are in store for the young Smart Grid and that the utilities are playing marketing defense, not working to shore up their security and privacy weak spots with vigor.
The power companies don't like it when people say things like this, as they showed by attacking me after my previous exposé of power-grid vulnerabilities. So far, though, every claim I made has been proved correct by documented attacks or government reports. Sadly, I know that I will be proved right once again.I hope he's wrong, but you know what they say about hope. Hope is not a strategy ... for protecting the Smart Grid. Let's hear it utilities execs ... we know you're busy working these issues, but please take a minute to tell the public what you are doing to prove Winkler wrong (Ira, not Henry).
Photo Credit: Igor Bespamyatnov @ Flickr
Sunday, November 29, 2009
- Will the fully deployed Smart Grid have borders?
- In North America, will the Smart Grid eventually transcend the current regional topology of Regional Transmission Operators (RTO's) and Independent Systems Operators (ISO's)?
- While the electrons that constitute my emails transit the continent (heck, most of the globe) with ease, the same cannot be said for the electrons currently bringing my monitor to life. Will the Smart Grid change this?
- Is there anything the US can learn from early international efforts in Europe, where Germany was a first mover?
[The US has] got to realize that the North American grid is international, it's interconnected, it's integrated. Consultations, cooperation between governmental authorities on both sides of the border is going to be imperative, otherwise you won't be able to ensure system reliability and you'll probably undermine system reliability.I realize my understanding of these issues is likely simplistic. Yet the ability to quickly "island off" healthy portions of the grid from unhealthy ones is key functionality every region and every nation is shooting for. But islanding should be an emergency response, not the square one status quo inside the US or among close allies.
Tuesday, November 24, 2009
"Smart Grid security” is most often discussed in the terms of national security — a hacker develops a worm that can jump across smart meters and black out neighborhoods, for example, or can make a generator blow up remotely. Privacy — keeping personal information in the hands of the consumer and away from advertisers, the utility or any other third party — is an entirely different concern that utilities have to be prepared for with the build-out of the Smart Grid.
Consent and Choice: The organization must describe the choices available to individuals and obtain explicit consent if possible, or implied consent when this is not feasible, with respect to the collection, use and disclosure of their personal information.
Thursday, November 19, 2009
What Once was Old is Old Again
It is not news that components of SCADA systems can be older and have been designed for reliability and stability on mainly protected networks populated with trusted people. In discussing his motivations for researching the attacker, and for calling the authorities, McGrew cites his current doctoral research in information security, particularly in SCADA security. When he discovered that the attacker had installed botnet software on a hospital HVAC system, his level of urgency shot up. He feared that even modest corruption of that system could cause real danger to patients, at one point referring to SCADA systems of the type as a sort of "rickety ensemble" of old and new pieces, which could not be expected to withstand much tinkering.
He is not alone in this expectation. In a presentation back in 2007, delivered at HITBSecConf2007 Malaysia, called "Hacking Scada", other statements supported this fear, including the fact that ordinary anti-virus software could be expected to crash many SCADA systems due to the increased load, and that simple utlities like "ping" had been shown to bring those assets down.
As an IT person coming to utilities, I had expected vulnerability, but did not expect the real fragility in these important systems.HMI by DIY
I was also surprised to learn that many of the front-ends ( HMI or Human-Machine Interface systems) of these newer SCADA implementations are actually created on-site. Think of it as a Do-It-Yourself graphical user interface. This is necessary, in as much as most of them are actually doing extremely custom things. The presence of different sensors, different arrangements, different control structures, demand that the interface itself be created in a way that is very much tailored to the environment that is actually going to be managed.
I learned this while researching the new importance of the internet protocol and even web-oriented interfaces, as components in the HMI interfaces of these systems. Packages actually ship with IDEs (Integrated Development Environments) containing libraries and widgets necessary to create useful, functional, and hopefully intuitive representations of the complex system of sensors, RTU's, PLC's, and more. It is not clear how seriously security is regarded in the creation of these custom interfaces, or how simple it can be to enable security controls available through the IDE's. It appears that there exist few standards and fewer tools relating to their certification.
Getting Warm in Here?
As it was with attacks and breaches in the early days of the Internet, the facts surrounding the means of identifying the actual attack and attacker are discouraging.
Based on the reporting from the hospital...which existed in Texas...in summer...it's hot there...the air conditioning system had failed multiple times, and they didn't check for, or find, the remote control software on the HVAC system. Instead, a researcher hundreds of miles away had gotten an unrelated message from a hacker, did some research, and discovered from pictures of the HMI screens that the system had been corrupted.
Admittedly, information security may be relatively new to the traditional SCADA user, but there needs to be better tooling, or better integrity assurance, or just better education and awareness to make some information security analysis more standard.
IT Hacking Ignorance
It could be that the most dangerous reality of this article could be summed up in the uninformed actions of the attacker, and the reactions of others to his arrest. The malicious software that was delivered through a USB drive into an exposed USB port, was a botnet, remote control software, and the attacker was planning a "massive" denial of service attack from all of his controlled machines.
I think it is pretty clear that this guy did not know how unstable this system would become, or how important HVAC is in a hospital in Texas. Operating room environmentals, pharmaceutical storage temperatures, patient recuperation, are all intimately connected to those systems. It is literally life and death. It is hard to imagine from the descriptions of the attacker and his attack that he construed his incursion as being as dangerous as it was. Similarly, the ignorance of many of the comments on his arrest miss this entirely, presenting their view of the attack as being that he "hacked an air conditioner or something".
Whether it be in the minds of the internal resources who do not think about information security and an HVAC system, or external attackers who do not understand the complexity, seriousness, and importance of these newly interconnected SCADA systems, the fundamental disconnect on action and effect need to be made much more visible.
The reliance of SCADA-enabled systems like HVAC on their actual software, and the reliance of the utilities and customers on these SCADA systems is a connection that is becoming obvious as the Smart Grid expands the number and the exposure of these systems to all.
Monday, November 16, 2009
MAD, or Mutually Assured Destruction, is a Cold War-era term which neatly describes why nuclear deterrence works and has so far kept our planet from being reduced to a glowing ember from a massive thermonuclear exchange. You are still relaxed I see ... that's good.
Last week we posted a link to, and a couple comments on, an alarming 60 Minutes episode on cyber security risks to critical US infrastructure. It described how vulnerable the US is to computer hackers and used examples from DOD, the financial sector and the electrical grid. An additional level of disturbing detail was provided by former Director of National Intelligence (DNI) Mike McConnell, who said he's certain that foreign code is resident on national grid systems. Our own anecdotal experience with critical systems in other industries corroborates this. In hacker lingo: we are "owned."
Still relaxed? You should be, because there's ample evidence, in the 60 Minutes material and elsewhere, that even as we are heavily targeted, we also have substantial penetration of our potential adversaries' systems. Hence, the resemblance to MAD. I'm making this comparison preemptively before some journalist or K Street analyst does, because I think it's worth laying a few of the cards on the table and thinking about this in a non-alarmist fashion. Here's a short list of attributes to compare and contrast:
- Once underway, nuclear war is for keeps: you're either launching nukes or you're not
- Though some once believed in it, "limited nuclear war" is generally considered unlikely
- While we work to make missile defense a reality, our best defense against nuclear attack has been a good offense (see: deterrence)
- Damage from nuclear exchanges is usually believed to be catastrophic
- With missiles and bombers heading our way, it's fairly easy to discern the origin of attack, and hence, the attacker
- There are currently 9 countries listed as nuclear nations. Others seek to join this group, but it's expensive, complicated and time consuming, not to mention dangerous and sometimes destabilizing
- Probes and attacks are happening all the time by multiple parties and damage of various degrees is being absorbed by all involved
- All cyber war is, by definition, limited
- Our best defenses are multi-layered, resilient and constantly evolving
- Damage is infinitely variable in severity and often hard to detect
- Often cannot identify attack origin or attacker
- Any country, organization or individual with access to the Internet can be an attacker
- Unlike with nukes, where deterrence between nuclear nations has worked so far, no one is fully deterred from experimenting with and sometimes wielding cyber weapons against our grid or other critical US infrastructure systems. Most nations do, however, seem deterred from launching massive cyber attacks on us and others ... and life and commerce go on
- International crime gangs and other non-state bad actors abide by completely different rule sets from those described above. Deterrence means much less to them, so we've got to continue to bring our cyber security "A game" to the Smart Grid build out as well as to the rest of our critical national infrastructure
- Understanding and accepting that all sides "own" other systems conjures up the alternative title to the Cold War classic "Dr. Strangelove," which was "How I Learned to Stop Worrying and Love the Bomb." I'm not suggesting you begin loving cyber risks to the grid or Smart Grid; just want you to worry a little less if the 60 Minutes piece has rendered you sleepless or immobile. Clearly we’ve got work to do, but as NASA and the NY Times said today, we’re not going to die tomorrow or the day after tomorrow
- For a somewhat more detailed, balanced examination of cyber risks to the grid, see University of Minnesota's Dr. Massoud Amin's short paper "Electricity Infrastructure Security", PDF downloadable here.
Thursday, November 12, 2009
"I am pleased to witness today the installation of the first smart meter for a PG&E customer," said Michael R. Peevey, president of the California Public Utilities Commission. "This technology will link the prices energy consumers pay to the costs of that energy in the wholesale market, empowering consumers with the information necessary to make sound energy choices. Research suggests that even modest levels of price sensitivity in the retail market can yield substantial benefits as customers decrease or shift their energy usage. These types of demand response programs are one of the best ways to meet the energy needs of California's growing population, as outlined in our Energy Action Plan."It is hard to know exactly when the honeymoon ended, whether it was when Bakersfield.com reported on a customer who found his power usage had tripled during a six-hour blackout, or at the town meeting in Fresno on October 20th which quickly became a unanimous indictment of Smart Meter-ing, or now in November, as a class-action suit has been filed against PG&E, asserting a variety of mistakes and misrepresentations. For those of us who have spent a fair amount of time researching the potential for advances derived from Smart Metering, these developments are disconcerting.
From a security perspective, there are two very important areas of guidance to take from these developments, and from the likely continuing negative perception of Smart Metering in some areas.
Integrity and Availability of Data
As we wrote here, and as others opined elsewhere, there is likely an abundance of information about to flood utilities. Some have rejected, or at least resisted, the idea that anything like high volume sampling would happen, and that aggregated data would be the more probable artifacts that utilities would store for billing and management. This suit and the ongoing outcry for justification of higher bills are exactly the reason why more detailed and regular metering information will need to be gathered and stored.
See, it is likely that these bills are actually accurate. As the commissioner stated at the outset, "modest levels of price sensitivity in the retail market can yield substantial benefits". Ok, so maybe the hot tar and chicken feathers are not necessarily a benefit, but they highlight a new awareness on the part of the consumers. It is surprising that this message of usage and contention for power has not been better absorbed by the public. Take an average citizen. They use power, like everybody else, from 8-6. Enter the Smart Grid, and the smart meters. In an attempt to incent off-peak usage, and to compensate for the increased cost of peak generation, power is more expensive from 8-6, and so the average consumer's bill, if they do not change their behaviors, is going to be higher. The smart meter only becomes an engine of positive financial impact for consumers when they figure out ways in which to really alter their power use to advantage the off-hour charges.
Until that happens, expect that there will be continuing challenges to the veracity of the smart meter data, and continuing scrutiny of the systems that collect and store it. This equals what we described in earlier posts, a need for lots of data, lots of governance of that data, and good security from authenticating the user to authorizing the billing.
Actual Smart Meter Opponents
Any publicly-perceived inequitable grab for cash by a business or utility can spawn a grass-roots movement in opposition. Ignoring the more fringe folks who bring you the youtube videos of jack-booted thugs monitoring your hot-tub to charge you with profligate energy spending, there are others who are more credibly mobilizing around this issue. An example is San Francisco-based TURN (Toward Utility Rate Normalization). With a 35 year history in utility consumer advocacy and activism, the have a new focus on the perceived inequity of a smart metering infrastructure that saves costs for utilities (better management, less truck-rolls, easier disconnects) while increasing the actual bills for consumers.
With group action, and organized effort, there comes increasing visibility and controversy around the issues, and there are likely to be more critical assessments made of Smart Metering infrastructures. This will naturally splash as well onto the overall Smart Grid approach of which smart meters are such an important part. With any such increase in visibility and controversy, individuals outside the credible groups may well begin to conspire to take more aggressive action, potentially creating a new wave of "hacktivism", with the focus in this cycle being the Grid. This will change the nature of the threat to the Smart Grid enormously, making it much more likely to experience the types of attacks that more typically plague governmental and military infrastructures.
Some of the Solution is in the Data
Many of the same constituencies who are actively opposing the Smart Meter evolution are also very much interested and involved in the promotion of more efficient energy usage and more integration of alternative sources. It is now the responsibility of the utilities to educate their customers about the actual dynamics of power and power pricing, to help them to better understand the choices that they will need to make.
For those utilities who have not yet begun to alter the finances of their customers through higher peak pricing, there is a cautionary tale here. It seems that it might well be worth 3-6 months of reporting on usage, with simulated billing and recommendations for changes, prior to actually instituting those changes. It would better showcase the insight provided by Smart Metering, would provide a sense of empowerment for the users, and would certainly eliminate some of what seems to be a sense of blindsiding on the part of the consumer.
Image thanks to the whimsical stylings of Roger Wood
Tuesday, November 10, 2009
- Smart Meter Roll-out
- In the FERC's Demand-Response Paper from September of 2009, the number of Smart Meters currently implemented is roughly 8 million. Looking at the total of the specifically identified smart meters implemented as a result of successful SGIG requests, that number is now funded to get to a total of 18 million with the SGIG funding. That means that the SGIG will carry smart meter deployment to more than 20% of the FERC demand response projection of 80 million meters by 2019. Let's hope that the meters are chosen correctly.
- Per-Meter Costs
There is enormous variability in the costs of the smart-meter roll-outs as described by the various grants. This is understandable in that the number of meters is only one criteria of many of these proposals. For some, these are an initial effort, for others they are scaling existing investment up. The meters, though, do loosely equate to the public involvement (connected by meters) that the SGIG is attempting to accelerate. As such the range and variety are worth noting.
- 79% of grants expect associated costs of < $500/meter
- 18% of grants expect associated costs of $500-$1000/meter
- 2% of grants expect associated costs of $1000-$2000/meter
- 1% of grants expect associated costs of >$2000/meter
The information is pretty scant in the released SGIG award documents, but there are some insights, if not actual conclusions, that can be drawn from it.
- Its about Usage
According to the rudimentary data that is provided, Smart Meter-related projects are consuming by far the largest section of SGIG funding, and at least 85% of the total investment (SGIG and Utility/Vendor) expected for these projects. There are mentions of accommodating other energy sources, but the projects seem pretty focused on how power is consumed, and how that consumption is measured, as opposed to how it will be generated and distributed.
- There is No Clear Standardization of Direction
While these grants are providing the impetus for some organizations to begin work on Smart Grid infrastructure, the sheer size of them make the investment much more about rapidly scaling that adoption. Given that, and given the need to maintain stability in power, the projects themselves seem to be surprisingly one-off's, each intending to validate or optimize one organization's view of the new generation of Grid. As an example of this, take a look at the wording provided for two projects in North Carolina, from Duke Energy and Progress Energy, respectively
[Duke Energy] Comprehensive grid modernization for Duke Energy’s Midwest electric system encompassing Ohio, Indiana, and Kentucky. Includes installing open, interoperable, two-way communications networks, deploying smart meters for 1.4 million customers, automating advanced distribution applications, developing dynamic pricing programs, and supporting the deployment of plug-in electric vehicles. Will also benefit customers in IN and OH. ($200,000,000 SGIG/$851,700,000 Total)and
[Progress Energy]Build a green Smart Grid virtual power plant through conservation, efficiency and advanced load shaping technologies, including installation of over 160,000 meters across its multi-state service area. Will also benefit customers in SC. ($200,000,000 SGIG/$520,000,000 Total)It is hard to think of projects of this magnitude as test beds
- Ready or Not, Here We Come
From a security perspective, this is a massive investment in expanding the exposed surface of the grid, and it will impact a new generation of underlying communications infrastructure. Most of the synopsis data includes things like two-way communications, interactivity, new networking infrastructure, etc. That is a wholesale shift for millions of customers, and we continue to hope that people are putting hard thought into it, because those dollars will be spent, and we will need to reconcile the security one way or another.
I guess that last conclusion that I draw is that this program also tells us that even in these small-ish numbers, the costs are huge. Through either market forces or another wave of government investment, getting to the FERC's "partial adoption" could easily cost another $15B of government funding on this route, and another $20-30B in private investment. The numbers to get to a fuller adoption are far higher. From a security perspective, all of this continues to point back to understanding what is necessary within the new infrastructure, and what acquisition guidelines should drive these enormous purchases, because it will be impossible to unwind this once it gets moving.
The SGIG has put fuel into a very powerful and creative technical engine within the energy industry, and like an automobile, that power is generating speed. As that speed builds up, we need to see similar emphasis on keeping the headlights on so we don't crash on these unfamiliar roads.
Sunday, November 8, 2009
Remember, the subject here is the current Grid, the pre-cursor to the future Smart Grid, which will bring with it new types of additional abilities but also better ways of isolating some of them when necessary. The segment is called "Sabotaging the System" and you can watch it in its entirety right here, right now ... after a brief commercial, that is.
Thursday, November 5, 2009
As such, last week I presented the following deck at the CSI IT show at the Gaylord National conference center, and it was meant to give just a taste of the Smart Grid to traditional IT security professionals, and to give some security information and guideposts to any utility folks that were there.
It turned out that we had representatives of both groups in the audience, and I have had several requests for the materials, mainly because these people wanted to begin the process of informing their own colleagues and managers. Be aware that it is intentionally light, it touches a few of the areas that are important, but it is by no means supposed to be an education on Smart Grid Security. It is more like the free chapter you would get if a book existed on the topic. Hopefully it was enough to energize some of these people who self-selected into the room and who are at least aware that there is a grid that is Smart, and there are security issues that may plague it.
Here is the deck. Please feel free to share it, and to generate a more aware population wherever you are. Andy and I expect to launch a version with voice-over in the next few weeks, so stay tuned for a truly simple way to get people to understand more about the nature of some of the challenges of securing the Smart Grid.
Wednesday, November 4, 2009
There was a known security rule set in the Cold War. We knew and expected behaviors. We could calculate escalation. We took this into any account when we planned any action. When cyber defenses and communications entered the military, it was a force multiplier. We appreciated what it gave us. What we didn't realize was that cyber would be the thing that destroyed the rules of order.That last line really got my attention. We are just beginning to learn the new rules. But you have to be careful and alert. So many experts from other domains giving advice about how to secure the Smart Grid these days, pretending they understand what it's ultimately going to look like. When in fact, these are still the early days and, given the pace of technological change we've witnessed in recent years and decades, the Smart Grid of 2020 will look quite different than we imagine it today. Like Assante and NERC, all of us "good guys" need to make ourselves ready for what's coming.
Photo Credit: US Army on Flickr
Monday, November 2, 2009
Unsurprisingly, there was a fair amount of interest in both the conclusions I had reached and in the substantiation of the data I had used. Some of the inquiries were pretty straightforward. My thanks to Editor Katie Fehrenbacher from Earth2Tech for her thoughtful questioning and for introducing me to some equally reasonable experts from the IEEE.
Others were less open to the concept, and there were two main objections to the data. The first was based in existing utility practices. This line of questioning had within it the expectation that a meter read would only contain basic information about the identity of the power meter, the timestamp, and the meter reading itself. Were that the case, it would be possible that the data would be in a paltry range, around 14 bytes per read, resulting in a belief that such a small amount of data would never amount to anything like the avalanche I had described in the piece. The second objection was that there was little likelihood that such data was going to be stored for long, meaning, I guess, that we could design the system as though it had never arrived at all. Many of the questions came from individuals with strong/long histories in utilities, so I felt it my responsibility to validate, again, my data.
While I consider myself to be relatively well-versed on the core of these topics, it is the nature of this blog to focus on my expectations of the future based on information provided elsewhere, by others more directly in the path of the Smart Grid. That said, credibility is a big deal for us, and I decided to go back to Austin Energy, and understand better the reality of the situation from the folks who are actually doing the job, and who are considering these concerns as fundamental parts of their planning for successfully serving their clients on the new grid in the years to come. Andy and I called Andres Carvallo and Karl R. Rábago at Austin Energy, and they generously agreed to help us understand the world and the Smart Grid that they are planning for.
Smarter Grid versus Simpler Meter-Reading
Device Health InformationBy watching for varying temperature, periods since outage, battery power, heartbeat, and other meter variables, it is possible to better predict and recover from any failures that may happen.
- Real Time Monotoring
- As has happened historically with most new technologies, it can be expected that people yearning for more data will only be satisfied by that which is most current. It is unlikely to happen in the general population immediately, but history shows us that it is likely that such a real time monitoring feed may be in demand almost immediately, as customers recognize that there is now more information through which they can better manage their energy.
- Energy Services Provision trumps Energy Provision Services
- There are doubtless going to be additional requirements from the newly informed and empowered customer base for functionality that is logically delivered by the provider. This was a real eye opener for me, that Power Providers are now actively thinking about services that they can offer over the new and smarter infrastructure. Things like profiled energy use: "I am going away, manage my power." or "There is a spike in prices, manage me down by 10%", or "I only want to use power that is generated from renewable resources." These all require data, new interfaces, and a channel overwhich all of the control and monitoring information can be passed. Winners in the new market will be finding ways to capitalize on the need for energy-related services, and will not limit their investment to further driving down the costs of simply providing energy.
- Networking Overhead
- Given the complexity, regularity, and importance of this data, it is clear that a protocol (Like IP) will probably be adopted to package up and send all of this information in a payload to central systems for analysis, aggregation, storage, and action. Protocols carry their own overhead in terms of describing their content, sources, destinations, etc. None of this is free from the perspective of the systems carrying or storing the data.
- Other Factors
- We are only just beginning to see the potential for Smart Grid and Soft Grid enablers, leading me to believe that even my estimates are very likely to be low, particularly as we clamor for realtime monitoring and data analysis.
So, is this a problem because the data going to cause the Smart Grid to explode like a flawed radiator hose in July? I don't think so. I think that time has proven that technical advancement has always helped us stay ahead of crushing data or processing burdens by decreasing computing and memory costs. This has allowed us to paper over our excesses with iron and silicon.
No, this is a problem because rushed, tactical, and incremental hardware adds will not make that data secure. It has to be expected that as organizations run out of room for data, they will simply rush to add more. Caught in a flood of data, the pressures for survival and successful operation will naturally trump any meaningful consideration of rearchitecting data storage for adequate and appropriate security.
This planning (and budgeting) needs to happen now. As Andres said on our call, "You cannot simply build an airplane for passengers who are 5'6" tall and weigh 140, because you can guess that your average passenger, much less your larger passengers, will simply not fit, because they are not that small." In other words, you need to plan for what you can reasonably expect, not for what will make your life, your business, or your CFO, ecstatic.
I think that this is the final insight. For firms that are seeing the Smart Grid as an enabler for cost-savings by transferring operations onto an IP infrastructure, or a wireless metering system, there is little reason to be concerned with a data glut.
For those who recognize that the Smart Grid and the coming Soft Grid will need data, and will need security, and will likely grow to fill whatever space is available, the call is clear. Plan for an avalanche, for a flood. Create systems and segregations that will allow for managing these flows reliably. Characterize what must come through, and what can be dropped, along the way to the back end. Do all of those things and the current systems will be fine, the next systems will not choke, and the ultimate end state will be similar enough to what has been planned to ensure stability, quality, and cost-effective services to all who connect to the grid.
Image Thanks to:
Sunday, November 1, 2009
The Ninth Control Systems Cyber Security Conference was hosted by Applied Control Solutions (ACS) the week of October 19 in Bethesda, MD. The festivities started Monday morning with parallel activities. A tour was arranged of Washington Suburban Sanitary Commission’s Rock Creek water treatment facility. In parallel, the initial meeting of the ISA Nuclear Plant Cyber Security Joint Working Group was held.
The ACS Conference started Monday afternoon with two introductory sessions: Control Systems for the non-Control System Engineer and IT for the Control Systems Engineer.The Conference began in earnest Tuesday with approximately 110 attendees from US and international electric and water utilities, chemical and oil/gas companies, IT and control system suppliers and consultants, universities, and US and international government agencies. The Conference is called Control Systems Cyber Security is that industrial control systems are common across multiple industries. The agenda can be found at www.realtimeacs.com.
There were two hacking demonstrations of control systems and several discussions on control system cyber vulnerabilities. There was also a discussion on the need for technical control system cyber security curriculum (policy programs exist). There were two keynotes: the Honorable Yvette Clarke (D-NY), Chairwoman of the Subcommittee on Emerging Threats, Cybersecurity, Science and Technology and member of the Intelligence, Information Sharing and Terrorism Risk Assessment Subcommittee provided the lunch keynote. Whitfield Diffie gave the evening keynote and discussed control system cyber security issues from the Tuesday’s session.
There were four different sessions on actual control system cyber incidents – none of which was public! In one session, two control system engineers from two different utilities that have control systems from every major supplier discussed their recent control system cyber incidents – one had his plant shutdown. A couple interesting side notes were that existing control system logging are adequate to identify control system incidents and their control system suppliers weren’t of much help when it came to providing control system cyber security support. Both engineers felt it was so important to share information they attended the Conference on their own nickel. This is in marked contrast to the utility and industry leadership who didn’t think this conference was important enough to attend even though many were based in Washington. Wednesday evening, the Honorable James Langevin gave the evening keynote. Congressman Langevin felt this was so important he spent 30-45 minutes after his presentation answering questions and talking to the attendees.
We received a summary of government activities including legislative efforts on cyber security, cyber security activities by the Nuclear Regulatory Commission, efforts on-going at the Bonneville Power Administration using the NIST Framework, and non-governmental activities in certification and cyber incident collection. Also got a very interesting presentation on cyber security legal issues and a discussion of the Russian cyber attack on Estonia.
On the last day, NIST held training sessions on two very relevant NIST standards:
-- SP 800-53 - Recommended Security Controls for Federal Information Systems - including those for the Bulk Power System
-- SP 800-82 - Guide to Industrial Control Systems (ICS) Security provides guidance on securing Industrial Control Systems (ICS), including Supervisory Control and Data Acquisition (SCADA) systems, Distributed Control Systems (DCS), and other control system configurations
Monday, October 26, 2009
On the positive side, as this article says, high performance, deep cycle lithium ion and lithium air batteries en mass may be the energy storage solution the industry has been searching for. Here's an example starring Duke:
Duke Energy committed to an electric vehicle future when it committed with the FPL Group to buy 10,000 electric vehicles and plug-in hybrids in the coming decade, as they upgrade their fleets. The energy storage in these vehicles could eliminate the need for peaking plants and enable the expanded use of renewable energy. Duke Energy’s electric vehicle future may save billions in future power plant investments.Sounds good, but others worry, here, that local electrical infrastructure can barely handle the additional iPods and iPhones it's had to deal with lately. Adding clusters of electric cars charging at approximately the same time each evening might break the camel's back in many neighborhoods. According to Peter Darbee, the CEO of Pacific Gas & Electric:
A high concentration of plug-in electric vehicles poses a serious challenge to utilities. Plug-in electric cars could draw electricity equivalent the amount needed to run one home, or up to three homes in certain places. You can see if you have three or five electric cars arrive in a neighborhood, you're going to overload the local circuits, and that will lead to blackouts. So we see it as an opportunity but we also see it as a challenge of significant proportions.We all know how neighbors like to mimic and compete with each other (have you seen the Halloween decorations next door !?). One electric car will beget two will beget ten or twenty. Scheduling software will help, but much depends on fast this goes, and how close to edge local circuit gear is at the outset.
Nissan Leaf photo credit: Wikimedia Commons
Monday, October 19, 2009
Thursday, October 15, 2009
- Desire for continuous operation and continuous service to customers by keeping core systems running during (possibly prolonged) power outages impacting local communities
- Energy efficiency savings via reduction in electricity and fossil fuel usage
- Demonstrating proactive/compliance measures vis-a-vis climate change and the increased use of renewable energy sources
- Maintaining confidentiality/privacy of data and doing all of the above is a safe and secure manner
Eighteen months have now passed since the public release the "Defense Science Board Task Force Report on Energy" This is from the section called "Managing Risks to Installations":
For various reasons, the grid has far less margin today than in earlier years between capacity and demand. The level of spare parts kept in inventory has declined, and spare parts are often co-located with their operational counterparts putting both at risk from a single act. In some cases, industrial capacity to produce critical spares is extremely limited, available only from overseas sources and very slow and difficult to transport due to physical size.
Granted, DOD's not the only organization with these concerns ... and the obligation to plan accordingly. Hospitals, police & fire, essential services, etc. all have to think this way. DOD is exploring campus microgrid strategies (including on-site power generation, energy management and energy storage systems, and more) to allow bases to "island" themselves away from commercial grid infrastructure.In many cases, installations have not distinguished between critical and non-critical loads when configuring backup power systems, leaving critical missions competing with non-essential loads for power. The Task Force finds that separating critical from noncritical loads is an important first step toward improving the resilience of critical missions using existing backup sources in the event of commercial power outage. The confluence of these trends, namely increased critical load demand, decreased resilience of commercial power, inadequacy of backup generators, and lack of transformer spares in sufficient numbers to enable quick repair, create an unacceptably high risk to our national security from a long-term interruption of commercial power.
The technology is getting to the point where this approach is becoming just as feasible for industry. We'll be investigating further and will post the results right here.
Photo Credit: Kristen Holden on Flickr
Tuesday, October 13, 2009
In the recent NIST strategy and requirement recommendations release, there was a substantial body of information to be reviewed, and this post is not meant to summarize or to supplant those results (obviously). This is a relatively lightweight view of heavy duty and high-level considerations in software as a critical element in the development of the Smart Grid. It is a practical list of questions that organizations should be able to answer before they commit to software that will either replace or broker their interactions with the Smart Grid.
- What is the software's provenance?
- Provenance is a term that gets thrown around a lot, but I use it to express the idea of origin. Where did the software come from? Who made it? What was it made from? While absolute provenance is difficult or impossible to ascertain, these answers can help to guide risk awareness and management. Is it new software built for me? Is it existing software that has run similar systems elsewhere? Is it a new solution from an existing partner, or revision 0.9 from a start-up? Is it built from the ground up, or does it contain elements of legacy applications, particularly those that my have been written with a different security mindset? By understanding more about the roots of software, the strategy to secure its use will be better informed.
Why ask the question? Unless you know about the origins of software, it is very hard to put together a plan to ascertain its security. Knowing who built it provides a resource to ask about the way in which it was built. Knowing about its components provides information to use in testing it or researching testing done by others.
- What is the plan for ongoing governance?
- Governance, similarly, has a variety of depths of detail and application, particularly in IT. For our purposes, the questions can be limited. How will the software be updated? Who will make those decisions? What is the process to initiate or approve a change? New software in any environment, and even established software in a dynamic environment, will face frequent opportunities and requirements to change. Understanding the models through which those changes are considered, approved, and delivered enables organizations to measure and manage their own risk from flux in the software, and in any collateral instability introduced to dependent systems.
Why ask the question? Instability = Insecurity. Haphazard or non-existent governance leads to more frequent changes, less testing time for the solution in place, and to inevitable discontinuities if the software is a component of a larger system. Weak governance also increases the opportunities and likelihood of malicious coding behavior by simply increasing the chaos during the software delivery process.
- What does the software do with data?
- Data is at the root of almost every application's function and purpose. Whether it exists to generate data, to gather it, or to analyze it, data is not only central to the application, it is often the prime target for an attacker. For that reason, there are multiple facets to consider. What kinds of data does the application gather, where does it come from, and how does it enter the system? Once the data has entered the system, does it get stored, and is it stored with appropriate protection of privacy and integrity? If the data ever moves between components of the system or between multiple systems, is it appropriately protected by the software for privacy and integrity? Does the system restrict access to the data, and is access control sufficiently granular to permit only authorized individuals to enter into the system? Each of these questions naturally results in a series of more technical and specific questions about the behavior of the application, but requiring answers to these high-level queries will mean that these will not be ignored.
Why ask the question? Data is central to the smartness of the Smart Grid, and its protection is expected by subscribers, is in many cases mandated by regulation, and is certainly necessary to ensure reliable operation of the Smart Grid.
- How has the software been tested?
- The testing of software, particularly for security issues, is still a developing field. There are a variety of approaches and mechanisms, each with their own strengths and deficiencies. What testing has been done, and on what components? What approaches were used, and with what results? Have all components been considered for security issues prior to their inclusion, and how were they vetted prior to selection?
Why ask the question? Understanding the testing process for the software can uncover blindspots to some sets of security issues, and can also identify weaknesses in methodology that can indicate systemic problems from the provider. If the testing ignores a specific area, like data storage or access control, then that lack of attention raises the likelihood that there could have been a similar lack of focus during its construction. Testing has many facets, and security must be among them.
These questions are intended to be a very brief introduction to some of the underlying and quite concrete issues that must be considered during the Grid's evolution to a Smart Grid. In time, each of these areas must be expanded into multiple levels of detail, but for now, this is a start. It is the start of generating more informed awareness, and of describing the types and amount of data that is required to feel secure during the adoption of new Smart Grid technologies.
Thursday, October 8, 2009
The security consultant interviewed in the piece, Chris Gatford from HackLabs mentions that in his experience there is ample evidence that the networks may well have been connected despite the efforts of the utility to separate them. This is particularly problematic, I am sure, because there are not only power control systems to worry about, but also online payment, user account management, and other relatively advanced functions at Integral Energy.
His comments seemed familiar to me, so I went back through my notes, all the way to a report from the team at Riptech in 2001 ( Bought by Symantec) called " Understanding SCADA System Security Vulnerabilities ", where the authors describe a very similar disconnect between assumptions and reality in these internal networks:
MISCONCEPTION #1 – “The SCADA system resides on a physically separate, standalone network.”
MISCONCEPTION #2 – “Connections between SCADA systems and other corporate networks are protected by strong access controls.”Most SCADA systems were originally built before and often separate from other corporate networks. As a result, IT managers typically operate on the assumption that these systems cannot be accessed through corporate networks or from remote access points. Unfortunately, this belief is usually fallacious.
In reality, SCADA networks and corporate IT systems are often bridged as a result of two key changes in information management practices. First, the demand for remote access computing has encouraged many utilities to establish connections to the SCADA system that enable SCADA engineers to monitor and control the system from points on the corporate network. Second, many utilities have added connections between corporate networks and SCADA networks in order to allow corporate decision makers to obtain instant access to critical data about the status of their operational systems. Often, these connections are implemented without a full understanding of the corresponding security risks. In fact, the security strategy for utility corporate network infrastructures rarely accounts for the fact that access to these systems might allow unauthorized access and control of SCADA systems.
Many of the interconnections between corporate networks and SCADA systems require the integration of systems with different communications standards. The result is often an infrastructure that is engineered to move data successfully between two unique systems. Due to the complexity of integrating disparate systems, network engineers often fail to address the added burden of accounting for security risks.As a result, access controls designed to protect SCADA systems from unauthorized access through corporate networks are usually minimal, which is largely attributable to the fact that network managers often overlook key access points connecting these networks. Although the strategic use of internal firewalls and intrusion detection systems (IDS), coupled with strong password policies, is highly recommended, few utilities protect all entry points to the SCADA system in this manner.
One sees allusions to the concept of separate networks, with various properties, in existing regulation, CIP descriptions, etc. If we can agree that there are likely to be unintended cross-overs between these systems and their populations, then we must also agree to stop considering the artifice of disjoint networks as being anything but an anachronism, and treat the security of each network with the same rigor and protective approaches, regardless of our faith in its isolation from sources of corruption.
Wednesday, October 7, 2009
So you're an executive in charge of security at a medium, large or very large organization. You might be called Chief Security Officer (CSO) or Chief Information Security Officer (CISO) or maybe VP or Director of Security. You most likely report to the Corporate CIO, or you're in a business division and you and your boss plug into a General Manager. You decide, with blessing from above for the big stuff, the following:
- Where you'll get the biggest risk reduction (or compliance) bang for your limited budget buck
- Which technologies get purchased and implemented
- Which vendors will augment your in-house security team, and,
- Corporate security policies, and how to best promulgate them to other parts of the co. for whom security is at best an annoyance, and at worst, something to be openly resisted
Hey, Have you Heard of Smart Grid?
So how much time do you spend on future threats? If you have heard of the Smart Grid, and if you've been reading up on it, then you probably don't need to read further here. You're in the top 10% of your class and get a star on your forehead. If however, you're like some CSOs I've talked with who claim to have never heard the term, then this is your wake up call. There has been little written to guide CSOs through the early stages of preparing to protect their organizations in a world where the power systems they rely on look increasingly like the Internet (and in some cases are the Internet!).
How is it different from today's electrical grid? For starters, it's a 2 x 2-way system. Thanks to advanced metering infrastructure (AMI) and net metering, electricity and usage information will flow from generators to consumers and back again. The total amount of information, which in the beginning will be substantial, will quickly become enormous. Data protection will be crucial, and demand management strategies which could save your organization significant money, could also get you in trouble fast. Water and other services will also be impacted for better and worse. In short, for each benefit a Smarter Grid will bring an organization, there is a commensurate risk to mitigate. And it's your job to know (and plan for) this.
Only CSOs at utilities see this world first hand, and even in the energy and utilities vertical, many of those CSOs work in a balkanized world where their policies touch only IT, and the "rubber meets the road" part of their company, field operations, doesn't want to anything to do with them.
So most CSOs are left to infer what they need to know from a mountain of Smart Grid articles and a multiplicity of Smart Grid conferences. My guess is once they've poked a toe into these confusing waters one time, they soon find their time better spent working on present challenges. The appropriate information has not yet been boiled down for this most important enterprise leadership function ... one that could and would do the right things, proactively, if it had the right knowledge to work with.
CSO Info Resources Not Too Helpful Yet
Where do CSOs turn for expert guidance and to learn from what their successful peers are doing? Why, the journals and other news sources that serve them. Yet from the looks of these two articles from CSO Online and the CSO Roundtable, all they're getting is high level introductory material that in no way considers how Smart Grid trends intersect with CSOs' particular responsibilities. I would advise these orgs to get on the ball: it's their job to see over the horizon and around corners to give their readers the info they need to protect their companies ... and their jobs.
No Answers Yet, But Here are a Few Starter Questions
NIST and other standards bodies are working around the clock to bring appropriate and helpful security standards to this new domain and you don't have to know them yet (however, for a sneak peak, here's the most recent draft edition of Smart Grid Cyber Security Strategy and Requirements from NIST). So much is still in flux that doing too much at present might be as bad as doing too little. But that doesn't mean you shouldn't start getting your head around this challenge and thinking through some of the scenarios. Here's a handful:
- Supply Chain - Similar to Y2K preparation in some respects, even if you get your house in order for the arrival of the Smart Grid, if the companies yours depends on are not prepared it may affect you. It's time to talk about this with them.
- Vehicle Fleet - More choices are coming, including hybrid electric, full electric, natural gas, etc. Are you thinking about the challenges and opportunities that present themselves in beginning to move away from gasoline and diesel? What are the security implications of your enterprise depending on these new transportation technologies?
- Local utilities - All utilities are under guidance to prepare for Smart Grid standards and technologies. What are your providers doing in your different locations and how soon will their actions begin to affect you? What do you need to do to not get blind sided?
- Smart Grid pilots - With stimulus help from the Fed Gov, pilots are springing up everywhere. Related to number 3 above, are there any pilots going on you could participate in? While this might take resources away from more proximate concerns, the education might more than pay for the time invested.
- Centralized policy and control - If yours is a geographically distributed operation, to what extent will you attempt to define and enforce Smart Grid-related security policy in a uniform way, versus allowing disparate facilities and offices to determine their own best approaches?