Tuesday, August 31, 2010
Monday, August 30, 2010
You’d be forgiven for thinking that with the recent excitement over the Stuxnet virus (here, here and here) and other cyber threats, that this blogger believes that security issues present the biggest challenge to the success of a national Smart Grid.
But there's something else that threatens the grand Smart Grid project on an even more fundamental level: we all have to believe in the goodness of this work enough to see it through ... even when there are setbacks. And sometimes it seems we might not.
The corollary of the oft-cited Field of Dreams baseball diamond axiom “If you build it, they will come” is the far less-often cited “… and if you don’t, they won’t”. In 2010 we’re still in the Smart Grid’s infancy, and while it’s not yet clear what’s the right way to build it, this case has shown that failing to plan and permit up front is one guaranteed way to fail. The net net is that the Smart Grid will not be fully deployed in Boulder … not for the foreseeable future anyway.
According to SmartGridNews, Greentech Media and earth2tech’s Katie Fehrenbacher:
The real problem is that [they] didn’t perform a cost-benefit analysis prior to starting the project. [Also] the group originally didn’t file for a “Certificate of Public Convenience and Necessity” … when the project started … a filing that would have enabled the PUC to cap costs of the project to protect rate payers.Go back to an online debate we held on the Smart Grid Security Blog and the SmartGridNews site almost a year ago. We began with a post I called “First Mover Disadvantage”, turning a standard business school strategy on its head. The basic idea was that in these very early days, there’s far too much uncertainty (e.g., technology, standards, business models, regulatory environment, etc.) for companies, especially electric utilities, to get a jump on the market without enduring substantial setbacks and risk enormous costs for themselves and their rate payers.
Jack’s response, "Not the Lead Dog? Get used to the View", made the case that despite the uncertainty, those utilities with enough chutzpah to get their hands dirty, make mistakes, learn from them and press on, would command a disproportionate share of influence in the market over those sitting on the sidelines waiting for the eventual shake out.
I like both of these ideas, and surely a decent university debate team could make a lot of hay advancing either argument. But I’m going to say that the SmartGridCity project is an example of moving big and early, and in-so-doing, doing it wrong from the get-go. Projects this complex, with this many players, will inevitably be quite risky, and therefore must be managed extra carefully. There is less room for short cuts, and even when designed and managed flawlessly, they may still endure their share of lumps. These folks sealed their fate in the beginning, and added insult to injury by boasting so publically about their achievements.
It’s that last part that bothers me the most as the biggest threats to the success of the Smart Grid aren’t what you might first imagine: it’s not cyber terrorists, regularity inertia, or flawed technology that most threaten the build-out of the US national Smart Grid. Rather, it’s a potential public perception that promised Smart Grid benefits aren’t nearly worth the costs that could kill it before it's born.
In the early days when we're still trying to figure out what works, there are going to be more Bakersfields, BG&E's and now Michigans for sure. But it's important that the industry ensure that success stories make their way to the media at least as often as the gotcha's. I want to focus on the security challenges facing the Smart Grid, but won't be able to do that for long if we don't get the thing fielded in the first place.
Photo credit:Nieve44/La Luz at Flick.com
Thursday, August 26, 2010
For those wondering whether the USB drive-facilitated Stuxnet virus is over hyped or not, Kapersky Labs senior security analyst Roel Schouwenberg has fifteen words for you:
This is without any doubt the most sophisticated targeted attack we have seen so far.
Wednesday, August 25, 2010
Monday, August 16, 2010
Friday, August 13, 2010
And also, in case you missed it buried inside a long post from the recent SG Cybersec Summit, THIS Symantec update is dense and rich in good Stuxnet info. One thing to remember as you read these write-ups, both co's acknowledge that analysis on Stuxnet is far from complete. Stay tuned.
Photo credit: Fred Hemerick on Flickr.com
Thursday, August 12, 2010
Like fraternal twins separated at birth, these two seemingly unrelated and elderly sectors of the US economy have more in common than you might think. Both are poised for immense change as “Smart” technologies are completely re-writing the workflows and even the business models of these formerly static industries. One way to know they’ve haven’t changed much over the last century: their 2010 products would be instantly recognizable to their inventors (though this Shelby SuperCar might induce Henry Ford to do a double, or maybe a triple) take. Another thing they have in common is that they have viewed their customers’ usage data as primarily their own.
Some More Similarities
Internal Smart car systems behave less like data centers and more like control systems. On board performance monitoring and diagnostic computers and sensors, coupled with wireless communications systems, are beginning to allow car companies to detect and sometimes resolve problems without requiring that the car be brought into a garage for repair.
Similarly Siloed: Meter Rolls vs. Rolling Meters
Prior to the advent of wireless car communications networks (e.g. GM’s OnStar, Ford’s Sync, Bluetooth, Wifi, etc.), automotive performance and diagnostic data remained in on board computers until technicians accessed them during visits to the repair shop. In-between regularly scheduled oil changes or check-ups, or without a break-down or crack-up, this data was out of reach. Now with communications enabled, daily access to this data is a new possibility. And as data on total electricity consumption and usage patterns in homes, the car companies clearly have rights, but the owner/drivers also have a stake as they own and operate the cars (especially if their identity is connected to the data).
But in both industries, there hasn’t previously been much thought given to the ownership or role of data in these scenarios. Or how that data might have value for new business lines or 3rd parties. Or how to protect that data in scenarios where multiple 3rd parties are allowed access.
What cars and utilities shared in the past, even as they came to rely more and more on electronics, was that these systems were relatively simple, understandable, and isolated from the networks bad guys are known to frequent. The hardware and software in most OT systems are not familiar to most of us, as their functions are not related to web apps, productivity or back office management, but to control sensors, actuators and other types of real-time devices.
Trends over the past few years, however, indicate complexity and connectedness will soon rule both of these worlds. Note that current cars of the standard combustion engine variety now depend upon 200+ million lines of software code in applications from a variety of sources with dozens of interfaces. Once “dumb” disconnected meters are being replaced by Smart Meters - networked computers on the side of homes and buildings which communicate with utility systems as well as systems on the inside, like Home Area networks (HANs) and Smart appliances. And all over, IT and OT systems are increasingly being interconnected.
That’s only going to increase as we enter the Vehicle to Grid (V2G) and Smart Grid worlds, with individuals and new companies clamoring for ways to gain access to and open up these systems, access their data, and re-invigorate these previously stagnant sectors with innovative new technologies, capabilities and business models. Open standards (and advocacy campaigns like OpenOtto) will hasten the arrival of all of the above, but in both the power and the car worlds, the impulse to open up has been largely absent, at least until recently.
Ah, we’ve saved the best for last. It’s been said before on this blog but it bears repeating: connecting systems that were once protected, in large part, by their isolation, creates many new vectors for attackers, and in general, many new ways to be insecure.
Designers of both Smart cars and Smart Meters share the objective that upgrades to software and firmware can be performed remotely, prolonging the lives, and increasing the flexibility, of these systems.
And the temptation to share customer usage data complicates both car and utilities’ thinking about their own data security measures. Ensuring proper data protections are in place in every entity that eventually has access, even with customer permission, is going to be a tough challenge. So let's get on it!
Photo credit: Bill Jacobus on Flickr.com
Wednesday, August 11, 2010
- Robert Former, Itron, Principal Security Engineer
- Edward Beroset, Elster, Director of Technology and Standards
- Stan Chan, Verisign/Symantec, Director of Strategic Initiatives
- We've gotten much more serious about security in the past year and we're making changes at a rapid pace
- All products go through rigorous security tests by reputable third parties pre-release, and security testing is continuous throughout the lifecycle
- Plans to share vulnerabilities ID'd in these 3rd party tests with PUCs and other regulators and stakeholders
- Additional attention to security driven by huge push for more security from customers: utilities
- A question was raised on whether Smart Meters could trust smart toasters. There was no answer to this question as it was rhetorical I believe. Certainly thought provoking
- Meters must withstand extreme weather conditions and consume no more than 5 watts. Think about it - a one watt difference per meter x 1 million meters = a megawatt
Murmuring and agitation ensued ... along with very many words flowing high rates of speed. To sum it up, I believe the response was along the lines of "hell yes and they're using lawyers and all other means at their disposal to slow microgrid deployments down." Personally, I don't believe that response captures what I see as a range of microgrid thinking by utilities. Some of them, I'm sure we'll see, want to get out in front of this movement and will make it another part of their offerings.
In marked contrast, the final panel, which included Elinor Mills of CNET, was a thoughtful and somber meditation on the near-perfect relationship between the media and Smart Grid utilities and vendors. NOT!!! It was fairly raucous and included a course mixture of literal and figurative finger pointing. In the end, neither side was completely innocent of wrong doing and neither side was completely guilty. Both sides agreed to keep talking with hopes that better understanding and communications would follow in the fullness of time.
As for the conference itself, I spoke with a couple dozen folks before we disbanded and all were well pleased with what they'd experienced and all pledged to attend the next Smart Grid Cyber Security Summit event. I have it on good faith that videos and other useful artifacts from the conference will soon appear on the summit site. When they do, I'll be sure to send out a heads-up here on the SGSB.
That's a wrap for now. I've got a red eye back to Beantown to catch. Go Sox!
Photo credit: The Social Blog Network on Flickr.com
- Scott Borg of the US Cyber Consequences Unit showed how the US economy can easily weather ~3.5 day outages, but that when you get beyond that duration across a broad region, you get into large and very large effects, as in "massive ... breathtakingly bad." So small, short duration security incidents we can handle and don't need to worry about too much. But we should move (and spend) heaven and earth to ensure we don't experience even one of the very big ones
- Bob Gohn of Pike Research gave us the latest Smart Grid security findings and trends, and announced the release of Pike's latest report on Smart Meter Security
- FERC Commissioner Philip Moeller, whose NERC CIP experience goes back to some of the earliest draft language from the year 2000, acknowledged the challenges NERC faces fielding a uniformly solid field of CIPS auditors, and told us to keep our eyes open for a possible collaborative effort involving FERC and state regulators
- I could do an entire post on Joe Weiss' presentation, but for now let it suffice to say that the Stuxnet virus is much more problematic than initial reports (including one made on this blog) indicated. Here's a decent Stuxnet update from Symantec. Among other things, note the lengths this malware goes to to protect itself from detection
- Joe also made it clear that Smart Grid or no Smart Grid, SCADA/ICS systems are a disaster waiting to happen and that there's not a heck of a lot we can do about it. He supported this point by saying: 1) we have basically zero forensics capabilities to investigate SCADA/ICS attacks; 2) OT hates IT in all sectors, not just energy, and that this culture war gets in the way of migrating good security practices to the SCADA/ICS world; 3) there's nothing at all comprehensive about NERC CIPS; 4) there are 5 or fewer utilities going beyond the security controls required by the CIPS; 5) to work, SCADA/ICS security must be a living program, as every time you change or add something, you impact security; 6) NERC CIPS have made the grid less reliable by enticing some utilities into removing IP connections from some important devices, which makes them exempt from NERC CIP while leaving them dependent on serial connections, which are themselves quite susceptible to attack
- After Joe left the NERC CIPS in smouldering ruins, Rob Shein, HP Cyber Security Architect, coaxed them back to life with a balanced review of what they do and do not cover, and provided reasonable steps orgs can follow to achieve compliance
- Lastly, I moderated a roundtable session on "The Perspective and Path Forward for Energy Utilities" with 3 outstanding panelists: Mike Echols of the Salt River Project, Bobby Brown of Enernex, and Chris Peters from Entergy. They hit a bunch of topics that even late in the day held the audience's attention and responded to lots of questions after they reached the end of my prepared list. But for me, the most memorable of all was also the simplest. Each was asked: would your org be more or less secure in a world without the CIPS? To which the unanimous response was less. So despite all the abuse heaped upon the CIPS during the day (and IMHO, they richly deserve it), the folks fighting this security battle in the trenches say they help far more than they hurt. For me, that fully topped off an already great day, and I'm really looking forward to whatever lessons we can tease out of day 2 of the 1st Smart Grid Cyber Security Summit.
Tuesday, August 10, 2010
Attacks that undermine trust could prevent the Smart Grid from happening.
Tuesday, August 3, 2010
Seems to me you are in a damned if you do, damned if you don't situation. On one hand, you must do everything you can to keep the processes in place that have kept the customers' lights on 24/7/365 over the past decades of your career. Moving too far too fast with new technology or methods puts that number one metric at risk. On the other hand, in order to put your organization in position to pass its NERC CIP compliance audits and avoid fines and other negative fallout, you're having to substantially upgrade and update the security controls on some of your most important systems.
Like the oft-referenced complex challenge of repairing an airplane in flight, you face the dilemma above in a time of unprecedented change in an industry ill equipped organizationally to make fast changes. For example:
- In a sector largely insulated from competition, deregulation (in some regions) now adds that factor to the mix. And some of the competitors are from another planet, culturally speaking (see: Google, Microsoft, etc.)
- AMI and Smart Grid initiatives are encouraging you to connect systems that were once protected, in part, through isolation
- Business models look like they're in position to turn inside out and dis-intermediation is a real possibility
- The FERC/NERC CIP cyber security regulatory regime is moving fast; you're given a scant 2 years to turn your ship in the right direction (impossible for some), and rumors of more stringent and burdensome standards coming abound
- And last but not least, what about the GRID Act? Its passage looks like a near certainty. You only thought you had compliance problems before !!!
So, in this climate, should you err on the side of doing too much? Moving your org rapidly towards better security and compliance but adding an unknown amount of reliability risk even as you seek to reduce it? Or lean towards preserving the steady state status quo and do too little, and risk getting slammed by fines ... or worse (Stuxnet anyone)? Often there's a middle path you can construct that gives you a nice balance of risk and reward, but I'm not sure that's the case here. But whatever you choose, the rest of us on this blog appreciate the tight spot you're in and will do as much as we can to make your world a little simpler.