Energy Security by Design

Jack's been busy making commercials for IBM's Smarter Planet campaign, describing the company's new security mantra, "Secure by Design" in the context of Smart Grid and energy systems. Click HERE to see the first one on Youtube. And it looks like the film crews indulged him with another on a topic even nearer and dearer to his true passion: FOOD security.

Security isn’t the Biggest Threat to the Smart Grid

You’d be forgiven for thinking that with the recent excitement over the Stuxnet virus (here, here and here) and other cyber threats, that this blogger believes that security issues present the biggest challenge to the success of a national Smart Grid.

But there's something else that threatens the grand Smart Grid project on an even more fundamental level: we all have to believe in the goodness of this work enough to see it through ... even when there are setbacks. And sometimes it seems we might not.

The corollary of the oft-cited Field of Dreams baseball diamond axiom “If you build it, they will come” is the far less-often cited “… and if you don’t, they won’t”. In 2010 we’re still in the Smart Grid’s infancy, and while it’s not yet clear what’s the right way to build it, this case has shown that failing to plan and permit up front is one guaranteed way to fail. The net net is that the Smart Grid will not be fully deployed in Boulder … not for the foreseeable future anyway.

According to SmartGridNews, Greentech Media and earth2tech’s Katie Fehrenbacher:

The real problem is that [they] didn’t perform a cost-benefit analysis prior to starting the project. [Also] the group originally didn’t file for a “Certificate of Public Convenience and Necessity” … when the project started … a filing that would have enabled the PUC to cap costs of the project to protect rate payers.

Go back to an online debate we held on the Smart Grid Security Blog and the SmartGridNews site almost a year ago. We began with a post I called “First Mover Disadvantage”, turning a standard business school strategy on its head. The basic idea was that in these very early days, there’s far too much uncertainty (e.g., technology, standards, business models, regulatory environment, etc.) for companies, especially electric utilities, to get a jump on the market without enduring substantial setbacks and risk enormous costs for themselves and their rate payers.

Jack’s response, "Not the Lead Dog? Get used to the View", made the case that despite the uncertainty, those utilities with enough chutzpah to get their hands dirty, make mistakes, learn from them and press on, would command a disproportionate share of influence in the market over those sitting on the sidelines waiting for the eventual shake out.

I like both of these ideas, and surely a decent university debate team could make a lot of hay advancing either argument. But I’m going to say that the SmartGridCity project is an example of moving big and early, and in-so-doing, doing it wrong from the get-go. Projects this complex, with this many players, will inevitably be quite risky, and therefore must be managed extra carefully. There is less room for short cuts, and even when designed and managed flawlessly, they may still endure their share of lumps. These folks sealed their fate in the beginning, and added insult to injury by boasting so publically about their achievements.

It’s that last part that bothers me the most as the biggest threats to the success of the Smart Grid aren’t what you might first imagine: it’s not cyber terrorists, regularity inertia, or flawed technology that most threaten the build-out of the US national Smart Grid. Rather, it’s a potential public perception that promised Smart Grid benefits aren’t nearly worth the costs that could kill it before it's born.

In the early days when we're still trying to figure out what works, there are going to be more Bakersfields, BG&E's and now Michigans for sure. But it's important that the industry ensure that success stories make their way to the media at least as often as the gotcha's. I want to focus on the security challenges facing the Smart Grid, but won't be able to do that for long if we don't get the thing fielded in the first place.

Photo credit: Nieve44/La Luz at Flick.com

Stuxnet Update III: Death to USB Thumb Drives

Funny, I just used a thumb drive to print out a presentation on a hotel business center printer last week. I put that drive back in my briefcase. Next thing I do after posting this post will be to put that tiny device on the rail of the Boston Green Line subway that runs just outside my front door. And you should probably do your equivalent of the same. And then we should all go cold turkey and not touch the things again (even they're kind of cute and convenient as hell).

For those wondering whether the USB drive-facilitated Stuxnet virus is over hyped or not, Kapersky Labs senior security analyst Roel Schouwenberg has fifteen words for you:

This is without any doubt the most sophisticated targeted attack we have seen so far.

You can read Joe's latest Stuxnet post HERE.

OK, off to catch that train.

Photo credit: Wheels00/Mark Brown at Flickr.com

A Simple Roadmap to NISTIR 7628

Don't thank me for this. Instead, thank Dale Peterson, founder of control systems cyber security consulting firm Digital Bond. He breaks the nearly-finished 7628 into bite-sized chunks and tells you how to eat it for maximum nutrition and comprehension. Read his post here and see more about Dale and Digital Bond here.

Map credit: Norman B. Leventhal Map Center at the Boston Public Library on Flickr.com

New Webcast on Security Standards

Here's the latest ...

SGSB Webcast 4: Smart Grid Security Standards in Mid 2010

View more webinars from Andy Bochman.

SGSB Stuxnet Update

It's been 2 weeks since my first Stuxnet post on July 27. Now here's the best update so far I've seen on Stuxnet as of August 12, 2010. It's an Industrial Defender Q&A session with some apparently very knowledgeable and very motivated webinar participants. You can see it HERE.

And also, in case you missed it buried inside a long post from the recent SG Cybersec Summit, THIS Symantec update is dense and rich in good Stuxnet info. One thing to remember as you read these write-ups, both co's acknowledge that analysis on Stuxnet is far from complete. Stay tuned.

Photo credit: Fred Hemerick on Flickr.com

Car Companies and Utilities at the Dawn of the Smart Grid: Twins Separated at Birth?

Like fraternal twins separated at birth, these two seemingly unrelated and elderly sectors of the US economy have more in common than you might think. Both are poised for immense change as “Smart” technologies are completely re-writing the workflows and even the business models of these formerly static industries. One way to know they’ve haven’t changed much over the last century: their 2010 products would be instantly recognizable to their inventors (though this Shelby SuperCar might induce Henry Ford to do a double, or maybe a triple) take. Another thing they have in common is that they have viewed their customers’ usage data as primarily their own.

Some More Similarities

While both car companies and utilities manage their business functions with modern data centers and IT, it’s the OT that makes them their money. That’s operational technology, and for utilities OT refers to the infrastructure control system components that make the grid go: generators, power lines, transformers, substations, etc. The Smart Meters, currently being deployed and networked in the millions by many large-market utilities to enable remote trouble detection and billing, can also considered OT systems.

Internal Smart car systems behave less like data centers and more like control systems. On board performance monitoring and diagnostic computers and sensors, coupled with wireless communications systems, are beginning to allow car companies to detect and sometimes resolve problems without requiring that the car be brought into a garage for repair.

Similarly Siloed: Meter Rolls vs. Rolling Meters

Looking at the two platforms from a customer data perspective, the similarities are even stronger. Electricity usage data was the reason utility trucks ventured to homes and businesses across the country. Utilities had no other way of knowing how much electricity was used at a given address, and they needed that data to calculate how much they were owed. You could make a case that this usage data belonged to the utilities, or to the customers themselves, or both. And today, different states have different rules on this issue.

Prior to the advent of wireless car communications networks (e.g. GM’s OnStar, Ford’s Sync, Bluetooth, Wifi, etc.), automotive performance and diagnostic data remained in on board computers until technicians accessed them during visits to the repair shop. In-between regularly scheduled oil changes or check-ups, or without a break-down or crack-up, this data was out of reach. Now with communications enabled, daily access to this data is a new possibility. And as data on total electricity consumption and usage patterns in homes, the car companies clearly have rights, but the owner/drivers also have a stake as they own and operate the cars (especially if their identity is connected to the data).

But in both industries, there hasn’t previously been much thought given to the ownership or role of data in these scenarios. Or how that data might have value for new business lines or 3rd parties. Or how to protect that data in scenarios where multiple 3rd parties are allowed access.

Complexity

What cars and utilities shared in the past, even as they came to rely more and more on electronics, was that these systems were relatively simple, understandable, and isolated from the networks bad guys are known to frequent. The hardware and software in most OT systems are not familiar to most of us, as their functions are not related to web apps, productivity or back office management, but to control sensors, actuators and other types of real-time devices.

Trends over the past few years, however, indicate complexity and connectedness will soon rule both of these worlds. Note that current cars of the standard combustion engine variety now depend upon 200+ million lines of software code in applications from a variety of sources with dozens of interfaces. Once “dumb” disconnected meters are being replaced by Smart Meters - networked computers on the side of homes and buildings which communicate with utility systems as well as systems on the inside, like Home Area networks (HANs) and Smart appliances. And all over, IT and OT systems are increasingly being interconnected.

That’s only going to increase as we enter the Vehicle to Grid (V2G) and Smart Grid worlds, with individuals and new companies clamoring for ways to gain access to and open up these systems, access their data, and re-invigorate these previously stagnant sectors with innovative new technologies, capabilities and business models. Open standards (and advocacy campaigns like OpenOtto) will hasten the arrival of all of the above, but in both the power and the car worlds, the impulse to open up has been largely absent, at least until recently.

Security

Ah, we’ve saved the best for last. It’s been said before on this blog but it bears repeating: connecting systems that were once protected, in large part, by their isolation, creates many new vectors for attackers, and in general, many new ways to be insecure.

Designers of both Smart cars and Smart Meters share the objective that upgrades to software and firmware can be performed remotely, prolonging the lives, and increasing the flexibility, of these systems.

There are also use cases where the ability to remotely shut down meters or cars is highly desirable, and include, for utilities when they don’t get paid, or when a residence is changing owners or occupants; and for car companies, the ability to team with the police to stop car thieves and other criminals. These capabilities, like so much related to the Smart Grid, Smart Meters and Smart cars, open new pathways for attackers.

And the temptation to share customer usage data complicates both car and utilities’ thinking about their own data security measures. Ensuring proper data protections are in place in every entity that eventually has access, even with customer permission, is going to be a tough challenge. So let's get on it!

Photo credit: Bill Jacobus on Flickr.com

The 1st Smart Grid Cyber Security Summit is Toast - Selected Notes from Day 2

As good as the utility panel was at the end of Day 1 (see final bullet here), Day 2's vendor panel comprised mainly of meter guys was another clear standout. On the stage were:

• Robert Former, Itron, Principal Security Engineer
• Edward Beroset, Elster, Director of Technology and Standards
• Stan Chan, Verisign/Symantec, Director of Strategic Initiatives

Here are some of what this panel conveyed, sans attribution:

• We've gotten much more serious about security in the past year and we're making changes at a rapid pace
• All products go through rigorous security tests by reputable third parties pre-release, and security testing is continuous throughout the lifecycle
• Plans to share vulnerabilities ID'd in these 3rd party tests with PUCs and other regulators and stakeholders
• Additional attention to security driven by huge push for more security from customers: utilities
• A question was raised on whether Smart Meters could trust smart toasters. There was no answer to this question as it was rhetorical I believe. Certainly thought provoking
• Meters must withstand extreme weather conditions and consume no more than 5 watts. Think about it - a one watt difference per meter x 1 million meters = a megawatt

Later, a question from the back row to the microgrid panel caused a stir. What was that question? Something like: "Are utilities aware of the possible disruptive nature of microgrids to their well established business models?"

Murmuring and agitation ensued ... along with very many words flowing high rates of speed. To sum it up, I believe the response was along the lines of "hell yes and they're using lawyers and all other means at their disposal to slow microgrid deployments down." Personally, I don't believe that response captures what I see as a range of microgrid thinking by utilities. Some of them, I'm sure we'll see, want to get out in front of this movement and will make it another part of their offerings.

In marked contrast, the final panel, which included Elinor Mills of CNET, was a thoughtful and somber meditation on the near-perfect relationship between the media and Smart Grid utilities and vendors. NOT!!! It was fairly raucous and included a course mixture of literal and figurative finger pointing. In the end, neither side was completely innocent of wrong doing and neither side was completely guilty. Both sides agreed to keep talking with hopes that better understanding and communications would follow in the fullness of time.

As for the conference itself, I spoke with a couple dozen folks before we disbanded and all were well pleased with what they'd experienced and all pledged to attend the next Smart Grid Cyber Security Summit event. I have it on good faith that videos and other useful artifacts from the conference will soon appear on the summit site. When they do, I'll be sure to send out a heads-up here on the SGSB.

That's a wrap for now. I've got a red eye back to Beantown to catch. Go Sox!

Photo credit: The Social Blog Network on Flickr.com

Day One Recap from the 1st Smart Grid Cyber Security Summit

I'm afraid it's a little too late to go for complete coherence, so here are some bleary eyed bullets summarizing a few (but not nearly all) of the first days' highlights:

• Scott Borg of the US Cyber Consequences Unit showed how the US economy can easily weather ~3.5 day outages, but that when you get beyond that duration across a broad region, you get into large and very large effects, as in "massive ... breathtakingly bad." So small, short duration security incidents we can handle and don't need to worry about too much. But we should move (and spend) heaven and earth to ensure we don't experience even one of the very big ones
• Bob Gohn of Pike Research gave us the latest Smart Grid security findings and trends, and announced the release of Pike's latest report on Smart Meter Security
• FERC Commissioner Philip Moeller, whose NERC CIP experience goes back to some of the earliest draft language from the year 2000, acknowledged the challenges NERC faces fielding a uniformly solid field of CIPS auditors, and told us to keep our eyes open for a possible collaborative effort involving FERC and state regulators
• I could do an entire post on Joe Weiss' presentation, but for now let it suffice to say that the Stuxnet virus is much more problematic than initial reports (including one made on this blog) indicated. Here's a decent Stuxnet update from Symantec. Among other things, note the lengths this malware goes to to protect itself from detection
• Joe also made it clear that Smart Grid or no Smart Grid, SCADA/ICS systems are a disaster waiting to happen and that there's not a heck of a lot we can do about it. He supported this point by saying: 1) we have basically zero forensics capabilities to investigate SCADA/ICS attacks; 2) OT hates IT in all sectors, not just energy, and that this culture war gets in the way of migrating good security practices to the SCADA/ICS world; 3) there's nothing at all comprehensive about NERC CIPS; 4) there are 5 or fewer utilities going beyond the security controls required by the CIPS; 5) to work, SCADA/ICS security must be a living program, as every time you change or add something, you impact security; 6) NERC CIPS have made the grid less reliable by enticing some utilities into removing IP connections from some important devices, which makes them exempt from NERC CIP while leaving them dependent on serial connections, which are themselves quite susceptible to attack
• After Joe left the NERC CIPS in smouldering ruins, Rob Shein, HP Cyber Security Architect, coaxed them back to life with a balanced review of what they do and do not cover, and provided reasonable steps orgs can follow to achieve compliance
• Lastly, I moderated a roundtable session on "The Perspective and Path Forward for Energy Utilities" with 3 outstanding panelists: Mike Echols of the Salt River Project, Bobby Brown of Enernex, and Chris Peters from Entergy. They hit a bunch of topics that even late in the day held the audience's attention and responded to lots of questions after they reached the end of my prepared list. But for me, the most memorable of all was also the simplest. Each was asked: would your org be more or less secure in a world without the CIPS? To which the unanimous response was less. So despite all the abuse heaped upon the CIPS during the day (and IMHO, they richly deserve it), the folks fighting this security battle in the trenches say they help far more than they hurt. For me, that fully topped off an already great day, and I'm really looking forward to whatever lessons we can tease out of day 2 of the 1st Smart Grid Cyber Security Summit.

San Jose Photo credit: the_tahoe_guy at Flickr.com

Live from the 1st Smart Grid Cyber Security Summit

We're underway here in San Jose and it's going great so far. Getting eighty folks to show-up in August is a pretty impressive feat for a first-ever conference. It's a good mix including utilities, regulatory and industry personnel.

This picture shows Mark Schaeffer of GraniteKey, Matt Carpenter of InGuardians and Gib Sorebo of  SAIC leading a talk called Best Practices in Grid Detection and Prevention. Won't have much of a chance to write up most of this, but will try to extract a few nuggets.

Most in keeping with one of the primary themes of this blog was a statement during Scott Borg's kickoff keynote on the value of Smart Grid security. Borg noted that:

Attacks that undermine trust could prevent the Smart Grid from happening.

IMHO, everything we're doing here at the conference and back in our days jobs stems from that one simple assertion. 

You can see the agenda we're working through right now HERE.

Anyway, just wanted you to know that good things are happening here and that, thanks to strong interest and attendance, this conversation will continue in subsequent versions of the conference. Maybe one back East.

Photo credit: Andy

Mid 2010 Snapshot: Utilities in Security and Compliance Double Bind

If you're not the head honcho for security at a medium-to-large utility company in the USA these days, you should consider yourself fortunate that, regardless of your profession, your life is much less complicated than theirs. If you are in such a position, you have my sympathy, and depending on how you're managing, my respect.

Seems to me you are in a damned if you do, damned if you don't situation. On one hand, you must do everything you can to keep the processes in place that have kept the customers' lights on 24/7/365 over the past decades of your career. Moving too far too fast with new technology or methods puts that number one metric at risk. On the other hand, in order to put your organization in position to pass its NERC CIP compliance audits and avoid fines and other negative fallout, you're having to substantially upgrade and update the security controls on some of your most important systems.

Like the oft-referenced complex challenge of repairing an airplane in flight, you face the dilemma above in a time of unprecedented change in an industry ill equipped organizationally to make fast changes. For example:

• In a sector largely insulated from competition, deregulation (in some regions) now adds that factor to the mix. And some of the competitors are from another planet, culturally speaking (see: Google, Microsoft, etc.)
• AMI and Smart Grid initiatives are encouraging you to connect systems that were once protected, in part, through isolation
• Business models look like they're in position to turn inside out and dis-intermediation is a real possibility
• The FERC/NERC CIP cyber security regulatory regime is moving fast; you're given a scant 2 years to turn your ship in the right direction (impossible for some), and rumors of more stringent and burdensome standards coming abound
• And last but not least, what about the GRID Act? Its passage looks like a near certainty. You only thought you had compliance problems before !!!

Just writing this list makes me gets me all worked up. Time to turn to the timeless wisdom of the Ramones; "I wanna be sedated". OK, better now.

So, in this climate, should you err on the side of doing too much? Moving your org rapidly towards better security and compliance but adding an unknown amount of reliability risk even as you seek to reduce it? Or lean towards preserving the steady state status quo and do too little, and risk getting slammed by fines ... or worse (Stuxnet anyone)? Often there's a middle path you can construct that gives you a nice balance of risk and reward, but I'm not sure that's the case here. But whatever you choose, the rest of us on this blog appreciate the tight spot you're in and will do as much as we can to make your world a little simpler.