Please Remain Calm: My Metcalf Substation Physical Security Take-Aways

Valentines Day update - Two more good links have surfaced for you since I wrote the original post a few days ago:

A PBS Interview with Jon Wellinghof and Mark Weatherford 

A 3rd WSJ article, this one largely a counterpoint to the more FUD-oriented first one

----

It's been nearly 10 days now since the Wall Street Journal published its big story on the attack on a transmission substation outside Silicon Valley in California.  Since then, the media, keying on words like "assault, military-style, terrorism" have had a pre-apocalyptic field day.

So in my own way, I've been running a counter-alarmism campaign when speaking with the press as well as with infrastructure security experts about to go live on one of the hysterical "news shows."

My main points are:

• This attack was significant but it didn't cause a blackout
• So be concerned, but don't overreact
• You can thank the hard work and preparation by Pacific Gas & Electric (PG&E) for at least 2 things: 1) rerouting energy flows so there was no perceptible customer impact despite the loss of many transformers, and, 2) getting the substation fully back on line within one month
• This was a great opportunity for utilities to refresh their physical security policies, and that's what they're doing right now
• Utilities are already taking concrete steps to deter this type of attack, including: erecting screens or walls to block a would-be shooter's view of his/her intended targets, inviting citizens living near substations to call their utilities if they see something suspicious, in the spirit of the "if you see something, say something" transit security campaign, and looking at the transformer stockpiling and loaner program 

Security FUD Alert: Flame On. Flame Off. Flame Out.

Here we go again, and this one is not (energy) sector specific. It's more geo-specific ... see: Middle East and North Africa, at least for now. This is a clear-cut case of marketing security through fear, uncertainty and doubt (FUD), and using the press's predictably Pavlovian response to maximize impact.

Depending on where you fit in the cyber food chain, maybe you like it, but I'm sick of it. Sick of it, I say. And I'm not going to take it anymore! (Yeah, right)

Here's the opening salvo fired on March 29 by InformationWeek (and many others), giving you the fever-pitch, straight up horror story, no chaser:

Step aside, Stuxnet: Newly discovered espionage and information-gathering malware known as Flame ... appears to be even more sophisticated than the Stuxnet.

And with that we were off to the races. Just about every IT, cybersecurity and even mainstream media outlet picked up and broadcast the story in the first 24 hours. No questions asked it seemed.

Then along comes CSIS Senior Fellow James Lewis, two days later, with something quite a bit more tempered:

Flame is not a weapon, it's not the most sophisticated, it's not really that new, but it might be part of a large battle shaping up over the future of the Internet. Cyberespionage happens every day. This should not be news.

With that, Lewis definitely helped bring the hysteria down a notch or two. Much appreciated, Jim.

Finally we've got what I hope becomes the final word on this event, in the form of a post from my colleague and friend, cybersecurity expert Chris Poulin of new IBM company Q1 Labs. Chris begins:

I’m not so impressed: I believe we’re seeing the beginning of a long line of copycats, and Flame is a klunky primate of the next stage in the evolution of advanced malware; it’s just another generation in the APT ontogeny.

And then Chris turns the mike over to IBM X-Force's statement on the subject:

At this time, Flame appears to be limited to a very small geography, primarily certain countries in the Middle East, and does not appear to autopropagate. This malware appears to be highly targeted and designed to infect a minimal number of specifically targeted individuals. Consequently, the immediate threat from this malware, in the general network population, remains very very low despite its high profile in the press.

Like the way that piece finishes: very very low threat vs. high profile in the press. Succinctly said, and to me, what should be the nail in the coffin of this ridiculous escapade.

Security professionals in the electric sector and elsewhere: how are we going to be taken seriously by senior business leaders if some of us, even a small percentage, keep using misleading, inaccurate and gratuitously sensationalist methods to try and drum up more business? It's embarrassing.

I don't need to tell you there's plenty of business out there for vendors who play fair and square. Don't cry wolf unless there's a wolf. Don't say the sky is falling unless it is. Be good: important businesses and other organizations need your help, but they won't let you help if they don't trust you.

Image credit: Wikipedia

Woolsey Ominous at GigaOm re: Smart Grid Security

I'm a fan of former CIA Director and energy security "Green Hawk" James Woolsey and find myself on the same page at least nine times whenever he voices ten opinions. But at a recent energy tech conference he weighed in pretty heavily against electric utilities taking security challenges nearly seriously enough.

Two links coming at you. In this one, from the SmartPlanet blog, the primary impression seems to be that Woolsey wants to move the US as quickly as possible to more distributed forms of generation as a means of diversifying and decentralizing our sources of power.  Hard not to agree there's goodness in that idea; it's the matter of expeditiously implementing that type of change on a large scale that's a grand challenge.

But in this post, from conference host GigaOm, it sounds more like Woolsey has an ax to grind against the utilities. This is a paraphrase I'm sure, but the point gets through:

Right now they’re more concerned with adding fun new features, but it won’t be so fun if the electric grid goes down for a few days.

"Fun new features" doesn't sound like the goal of any utilities I've been in contact with. Not even close. I assume that's his attitudinal short hand for modernization activities a la the Smart Grid. But nobody I've talked to is doing anything for the fun of it: not Smart Meters, not AMI networks, not distribution automation, not demand management, not efficiency.

Woolsey's been known to call the Smart Grid "dumb" and belittle new capabilities as if they were gadgets for consumers (e.g., saying people can turn down their AC with their phones on hot days, for instance, and then China-baiting by saying somebody in Beijing or similar can also reach your AC the same way).

To me this sounds like another voice in the growing chorus for more Federal regulation along the lines of the 2012 Cybersecurity Act. NPR had decent, relatively balanced feature on the looming legislation this morning, HERE. And we discussed the pro's a little and the con's a lot of this type of action on an SGSB post a few weeks ago, HERE.

I'm sure most would agree that improving the overall security of the electric system is desirable and doable. For example, perhaps adding a few carrots to the menu that's currently comprised of sticks might foster some better results.

While I'm confident their intent is constructive, IMHO, I'm not sure government is equipped to bring about the types of change Woolsey, CSIS's James Lewis, and many others think (or hope) they'll achieve through legislation. It would be great to see more utilities start taking the lead on this topic and control their own destiny, versus having it set for them.

Weekend Youtubing: "Smart Meters are not a Killer Fascist Conspiracy"

I have found the ultimate antidote to the sum of all Smart Meter fears in the form of this video. Before you start it, however, please note that it's really not entirely safe for work. It has many funny bits but a few naughty bits too. Ok, you've been warned ... now enjoy.

Just when you thought it was safe to Calculate: More "Incalculable" Smart Grid Security Doom for your Consumption

It might be a form of Tourettes, sorry. But every once in a while I feel compelled to shine a harsh light on articles that go too far or way too far in in the FUD department. Especially those from reputable publications.

What was Said

Here are a few selected citations from the first part of the less-than-soberly titled article in question:

• "Internet-based terrorists would be capable of causing blackouts on the order of nine to 18 months."
• “The dollars are incalculable.” 
• “There’s some percentage of utilities out there that just don’t take this seriously.” 
• "Energy companies including utilities would have to increase their investment in computer security more than seven-fold to reach an ideal level of protection."  

SGSB Non-Scientific Analysis

If the attacks come from bad guys based on the Internet, then the outages could be 9 - 18 months. I see. And the money at stake is so large as to be impossible to estimate. Thanks to recent debates over the US budget and deficit, my eyes and ears are now well accustomed to figures of $15 Trillion and beyond, but clearly the damages from hacking the grid must be even higher. "Some percentage" ?!?  You mean, a non-zero percentage that's so high as to be incalculable, right? And although I've never used the term WTF in this blog before, in the murky world of cyber attack and cyber security, WTF is "an ideal level of protection"?

In case you feel like I'm manipulating you, you can read the whole piece HERE. But suffice it to say, do we really need this? Are these types of "studies" and "journalism" doing much to advance thinking and spur action on securing the grid, or rather simply aiming at inciting panic?

I'll try to keep from blurting out what I really think.

Apocalytic image courtesy of PSD Collector

Notes from Smart Grid Consumer Collaborative (SGCC) Privacy Panel at Distributech

Just a couple things for you here related to privacy. First, here's a link to the good organization that sponsored this event, the SGCC.

One of my co-panelists from a Texas utility brought up a great point I thought ... a challenge that's facing most utilities these days, when she said that a big challenge for her team is how they can know, with confidence, if a 3rd party really has been authorized (by the customer) to access their data. That's a part privacy, part security question, and I'm going to have to ponder that one a bit, and maybe bring in a larger brained colleague or two.

So why does the SGCC need to exist?  First, it funds the research that provides a wealth of great consumer and marketing data to utilities, regulators, and other interested stakeholders. You can click HERE to get their 2012 State of the Consumer report (brief registration required).

But here's another reason, and we talked about this a little on the panel.  It's because absent a sane and sensible, reality-based organization like SGCC getting the facts out, many consumers might be swayed by the fear, uncertainty and doubt (FUD) they're exposed to in the mainstream media as well as in newer channels like Youtube.

This video you're about to see has been watched 1.5 million times, and during its 4 minute run-time the narrator calls smart meters" "power company surveillance devices" and closes with what has to be one of the greatest pieces of alarmist hyperbole I've yet come across. I think you'll like it too:

Those friendly guys on the sidewalk (utility servicemen and women) told me they plan to put a smart meter on every house in America. If they do that, it will no longer be America.

Jeez Louise. Good night America. Good night and good luck. Here you GO.

-----------------------------

And just in, here's a great reader response to the smart meter scare video above:

You’d think there would be more of an outcry over the fact an ISP can see everything they do online, mobile phone carriers can see every incoming and outgoing call and SMS, triangulate their global positions, etc., traffic cameras and OnStar know where their car is at all times, and yet they are worried about someone being able to see their energy data? Maybe opponents should just build their own private power plants and take themselves off the grid completely.

The day may come to pass when that last suggestion is feasible for the mainstream. But for now, your local utility is still far and away your best bet for large quantities of reliable and reasonably priced electrons. Why not help them as they help you, by letting them upgrade equipment to improve their own operations, and serve you and your fellow customers better? I'm just saying ...

Follow-up on Illinois Water Pump Hack Case

This isn't pretty, but it would be good if you knew the whole, emerging, story. My recent post said it wasn't an international cyber attack ... or a cyber attack at all, and that we had been through yet another round of grid security FUD.

But the truth seems to be worse that that. I've got a fuller picture now, having had some contact with Joe Weiss who is, for better or worse, in the thick of it. Here's yesterday's post from his Unfettered Blog:

This story would be funny if it wasn't so scary. Wired magazine has broken the real story (or the latest iteration of the real story). The link is here. So it wasn't evil hackers from Russia after all. From the sound of it, more like a Keystone Cops fire drill. Nobody checked with anybody. Lots of people assumed things they shouldn't have assumed, and now it's somebody else's fault and we're into a finger-pointing marathon.

Securing our infrastructure is complicated and tough enough as it is, without self-inflicted wounds of this type. From what I could see, the water pump control system in question was a complete security mess, connectivity and configuration-wise. It's connection to the web easily visible with Shodan.

Don't know Shodan yet? You should. Seriously. Here's a nice intro from John Matherly on it. If you're an asset owner and you can see your system on Shodan, you've got some work to do. 

And if you're part of a government or industry org charged with getting information out to help keep owners and operators appraised of threats, please do a great job. We're depending on you.

Security Scare Tempest in a Water Pump

There's an adage that goes something like this: think before engaging mouth. Though sadly I'm not always successful, I try to adhere to a modified version of the same principle: wait a while before posting on breaking (and especially alarming) news.

This approach paid off again, as the facts are now officially available. Here's what you need to know about the recent, widely-reported water utility control system attacks ... from the US Industrial Control Systems Cyber Emergency Response Team (ICS-CERT):

After detailed analysis, DHS and the FBI have found no evidence of a cyber intrusion into the SCADA system of the Curran-Gardner Public Water District in Springfield, Illinois.

and furthermore ...

There is no evidence to support claims made in the initial Fusion Center report – which was based on raw, unconfirmed data and subsequently leaked to the media – that any credentials were stolen, or that the vendor was involved in any malicious activity that led to a pump failure at the water plant.  In addition, DHS and FBI have concluded that there was no malicious or unauthorized traffic from Russia or any foreign entities, as previously reported. 

So what can we/you do?

At this time, there are no specific recommendations other than to ensure you are following security best practices. ICS-CERT recommends reviewing Improving Industrial Control Systems Cybersecurity with Defense-in-Depth Strategies.

So it's time to once again to get back off the ledge and go back to work. Thanks to various Big Blue and other industry colleagues who helped keep me up to date on this. For the full ICS-CERT note, click HERE.

Photo credit: Tim Parkinson at Flickr.com

This Week the Economist Loves and Hates the Smart Grid

I confess I typically love The Economist magazine. Its tempered and wide-ranging world news reporting and "tough love" takes on the US culture and economy form a nice middle path at a time when many media outlets have gone decidedly left or right.

But while it's unusual for me to find much fault with their news, the opinion piece in this week's issue "Reliability of the Grid: Difference Engine - Disaster Waiting to Happen", about the recent San Diego outage and the current state of the grid really rubbed me wrong.

By now you probably know the drill:

What is rarely mentioned in all the proselytising about the smart grid is that it adds a vast layer of hackable points to the network—some 440m by 2015, according to Lockheed Martin’s Energy and Cyber Services. Every smart meter in the home will be a hackable device. The same goes for all the routers at substations. As the saying goes, if you can communicate with it, you can hack it. Today, you can cut off the power to someone’s home by shinning up the nearest electricity pole and throwing a switch at the top. Once smart meters become widespread, you will be able to do that remotely, from the far side of the world.

Proselytising? Jeez. Security challenges are "rarely mentioned"? Yeah right. This blog's primary mandate is countering, in its own modest way, the overwhelming ratio of FUD based Smart Grid scare articles with ones that tell a fuller albeit less dramatic story. And thank you, large defense contractor, for adding fuel to the fire (not). The author of this Economist piece went back almost a year to find a FUD-soaked interview with a now departed Smart Grid security practice manager for the 440 million hackable points factoid. There's more I could say about this excerpt and the rest of the article but let's move on. This is supposed to be a short, readable post after all. Get in, get out.

In a piece dated one day later, September 17th, titled "Energy in Japan: Out with the Old" we get the counter argument for a Japan recovering from Fukushima :

Japan needs a smarter grid, with electricity prices that vary according to demand. Power should cost more when demand is high and less when it is low, giving people an incentive to run the washing machine in the middle of the night. It should also be simple for new producers of electricity—from clever start-ups to big industrial firms—to sell power back to the grid.

Nice, but oh so different in content and tone. So what's your ultimate recommendation, Economist? Should we freak out and do our best to scuttle all local, regional and national Smart Grid initiatives due to the looming horrors you describe in article 1? Or should we keep our heads on straight, and build out the Smart Grid for the sound economic reasons you give in article 2, while working overtime to ensure it's as safe and secure as possible? Inquiring minds want to know.

Photo credit: Steve Snodgrass on Flickr.com

Silly Smart Grid Security Headline Winner

Here it is: "Survey: 77% of IT Security Professionals Concerned about Smart Grid Cyber Security"

Question: What's going on with the other 23%?

In my experience (and probably yours as well), "IT Security Professionals" are nothing if not concerned ... about almost everything. Maybe the relaxed 23% taking the survey didn't understand the question. Or maybe they didn't bring a #2 pencil.

Well, at least the writers didn't invoke the usual FUD hysterics:

• Cyber Pearl Harbor
• Armageddon
• Apocalypse
• Alarmed, Alarming, etc.
• amd of course ... Cyber 9/11

Compelling (not) full article HERE.

Why I am no Fan of SciAm's recent "Hacking the Lights Out"

For three reasons, primarily:

1. Misuse of the term "Hacking." The man on the street may have trouble using words correctly from time to time, but Scientific American is supposed to know better. Especially with terms, like hacker, that are clearly loaded. Hacking, by the way, used the proper way, doesn't constitute a bad thing. To the hacking and security conscious community, it's more like a creative (and often good) thing. This headline is not helping.

2. Can't read whole article and it costs $7.95 to buy the whole issue. And I don't see an option to buy just the article for less. IMHO that's way too much mula for one article by today's standards.

3. OK, the first two are really small potatoes compared to this one. How many times do I/we have to say it? Enough with the FUD mongering. Tabloids and other lower forms of journalistic life: from them I expect anything. But SCIAM, for me, anyway, is something greater ... better. Or at least I thought it was.

The "In Brief" section on page 1 lets me know up front they're going to discuss problems and threats, but it also says it's going to end with how security is being "ramped up". Fair enough.  I definitely want to hear about what the good guys are doing so our lights don't get "hacked out". But if you get a chance to read the whole article, you'll be surprised by how little time it spends on proactive, defensive measures being taken. My non-scientific estimate of FUD-to-what we're doing is about 9 to 1.

I want more balance. I want less alarmism. That's all I want. You can read the first page HERE.