Do Utilities need a Security Operations Center (SOC)?

Of course, it's presumptuous for me to presume to know what work be best for any given utility. I can only work from generalizations of the industry as a whole, so please don't take this the wrong way.  

But yes, I most certainly think they do. And a CSO as well. I support anything that can make security a more tangible, centralized, measurable and manageable enterprise function. But of course you already know that.

However, it's not just me. Read THIS, from Dark Reading. Before that, though, a couple of snippets you may find useful.

After you decide to create a SOC ...

A good next step is to create the position of chief security officer or chief information security officer to place responsibility in a single executive-level employee, says Doug Graham, a senior director of information risk management for EMC. Putting the responsibility for security in a single position can help focus an organization's security efforts.

And according to Nicolas Fischbach of London-based Colt Telecom Services ...

As the security initiative develops, a company will typically seek out better visibility into what is going on in its network. Many companies do not have a full inventory of their information assets, and embarking on a program to create a security operations center can be enlightening.

Fischbach also offers this zinger, which may be counter intuitive to some folks:

The first reason to have a SOC is not to do security enforcement, but to get visibility into your environment.

After all, you want to know your weaknesses before others find them ... which can lead to unhappy things like THIS.

Full Disclosure from 2012 Distributech's Keynote Security Panel

It's fun to connect with and catch-up with energy sector security friends, and not always at security conferences. I think we all get a kick out of seeing each other and then dispersing back out into the world to promote the cause and fight our battles in all the different ways we do it.

In fact, it feels a little more special when gather inside a larger conference context, which without a doubt is what you get at the mighty annual Distributech, which took place this year in sunny San Antonio, Texas.

So, enough chit chat. Let's dive into what was discussed on Thursday morning by these folks. Moderator Mike Ahmadi of GraniteKey expertly led a panel of experts on the topic of Security Standards, including:

• Bobby Brown, Enernex 
• Alan Rivaldo, Texas PUC 
• Nate Kube, Wurldtech 
• Darren Highfill, Man of Many Hats 

The guys covered several different topics in depth, including security metrics, vulnerability handling in IT vs. OT, social engineering, and perhaps, most provocatively, security information disclosure ethics and ramifications. Below find a few highlights for each one:

Metrics and Measurement

• In the shadow of Basecamp (which we'll get to shortly), trying to gauge industry progress on security or lack thereof, Mike asked: "are products getting better?" and the response surprised some of us I think. Nate, who has been testing grid products and systems since he was knee high said "absolutely!"
• Others chimed in that, slowly but surely, increased awareness has raised the bar for what's expected from vendors. Sometimes it's because utilities' RFPs' demand it, other times it comes from the vendors themselves. Altogether it's certainly too slowly for many of us, but the consensus seemed to be: tangible improvement is happening out there
• Darren introduced the new DOE RMMM (in early development), referenced other maturity models and frameworks, and he and the panel seemed to contend that all of these, to a greater or lesser extent, help organizations baseline and roadmap their security functions and goals ... and who wouldn't want that!
• Bobby Brown got some laughs (from me, anyway) when he likened the concept of security maturity standards for SG products to the carnival sign we all know that says "You must be this tall to ride this ride"
• Nate praised an audience member's phrase: "at the speed of Metasploit". This set the stage for the later discussion on disclosure. (There's more on the Metasploit vulnerability and exploit development framework HERE if this is your first time hearing the term.)
• Much to my delight, much was said about metrics and measurement in the early going, as we moved back and forth between contrasting the development and evolution of standards and guidelines (e.g., NERC CIPs, NISTIR 7628, IEC 62443 2-4, etc.) with demonstrable improvement in the security posture of utilities

Vulnerabilities in IT vs. OT

This may be obvious to many folks, and I've heard it mentioned quite a bit myself especially concerning meters. But the point was made that in the IT universe, one of the primary modes for dealing with newly surfaced vulnerabilities as well as new types of threats, was rapid change. Rapid change of hardware (we all want the latest gadgets, laptops and servers) is facilitated and driven by customer expectations a refresh on these items every few years or so.

And we see even more rapid change in IT software, as patches to some systems are generated once a month, once a week or pretty much any time. We not only tolerate this pattern, we've come to expect it as a natural part of using the latest and greatest (and safest) software.

That of course brought us back to the OT part of our world, and its intrinsically different set of economics, values and certainly, hardware and software lifecycles. For many good reasons, the systems that support our operations centers, generators, transmission and distribution functions, to include both the hardware and the software, have simply not been built to accommodate frequent change. 

And the culture which wraps around these systems, both the users and the suppliers, is still largely hard-wired to make decisions based on comparatively very lengthy spans of time elapsing between changes.

According to Darren, factors that play into the longer OT hardware and software version lifecycles include:

• How a system is built
• How systems around that system are built
• How we use these systems

And a question arose: are systems that are being designed today looking like they're more able to facilitate faster change cycles? Don't think we arrived at an answer on that ... and that means the answer might be "no"

Social Engineering

The panel got a question from an attendee on social engineering, that is, using plain old people skills (e.g., charm, friendliness, charisma, urgency, faux credentials, etc.) to gain physical access to secure areas, access control information, system configuration information, and just about anything else.

All agreed that typical utility workers' (stereotype to follow) inherent goodness and sense of trust and helpfulness made the energy sector more susceptible to this type of threat than say financial services on Wall Street, where (only slight exaggeration to follow) everyone is mean, greedy and suspicious of everyone else

One of the panelists from a testing org said social engineering is 100% whenever they use it (ouch). Though the same person that social engineering assessments often one of the first services lined out by a utility when negotiating a contract for a comprehensive assessment.

Allan Rivaldo, the Texas PUC representative, after he made it perfectly clear that his statements made on the panel were not necessarily representative of his org, followed by saying that Texas takes insider and social engineering threats very seriously.

Disclosure and Information Sharing

Someone dropped a bomb (of a question) near the end. The panel was asked what it thought about the recent public disclose of PLC/SCADA vulnerabilities in the OT products of half a dozen vendors, to include the attack code for each crafted in Metasploit. 

While it seemed like most panelists believed that Dale Peterson of Digital Bond had acted with good intent: to speed up the remediation of the vulnerabilities by their respective vendors, there was substantial disagreement on whether this approach was justified and on whether it would induce the result Peterson said he sought.

One panelist contended that this action was necessary and valuable for "shining a light" on a broken process related to how DHS's ICS Cert works with vendors to resolve known vulnerabilities. The point being, I think, that following the official policies, many vulnerabilities go unremediated if the vendor provides a reason for leaving the vulnerability alone.

But another said that the Basecamp project researchers' unilateral release of vulnerability details and exploits did little except increase the level of risk to asset owners.

The thing that got me was that, knowing the guys on the panel as well as I do, knowing that they are all men of extremely high intelligence and good will, and that they only want what's best for the community, I was really surprised that they disagreed substantially on the issues that the Basecamp disclosure episode surfaced. 

Clearly this is complicated stuff: ethically, technically, culturally. But I think there's no doubt that our thinking is maturing in some respects, and that the industry community, both the users and the vendors, is responding. It will take a long time for Basecamp to fully play out. Hopefully we'll mainly agree, when it does, that it had a net-positive affect on the electric sector's security posture.

A Brief Note to IBM Colleagues apres Distributech 2012

I feel compelled to say that, though for several good reasons I rarely discuss IBM or IBMers on this blog, I'm going to make a brief exception because of the experience I just had at an annual electric sector conference where, as usual, IBM had a big booth.

One can easily feel lost in a such a huge company; this was clear to me when the tiny but beloved start-up I worked in for 6 years was acquired by Big Blue 2.5 years ago.

For those of you who've had a start-up experience or two, you know how close you can get to your teammates. The blood, sweat and tears experiences you share can't help but bind you together into something not much different than a close family.

I'm a nostalgic person, so seeing comrades from that company disbanded, either blending into different organizations in IBM, or else leaving altogether for different opportunities, was sad and difficult.

But now, after having "put faces to names" of people from around the country and around the world I speak with nearly every day but have never met in person, and reconnecting with others I've encountered before at previous conferences and on customer visits, I feel a similar and familiar sense of connection.

Many of these folks, besides ranging from somewhere between bright and brillant in intellect (and skewed towards the latter), also have hearts of gold and work their butts off to make things good happen for the company, its customers and partners, and their colleagues. I won't name names, but I feel lucky and proud to have the opportunity to work with so many of them.

As for security, several IBM energy sector security gurus and I responded to some wide-ranging security, privacy and compliance questions throughout. I count these guys as friends, and we had a great time hanging out together.

And finally, check this out: our teamwork seems to be paying off as IBM was just listed as one of the very top Smart Grid security firms in the business. We're all pretty darned happy for that recognition. And this announcement, made at the conference, describes new work IBM is doing with transmission provider Velco in Vermont to improve substation communications, with a good dose of cybersecurity, of course!

Image credit: IBM SmartrEnergy

A Runner's Ode to San Antonio's River Walk

Prefatory note: if you only want to read about the Smart Grid and/or security, you'll want to skip this post.

Because it's only about how I came to an electric sector industry conference, and, running sneakers in hand (so to speak), fell in love with an amazing concept, that's equal parts hydraulic engineering, design, landscape architecture, and xeriscaping, all coming together to express a colossal and coherent artistic vision.

That's the River Walk. which you can read about here on its official site, or for something a little less promotional, here's its page on Wikipedia. Many folks pass through quickly and think it's just a glittery and gimmicky place to which one comes to consume a few mariachi-accompanied margaritas. Oh how wrong they are.

To a native Bostonian such as myself, the first and best comparison, I think, is to the work of the landscape architecture rock star of his age, Frederick Law Olmstead and his fantastic Emerald Necklace.  Of course, the two projects are in some ways nothing alike, separated as they are separated by at least a century and two thousand miles of latitude and longitude.

But for me, it's like Olmstead drank a shot of picante sauce (mild, not too spicy), chased it with a little citrus, guac and mole, and then, in an ecstatic Tex/Mex vision, went right to work. Of course, as Wikipedia reveals (and some locals just know), it wasn't Olmstead or any other city-slicking easterner who conjured up the River Walk, but rather San Antonio native and architect Robert Hugman, who, with a little help from mother nature and the WPA, got this thing off the ground.

In 2012, though I understand one wouldn't want to swim in it, let alone drink it, the walks and grounds are virtually immaculate, and several species of exotic birds seem to enjoy calling it home. On my third and final run in as many days, as I approached a large highway bridge, I came upon the most amazing school of dozens of colorful fish, each about 5 feet long and floating below the bridge but well above my head, suspended by thin wires, transforming an otherwise bleak urban landcape into yet another place of wonder. The whole creation is full of subtle and sometimes less than subtle touches like this.

All I can say is I plan to return, whether or not work takes me here again or not.

Photo credit: Mike Tex on Flickr.com

Attention Electric Sector: Wired Reports on Basecamp - SCADA Exploits in the Wild

Several vendors of PLCs and other equipment related to grid operations, in a study described in a recent edition of Wired's "Threat Level" blog, have had their wares probed by a team of experts led by Dale Peterson of Digital Bond, a respected boutique energy-sector control system security shop.

Before saying more, I keep going back to the post called the Value of Black Hat for Smart Grid Security, and maybe now also the Travis Goodspeed Smart Grid Skunkworks piece, because they both showed security technologists trying to spur vendors into action to improve the cybersecurity characteristics of their grid products by describing and sometimes demonstrating vulnerabilities they've found to audiences of cyber security professionals.

This is different, however. Saying they were concerned that their findings might be downplayed and/or ignored by the vendors in question, this time the Peterson-led researchers not only identified the numerous vulnerabilities, but they developed the attack code required to take advantage of them using a tool called Metasploit, and they didn't stop there. They also made the exploits available to the general public without giving the vendors or DHS' ICS Cert a chance to intercede.

As Peterson puts it:

... a large percentage of the vulnerabilities the researchers found were basic vulnerabilities that were already known to the vendors, and that the vendors had simply “chosen to live with” them rather than do anything to fix them.  Everyone knows PLC’s are vulnerable, so what are we really disclosing? We’re just telling you how vulnerable they are.

I definitely have mixed feelings about this. It's certainly raising the stakes to a whole new level. Utilities probably need to double-check their assets to see how many of them match those in the study, and see if there are any vulnerabilities they didn't know about previously. Chances are most if not all have mitigating strategies in place already that should cover them ... but still.

The vendors identified in the report are likely in turmoil as result of the report, and my guess is this topic is going to be owned by their lawyers for some time, if not from now on. And that might mean that instead of accelerating remediation efforts by vendors, this action may contribute to an unwitting slow-down. But I don't really know, and we'll all have to see how this plays out.

On the plus side, the research has led to some new products and plug-ins for utilities that can simplify the job of identifying insecurely configured control systems. Not sure if they'll trust them enough to use them, but maybe.

That's it for now. My highest value on the blog is accuracy. I would be happy to get reader clarification if I've garbled this somehow. Thanks and stay tuned.

BTW: You can read the full Wired article HERE.

Photo credit: tallkev on Flickr.com

Notes from Smart Grid Consumer Collaborative (SGCC) Privacy Panel at Distributech

Just a couple things for you here related to privacy. First, here's a link to the good organization that sponsored this event, the SGCC.

One of my co-panelists from a Texas utility brought up a great point I thought ... a challenge that's facing most utilities these days, when she said that a big challenge for her team is how they can know, with confidence, if a 3rd party really has been authorized (by the customer) to access their data. That's a part privacy, part security question, and I'm going to have to ponder that one a bit, and maybe bring in a larger brained colleague or two.

So why does the SGCC need to exist?  First, it funds the research that provides a wealth of great consumer and marketing data to utilities, regulators, and other interested stakeholders. You can click HERE to get their 2012 State of the Consumer report (brief registration required).

But here's another reason, and we talked about this a little on the panel.  It's because absent a sane and sensible, reality-based organization like SGCC getting the facts out, many consumers might be swayed by the fear, uncertainty and doubt (FUD) they're exposed to in the mainstream media as well as in newer channels like Youtube.

This video you're about to see has been watched 1.5 million times, and during its 4 minute run-time the narrator calls smart meters" "power company surveillance devices" and closes with what has to be one of the greatest pieces of alarmist hyperbole I've yet come across. I think you'll like it too:

Those friendly guys on the sidewalk (utility servicemen and women) told me they plan to put a smart meter on every house in America. If they do that, it will no longer be America.

Jeez Louise. Good night America. Good night and good luck. Here you GO.

-----------------------------

And just in, here's a great reader response to the smart meter scare video above:

You’d think there would be more of an outcry over the fact an ISP can see everything they do online, mobile phone carriers can see every incoming and outgoing call and SMS, triangulate their global positions, etc., traffic cameras and OnStar know where their car is at all times, and yet they are worried about someone being able to see their energy data? Maybe opponents should just build their own private power plants and take themselves off the grid completely.

The day may come to pass when that last suggestion is feasible for the mainstream. But for now, your local utility is still far and away your best bet for large quantities of reliable and reasonably priced electrons. Why not help them as they help you, by letting them upgrade equipment to improve their own operations, and serve you and your fellow customers better? I'm just saying ...

Conference Alert: European Smart Grid Cyber Security

It's going to be in London on 12 and 13 March 2012

Great speaker line-up with experts from both sides of the pond, includes:

• Office of Cyber Security and Information Assurance, Deputy Director, Mike St John Green
• European Commission, Policy Officer, DG Information Society and Media, Alejandro Pinto
• National Information Security Authority, Israel , Director, Erez Kreiner
• Enisa, Program Manager Resilience and CIIP Program, Dr. Vangelis Ouzounis
• Queen’s University Belfast, Director of Research, Professor Sakir Sezer
• NIST, Chief Cyber Security Advisor, William Barker
• Con Edison New York, Smart Grid Project Manager, Patricia Robison
• Swissgrid ag, TSO Security Cooperation, Senior Advisor Operations, Rudolf Baumann
• EDP Energie SA, Information and Cyber Security Officer, Nuno Emanuel Pereira
• Sirrix AG security technologies, Project Manager, Michael Gröne
• GDF Suez, Information Security & Business Continuity, Phillip Jones
• IOActive, Vice President, Services, David Baker
• Institute for Information Security, Executive Director, University of Tulsa, David Greer
• Alliander, Senior Consultant Intelligent Netbeheer, Frans Campfens
• Saudi Aramco, Information Protection Specialist, Saad Alhowaymel
• Zigbee Alliance, Security Working Group Chair, Robert Cragie
• Alliander, Privacy & Security Officer, Johan Rambi
• Energy Networks Association, Head of Strategic Telecommunications, Mark Simpson
• Riscure, Director Embedded Technology, Job de Haas
• SAIC, Chief Cyber Technologist, Gilbert Sorebo

Click HERE for more information.

Photo credit: Matt from London on Flickr.com