Grid Attack Simulation Just Completed: “It was More Severe than Anything We’ve Drilled"

So said the President and COO of AEP subsidiary Southwestern Electric Power Company, of scenario she and her people faced during NERC's second GridEx exercise.

Sounds like NERC CEO Gerry Cauley and his team brewed up something pretty potent this time.  Heck, it even included 7 deaths and 150 casualties ... in quotes of course.

NERC will issue an "after action" report including objectives, what actually happened, lessons learned and recommendations as soon as they get some sleep.  In the meantime, this account from the NY Times Matthew Wald is pretty darn good.  You can check it out HERE.

Photo credit: The Guardian

Because Excercise is Good for US, GridEx II is Coming

In case you've been wondering what kind of shape our North American grid incident response and information sharing system is in, now's your chance to find out.  You can click HERE for more details on what's coming up and register to participate if you're an asset owner one of the other types of orgs that have an official role to play.

• When: 13-14 November
• Where: North America
• Dress: Business Casual

While you're here, here are a few other items of possible interest:

• You can read a decent GridEx II intro HERE, from the NYTimes
• Findings and recommendations from the first GridEx begin on page 10 of the After Action Report
• Click HERE for news on a recent disruptive control system cyber attack on a tunnel traffic system in Israel

Poster image courtesy of Crossfit.com

NERC CIPs Catching up to iPhone Version Numbers

OK, imagine an auction barker: "Bids opening at NERC version 3, do we have a version 3? ..."

"Yes! How about version 4?" Etc.

Well, according to Honeywell's NERC CIP guru Tom Alrich (of the famous, eponymous and quite helpful blog), it now appears that the next version of the CIPs to which utilities must comply will be neither version 4 nor  5 but rather version 6!

I was stunned as were many of the those in online attendance. Tom explains his reasoning on the EnergySec webcast and much more, which you can see HERE. There's a lot of helpful information for utilities of all sizes dispensed in this hour long piece, with some deep dives into the ramifications of high, medium, and low risk assets.

Now, depending on whether we get an iPhone 5s or 6 next month, it's clear that NERC will not allow the CIP versions to lag far behind.

----------------------------

Attended most of this webinar online, but hat tip to my former colleague and (hopefully) current friend, Nebraskan Dave Hemsath of IBM for sending me the replay.

Photo credit: KCRW.com

SANS cyber security awareness training for eager utility employees ... and their regulators

I recently stumbled upon some excellent online training materials from the well respected SANS Institute that could be quite useful to you and your organization.

In a series of online modules, many of them tailored to the particular needs of utilities, SANS "Securing the Human" courseware seems to be an easily digestible, self-paced way to get important cyber security awareness messages across to a large number of users.

Note: NERC CIP content here is constructed around version 3, so with newer versions now approved by NERC and FERC, SANS will want to update certain modules accordingly. But 99% of the material is right on the mark, and would be appropriate for electric sector personnel outside the US as well.

Wherever you fit in the ecosystem, whether you're an executive or a rank and file worker bee, whether you're in a utility, a regulatory agency, a vendor, or just a user of digital technology who wants to stay safe, recommend you check it out.

---------------

SANS URL:

http://www.securingthehuman.org/utility/index

Grid Security Keynote of Note at May 2013 ISO Conference

Since you can't be everywhere, there's the SGSB (which can).  Former Seattle City Light CISO and current Verizon control systems security ace Ernie Hayden gave a keynote presentation at the recent ISO New England and New York ISO Energy Conference held in Boston, and we've got it for you.

If you don't know ISO, it stands for Independent System Operator, a term which is often used interchangeably with another acronym: RTO, or Regional Transmission Organization. In North America, these organizations are like referees and traffic cops, trying to keep the peace among utilities and ensure the smooth and reliable flow of appropriately priced electricity across multi-state regions.

It's good to see Security get such a prominent platform at a high profile industry event like this. Certainly a sign of the times.  Ernie's slides will take you through the past, 2013/present and future of grid security, and though some of the info would clearly benefit from his accompanying narration, a lot of this works quite well as is. And if you really want the audio, then I'm sure Ernie will agree to come to you and do it again, as long as you treat him right.  URLs below.

-----------

Ernie Hayden deck

http://www.isoenergyconference.com/pdf/Ernie-Hayden-Keynote.pdf

Conference home page

http://www.isoenergyconference.com

House of Reps Report Reams Utilities on Cybersecurity

Was trying to capture spirit of Jesse Berst's headline on the same subject:

Utilities to FERC: Take your security measures and shove it

That's not very nice, is it?  I think they toned it down with a later change, but this headline was what was in my inbox in this morning's SmartGridNews.com newsletter. The subject is a recent report published by the House of Representatives that's highly critical of electric utilities behavior to date re: grid cybersecurity.

Moving on! The Wall Street Journal's Rachel King did a fine write-up of recent testimony from the CEO of the American Gas Association (AGA), Dave McCurdy. King began by noting that:

The oil and gas sector faces many of the same cyber security challenges as the electric industry. Yet, there’s one major difference between the industries, both of which need to secure software-based industrial control systems from intruders. There are no regulations governing cyber security among the oil and gas companies.

Helpful Clarifications Still Leave NERC CIP Version 4 Changes Feeling Overwhelming

If your job is to ensure your utility complies with new version 4, certainly you've been scouring info like this for a while now. But if you're a member of electric sector support or regulatory communities, including services providers and state commissioners, it'll behoove you to get a better feel for the massively numerous and often ambiguous compliance hoops through which these folks have to jump.

EnergySec Welcomes NERC CIP Virgin Utilities with Version 4 Briefing

Titled "Get Ready for Version 4" the deck linked at the end of my words has some great and helpful info in it, not just for utilities transitioning from version 3, but for brand new utilities who for the first time come under the tender embrace of the CIPs. To them, all of us in the community say, "Welcome aboard !!!"

An all-star energy and security team, including Steve Parker, new President of the Energy Sector Security Consortium (EnergySec), and Honeywellers Tom Alrich and Donovan Tindill, shared some sobering, cold, hard, urgent facts. Those from Tom Alrich were particularly rich:

Alrich on Distributech's 2013 Cybersecurity Focus Panels

I couldn't make it to the panel sessions but fortunately Tom Alrich could and did. Here's are his short-takes on 3 different panels:

Substation Integration and Automation: The Cybersecurity Landscape is Changing - Didier Giarratano of Schneider Electric discussed Role Based Access Control (RBAC) and how to do good job applying RBAC to the challenges of substations. Anthony Eshpeter of SUBNET Solutions discussed “Complexities of Substation Cyber Security”. He provided a very good, lucid discussion – pointing out the need for solutions like those SUBNET sells but without ever making a sales pitch. Bradley Tips of Cisco addressed “Real-world Deployment of Network Security for NERC CIP Compliance”. A good overview of what CIP requires for a substation these days.

The Quest to Better Understand the NERC CIP Bright Lines

Tom Aldrich and Rick Kaun have written some of the best material we've seen on the topic of the evolution of the CIPs, and Tom has new piece clarifying the lack of clarity of the Bright Lines language.

This is something of interest to all utilities currently busy achieving and demonstrating compliance with version 3, and grappling with what they should do to best prepare for versions 4 and/or 5 coming at them sooner than they would probably like.

Here's Rick's intro followed by a link to Tom's article. Note: you may not like what Tom has to say, but it's better to get this news now than to go ostrich ....

2 Control Systems Metrics Movers/Shakers: Jim Brenton and Joe Weiss

The more metrics the merrier, I say. After yesterday's post on IDC's take on energy sector cybersecurity metrics, let's pivot to control systems, where the three most important things are:

1. reliability
2. reliability, and
3. reliability

... and where what passes for cybersecurity in IT realms just doesn't cut the mustard.

First see this NEW POST from Joe Weiss on how to begin wrapping your head around what control systems metrics could look like.

Then I'd recommend Jim Brenton's PRESENTATION at GridSec earlier this year on a new group that's formed to look at developing security (or should I say reliability and resiliency) metrics for the Bulk Electric System (BES).

OK, that's it, this is a short one. You can go back to what you should have been doing all along.

Security Checklists, Compliance Cultures, and Finding a Better Way

Fixating on responding to a compliance regime is in a sense, like agreeing to not learn. You know how when you're in the passenger seat and even if you go to the same destination a hundred times, if you weren't driving you don't remember how to get there cause you never learned in the first place? Fortunately, the GPS came long just in time to save your butt.

Before I go any further, please note this is not intended as invective against the NERC CIPs. I know why they were made and why we still have to have them. Most of us wish it were otherwise, but if and until the majority of utilities are observed performing the activities the CIPs require (and more) on their own initiative, this is the way it's going to be.

Have to tip the figurative hat again to Ernie H, who once again can be counted on for spotting and forwarding the good stuff. THIS recent Dark Reading article lays bare the downside of a checkbox compliance mindset. In it, I like this analogy from security researcher Lamar Bailey:

[Security] standards are like training missions in video games: They can help you acclimate, but they in no way represent the real game. If you can't pass them with two hands tied behind your back, your need to quit and find another game.

I could see how that might make some folks say Ouch!  Anyway, it prompted me do a search to see if
utilities were embracing or resisting some of the downsides of this mindset. Well, didn't take long before I happened upon this. Imagine you pay the salaries of dozens of folks, as many utilities do, engaged in activities described by one CIP compliance manual this way.

It begins, "Evidence is more than just documentation" and continues:

Demonstrating compliance usually means “corroborating” evidence. In other words, compliance programs should be designed to produce an output of several auditable records for each requirement to demonstrate performance. A documented process is the first part of demonstrating compliance. Additional examples include a log file, a change request form, a visitor log, or an incident response drill attendance sheet. For each requirement, a documented process and at least one other additional record should be available to demonstrate compliance.

There have got to be better ways to run and secure a utility, no? And of course, reading to this point is worth it if it means you get to a gem of an article I posted a link to almost exactly one year ago. Once again, HERE's Stephen Flanagan of FERC, giving a keynote on what he terms compliance cultures vs. cultures of commitment, who even as an auditor, keeps this in mind: "Security, not violations, is the primary focus." Like that.

Tweeting from GridSec conference this week

Howdy from Dallas. This is the evolution of Mike Ahmadi's Smart Grid Security East and West events, which have been running twice a year since the fiest one in San Jose in 2010. Will shoot to summarize key messages in a post when it's over, but also will blurt out the occasional tweet on the fly using the #GridSec hash tag on Twitter.

A Grid Guy's Perspective on James Lewis' Testimony re: the Cybersecurity Act of 2012

James Lewis is Mr. Cybersecurity these days. A colleague (hat tip: Steve O) just sent a note out pointing to a new article appearing front and center on WSJ.com tonight, featuring prominent statements by Dr. Lewis, the Tech Policy Directer of K-Street think tank CSIS.

Two weeks ago I wrote a post that ridiculed as alarmist a few quotes, including one ostensibly made by  Lewis, that appeared on another well known financial media site.

And just last week he testified before a Senate subcommittee about what he likes, and what he finds wanting, in the draft bill that's looking increasingly likely to make it through Congress sometime soon.

You should note that unlike last year's Grid Act which passed the House (HR 5026 Grid Reliability and Infrastructure Defense Act), the focus of the current bill, and therefore of Lewis' testimony, is not energy sector specific. Here's one of his opening sections in which I find nothing not to like:

Reducing risk and vulnerability in cyberspace is a fundamental challenge. In considering this  problem, we have learned through painful experience that market forces will not secure cyberspace and that existing authorities are inadequate for national security and public safety. The list of private sector companies, including technology leaders, whose defense have failed is long and would be longer if all breaches were disclosed. Continuing to use voluntary, market driven approach to this new national security concern is irresponsible and guarantees a successful attack against our nation.

Our sector, of course, has the NERC CIPs. Much derided in some circles, though in my mind a huge improvement over the kind of security we'd likely see from pure "market forces," the NERC CIPS are anything but voluntary. And when versions 4 and/or 5 go into effect, they'll cover many more systems and require more security controls for most.

The 2012 Cybersecurity Act aims to give DHS the lead in securing critical infrastructure and it's unclear to me how it might supplement or complement current the NERC CIPs. More on that later.

Meanwhile, towards, the end of his testimony, Lewis sounds a positive note that quickly turns ominous:

Anyone who tells you that we do not know how to do cybersecurity is sadly out of date. The National Security Agency, the National Institutes of Standards and Technology, and other Federal agencies are pioneering techniques that can strengthen America’s defenses. But while we can require implementation and measure the rate of implementation in the Federal government, there is no comparable ability to measure and secure commercial critical infrastructure. This remains the single largest vulnerability for America in cyberspace. 

So while we have the NERC CIPs, you can take his point about "no comparable ability to measure" critical infrastructure to mean that while audits occur and fines sometimes levied, neither DOE, nor FERC, nor NERC keep track of how the utilities are doing. There's no standard framework that tells us which utilities are doing a great job and which ones are lagging. IMHO that is a problem.

You can read Lewis' full testimony HERE.

And one more thing: on Lewis' CSIS page he also includes a link called Serious Cyber Events. It's a comprehensive list of the most noteworthy known attacks and breaches since 2006 till present. Out of a total of 87 events cited, only 2 involved power systems:

• January 2008. A CIA official said the agency knew of four incidents overseas where hackers were able to disrupt, or threaten to disrupt, the power supply for four foreign cities
• April 2009. Wall Street Journal articles laid out the increasing vulnerability of the U.S. power grid to cyber attack also highlighted was the intrusions into F-35 databases by unknown foreign intruders

2 out of 87 would be a horrible batting average (.023 - yikes!) on any baseball team. But in this game, which really is no game, it's an average I'd like the sector to maintain. So keep one eye on the NERC CIPs and beyond, and keep the other eye on what James Lewis and Congress have in store for us.

PJM CEO Speaks Out on Cyber Security and Resilience

In an interview published a couple of weeks before Christmas, Linda Evers of the excellent Smart Grid Legal News blog conducted a brief Q&A with the PJM CEO Terry Boston and got quickly to the subject of grid cyber security.

PJM, in case you're new to this, is the Pennsylvania-New Jersey-Maryland Interconnection, an RTO that balances power and oversees wholesale transmission markets across thirteen states and the District of Columbia.

When Evers asked the classic "What keeps you up at night?" Boston responded:

Cyber security. It has changed in the last three to four years. It’s no longer just a matter of trying to keep kids out of the system. Making sure we have security built in not bolted on to all of our networks and systems is probably the most important part of what we do. You have to realize this is a new world we’re in. We have to be very diligent, and we need resilience. Resilience is the ability to recover after a breach or intrusion.

Can't help but feel this approach is realistic and fully in tune with the times, especially in light of the numerous cyber security attacks of 2011 that successfully targeted many different sectors.

With or without a forward-leaning CEO, utilities are regulated to think this way to a certain extent. NERC CIP 009 - Recovery Plans for Critical Cyber Assets insists that asset owners makes plans for responding when their cyber systems are under attack, including when they fail outright or come under the control of the attacker. NERC also wants to see evidence that regular practice sessions and exercises are being conducted, though I don't know how detailed and realistic these exercises are. Looking at the language of CIP 009 it appears that an exercise of some kind, once a year, may suffice to get a clean bill of health in this category.

In my mind, connecting the dots from the reliability of cyber systems to the reliability and quality of performance of generation, transmission and distribution equipment and revealing the potential impacts to the utility and its customers is the work required to build the case for bolstering resilience efforts.

Greatly appreciate it when senior energy-sector leadership articulates practical approaches to dealing with always evolving cyber threats. Feels like a great place to start for 2012. 

GridEx 2011: NERC CyberSecurity Exercise is Upon Us

Practice makes perfect ... or at least makes you better.

I mentioned this back in July HERE, now thanks to Dave Dalva of BAH, I can tell you a big exercise is coming up this week, starting tomorrow:

The grid security exercise, scheduled for November 15-17, will test NERC’s and the electricity industry’s crisis response plans, and validate current readiness in response to a cyber incident. The exercise also will serve as an opportunity to enhance collaboration and strengthen industry security processes and capabilities.

Follow this LINK to a bulletin on the exercise as well as a compilation of some of the best grid security presentations I've ever seen, from NERC's recent conference in New Orleans (see Presentations tab at bottom of page).

Results and findings should be available around mid December, and I'll be sure to post material that's cleared for public consumption.

Welcoming Weatherford to his new DHS Cyber Security Post

I've got a note here this morning from National Bureau of Information Security Examiners (NBISE) founder and former NERC CSO Michael Assante. Perhaps there's no one who understands the challenges Weatherford faced at FERC more than Mike. As a frequent advisor to FERC and Congress on critical national infrastructure security issues, few are better placed to know the obstacles and opportunities that await the new DHS Cybersecurity leader:

I would like to extend my congratulations to Mark Weatherford on his appointment as the new Deputy Under Secretary for Cybersecurity for the National Protection and Programs Directorate (NPPD) and am very pleased to see such a capable and experienced leader take the helm. 

Mark has always carried a deep sense of mission into his assignments and in doing so has been able to motivate people, build teams, and mobilize entire communities. His background makes him an ideal choice for the Deputy Under Secretary position as he has experience working across large government enterprises and his most recent post, as the NERC CSO, has prepared him to appreciate the unique challenges involved with cybersecurity and industrial control systems.

At NERC, Mark helped broaden our thinking about cybersecurity and our digitally reliant infrastructures. His vision has pushed organizations to look beyond compliance to develop a comprehensive approach by including system engineering, planning, operations, risk management and security into efforts to secure our infrastructures. Mark’s leadership will help ensure national efforts align with front line reality as our nation continues to modernize our grid to increase productivity and efficiency.

We should look for opportunities to support Mark and the department in the months ahead to achieve greater cyber-resilience in our nation’s critical infrastructure.

Hear hear. Mark Weatherford has now seen how the cyber security policy sausage is made at the state level twice and Federal level once, in a large company, and in the DoD for the US Navy at the beginning of his career.

Sausage making is never pretty. But if you know how it's done, how it can go wrong and what ingredients are required to produce the best stuff, you can do a lot of good. Let's wish him well, and, seconding Mike's call to assist, pitch in wherever and whenever we can. Even with a strong leader, this type of sausage making is, after all, a team sport.

Photo credit: Govinfosecurity.com

Electric sector security evolution: forward leaning exemplars vs compliance-focused knuckle draggers

This is the last of my posts from last week's Smart Grid Security Summit West, held in an unusually damp San Diego.

OK, knuckle draggers may be a little harsh. I apologize. But there may be a whole new approach emerging, to meeting security, privacy and compliance demands in the electric sector, and, depending on where you work when you read this, it's one I think you'll like a lot.

The outlines of a new approach appeared during the security metrics panel on day 1 and continued to resonate till the end of the conference on day 2, and basically it came across like this:

While the vast majority of utilities today seek to achieve an acceptable level of security and risk reduction via compliance with version 3 of the NERC CIPS, and preparation for what looks likely to come from NERC in subsequent versions, a couple of utilities, supported by their CEOs and/or empowered by recent crises, intend to set and implement higher-level security baselines for themselves.

I won't say who they are; it's probably best if you hear that directly from them or infer it yourself. But if these 2 can get the process started, and perhaps coax another 1 or 2 to join them, then they may be able to carve a wide path that many of the precedent-following rest can follow.

Imagine an industry where mere compliance with the lowest government enforced controls is no longer considered a best, or even a good business practice. Wait, this is starting to turn into a John Lennon song. Probably a good idea to stop here, but stay tuned for more on this.

Newsflash! A Reasonably Balanced Article on Grid Security

First of all, kudos to Discovery News writer Eric Niller for penning a relatively fair and balanced piece this week on Smart Grid Security, with a decent, non-alarmist headline to boot. He quotes me a fair amount, but enough about me, it's two of the other quotes I'd like to address.

First, here's one I don't like, attributed to a large and otherwise highly reputable security firm:

One of the more startling results of our research is the discovery of the constant probing and assault faced by these crucial utility networks. Some electric companies report thousands of probes every month ..."

As you know I'm not a big fan of using words like startling in this context, especially in describing phenomena that are not at all surprising, let along startling. Of course utilities' networks are being probed. And it's a good sign they've got the systems and processes in place to be aware of it. 

Go ahead and plug a new PC in and turn on its wifi radio. Within minutes, if not seconds, even with good security controls enabled, that machine is going to come under some serious scrutiny. It's a fact of life these days. Bothersome? Yes. Annoying? Definitely. Startling? Not in the least. Get real, above-mentioned report writer for large and otherwise highly reputable security firm.

This one I like better. It's a straightforward statement from a straightforward person:

What we are doing is laying a new digital infrastructure over the very reliable and sturdy bulk power system. This digital infrastructure provides a lot of new attack vectors into the electrical system that didn't previously exist.

That's NERC CSO Mark Weatherford speaking, and as you can see, he balances the comment about new attack vectors by reminding the journalist (and thereby, the readers of this piece), that underpinning all the new Smart Grid stuff  is a very robust legacy system. A system that's delivered increasing volumes of reliable power to hundreds of millions of customers for a long, long time.

Overall, pretty good work, especially when so much of the popular press delivers, on a daily basis, heaping helpings of unmitigated FUD. You can read the whole piece HERE.

NERC set to Excercise Grid Cyber Security

We all know exercise is good for us, but not all of us regularly act on that knowledge. Well, NERC has seen our flab and is recommending we hit the gym.

NERC is sponsoring GridEx 2011, a cybersecurity exercise dedicated to incident response in the electricity sector in North America. The event will be held mid November 2011, and hundreds of utility companies are participating in various capacities.

You can see the press release HERE and if you work for a North American utility that's not involved yet, you can write NERC's Brian Harrell and he'll get you up to speed fast.

But remember this before you go getting all giddy: no pain - no gain.

Photo credit: Lululemon Athletic on Flickr.com