Smart Grid Security East and the Software Security Panel

Today I had the good fortune of being on a small panel, moderated by Matthew Carpenter, and with a representative of embedded software security provider Green Hills Software. We focused on grappling with how utilities and their suppliers are confronting application layer vulnerabilities not just in key systems, but across their entire application portfolios. Here's a summary of what I think are some of the interesting facts and other points we touched on:

• Application (or software) security is one of the newest (i.e., least mature) security sub-domains in every sector, which means utilities are not substantially further behind in this domain than some of their similarly sized, non-electric utility peers
• Large and very large utilities can have anywhere from several hundred to several thousand applications ... that they know of and track. A somewhat unsettling percentage of utilities don't know how many apps they really have. It's an often neglected form of asset management and some are working hard to figure this out. And some aren't.
• These same utilities often have one-to-two hundred developers in their internal development teams, most who have not yet been introduced to secure development principles, and with SDLC's that fail to leverage current tools that can really help
• Many utilities haven't yet formulated an application security policy, meaning, among other things: they haven't yet determined which types of software vulnerabilities add so much potential risk that they simply aren't allowed to exist in operational systems. Again, some are moving out with security policies that drive helpful behaviors in this area, but the majority (IMHO) aren't in motion yet
• I was asked what my Big Blue company is doing to help in the app sec area, and responded that we're working on three levels: (1) providing app sec training, consulting, services and tools to utilities, (2) bringing the same to vendors who supply software and software-intensive system to utilities, and (3) adding secure development processes to the SDLCs of the products we market to utilities, including those that comprise the Solutions Architecture for Energy (SAFE) framework

One point I meant to mention but didn't is that in the spirit of walk-then-run, before trying to develop policies and procedures to harden the entire application portfolio, many of the utilities we've worked with to date start at the project level with AMI and / or Customer Portal implementations. With AMI, we've seen utilities run application security tests on both the internally developed as well as vendor supplied software with good results. So good, in fact, that some of the related meter vendors, seeing the results, have procured our tools for their own internal use in their SDLCs, which again benefits the utilities when they buy these new, more secure products. And ditto for customer portal projects.

As this was a Powerpoint-free zone by design, in today's session we were just guys talking. But I've been building a short slide deck called "Securing Your Smart Grid Customer Portal" and plan to make it available, via the blog, to attendees shortly after the conference concludes. I think (and hope) you will find it helpful.

I Don't Want to Talk about Night Dragon ...

... but looks like I have to. We're still digesting the energy sector cyber security implications of 2010's attacks on Google + 30 (confusingly named Operation Aurora), Stuxnet and Wikileaks, and now we've got another whopper.

Looks like energy sector, or more specifically, oil & gas companies were the primary target. Here's a short synopsis of the attack techniques used, which begin of course, with one of the most common (and easy to defend) attack vectors:

The attacks began with a SQL-injection technique, which compromised external web servers. Common hacking tools were then used to access intranets, giving attackers access to internal servers and desktops. Usernames and passwords were then harvested and after disabling Internet Explorer proxy settings, hackers were able to establish direct communication from infected machines to the Internet.

In my experience, oil & gas co's generally have more budget to spend on security protections than their electric utility brethren. So if they don't have their cyber houses in order yet against simple stuff like this, then it's quite likely that the same attacks would have breached electric co's as well.

Click HERE for a short article on this, and HERE for the more detailed report by McAfee.

Securing Smart Grid IT Systems

We're halfway to the next Smart Grid Security show (# 3 on IT systems security on June 30) but have started doing some of the preparatory work. Essentially, what this session's going to focus on is the different IT systems (legacy and new) that need to be shored up. (Note: SCADA/control systems are purposefully excluded from this discussion as they are quite a bit different beasts, and we'll cover them in some depth in the not-too-distant future.)

You may ask, why the special emphasis now? Well, until recently and with no offense intended, utilities were an Internet backwater. They were (happily for them) way down on attackers' list of targets, partly because of their reputation as technology laggards, and partly because many of their systems were standalone, or nearly so. Folks we've met who've worked in utilities for decades, as well as those who've helped take care of their technology needs, attest that they've worked un-harassed in relative obscurity, until recently that is.

Emerging Center of the Universe

Now all eyes are on these guys: the press and analysts, Congress, the Department of Homeland Security (DHS), regulators NERC and FERC .... And two groups who more than any other are putting pressure on the utilities to perform, security-wise:

• The aforementioned attackers, who now like what they see a lot more as utilities bring new web apps on-line, begin to aggressively interconnect their systems, and enable two-way communications to/from some of their most important systems, like the head-ends that aggregate much of the incoming traffic from customer systems
• And of course, customers. Long dormant with only the absolute minimum interaction with their electricity providers, thanks largely to the press, customers are waking up and beginning to raise their voices demanding better service and control over fees

Which Systems Need (Better) Securing

In addition to what you can see in the Forrester slide, both the old and the new, there are numerous other types of systems, not the least of which (in importance) are "outage management systems". From our survey of utilities' IT managers and their service providers, we can place all into one of several categories:

• Classic Cobol/Mainframe - As everyone knows, mainframe apps have been around forever and are always just a year or two away from replacement. This will (almost) never change. Many, if not most were developed initially deployed pre-Internet era and therefore security was neither designed in nor bolted on. Formerly protected primarily by their isolation, these old workhorses are becoming increasingly connected as their data (e.g., customer, financial, accounting, etc.) become increasingly important to other systems in a Smart Grid world. What's our advice for securing these systems ... stay tuned
• Client/Server - Most often found in the form of packaged or "commercial off the shelf" (COTS) applications, these include a server component including logic and a database, and client-side software that sits on PCs. Typically manufactured by large, well known software vendors, these systems are most secure when configured properly, patched quickly, and kept up to date on the most current release. Note: these systems are as secure as their vendors have chosen to make them
• Web Apps - Here we find some of the utilities' efforts to establish better rapport with business and residential customers. Some are purely informational, but others use access controls to enable account management, bill payment and other self-help features. These are typically developed using a mix of COTS packages, custom code and free and open source software (FOSS), and security vulnerabilities can lurk in any of those three pieces, as well as from improper configuration. Note: these are as secure as the requirements stipulated they must be. If there were few/no requirements for security in the design docs, barring a major overhaul at some point, that's how much security you can expect to find in them.
• Web Services and Cloud - Code words connoting using remotely hosted application logic and data storage. We all use them today, and utilities, while sometimes slower to adopt new tech than others, are no exception. Examples include Geographic Information Systems (GIS), email, productivity apps, etc. These too, are as secure as their designers have chosen to make them, and in particular, users need to ask about how their data is protected, in transit and at rest

Parting Thoughts

In some ways, securing IT systems is the same job for utilities as it is for other sectors. It's been done before and is clearly not rocket science; yet doing it very well over time is a major undertaking for an organization, and requires solid commitment from the highest levels in an organization.as well as steady and adequate funding. It's not clear that as presently staffed and budget, most utilities can fully meet this challenge.

In other ways, of course, the ramifications of significant breaches are on quite a different plane altogether. As some of these systems will connect directly or indirectly to control systems that monitor and sometimes drive important physical power infrastructure, we should treat securing utility IT systems levels of gravity and rigor similar to FAA control tower applications or DOD command and control systems. The costs of failure in the energy sector are indeed often life threatening, not to mention economically and socially hazardous, and merit the community's absolute best efforts.

Chart courtesy of Forrester Research, 2009

SmartGridCity Competition: Infrastructure vs. Applications

You've got to deploy new meters and AMI infrastructure just to get in the Smart Grid game. But what if the costs go beyond your projections and you've agitated your rate payers well before delivering to them pledged new benefits and capabilities? At this point you're ready to buy or build the customer-side software applications that can begin to deliver on the many and several promises of the Smart Grid, but your hands are tied by tightening budgetary strings. As energy journalist Stephen Monroe puts it:

There is plenty of precedent for utility-scale subsidization of such "behind the meter" costs as programmable thermostats, compact fluorescent bulbs and high-efficiency furnaces. But with sunk costs for SmartGridCity already in the thousands of dollars per meter, regulators and ratepayers this year must decide how much more the system can bear before the project crosses from a forward-looking investment to one of never-present value.

Two risks come to mind in considering Boulder's current dilemma. The first is that the AMI/Smart Grid build out falters due to loss of regulator and ratepayer confidence. We need these advanced capabilities badly and failure to deploy the new grid, and its enabling applications, is unacceptable ... but possible, if we don't learn and adapt from the experiences of first movers.

The second risk is that in situations of substantial financial stress, security requirements are sometimes tossed out the window. Whether buying or building, deploying secure applications takes time and money, and the impulse to "deploy now and secure later" will be very strong.

To do so would be to put off failure for another day, in another form. Winning hearts and minds via powerful applications is a winning formula for the Smart Gird. Unless its merely prologue for widespread disappointment or anger from breaches involving loss of private data and/or system outages.

The Very Accessible, Very Familiar Google PowerMeter

The web has upset many industries' apple carts; is Google now set to outflank some of the utilities as they move to provide real-time info to their customers? See here from the EU Energy Policy blog.

Smart Grid Knowledge Boost: New EPRI Conference Announced

Come to New York City in late June for EPRI's EPRI Power Quality Applications (PQA) and Advanced Distribution Automation (ADA) 2009 Joint Conference and Exhibition. Lots of focus on new applications and some on smart grid security as well. You'll find a nice overview here and EPRI's own listing here.