CSOs and the Smart Grid

Setting the Stage
So you're an executive in charge of security at a medium, large or very large organization. You might be called Chief Security Officer (CSO) or Chief Information Security Officer (CISO) or maybe VP or Director of Security. You most likely report to the Corporate CIO, or you're in a business division and you and your boss plug into a General Manager. You decide, with blessing from above for the big stuff, the following:

• Where you'll get the biggest risk reduction (or compliance) bang for your limited budget buck
• Which technologies get purchased and implemented
• Which vendors will augment your in-house security team, and,
• Corporate security policies, and how to best promulgate them to other parts of the co. for whom security is at best an annoyance, and at worst, something to be openly resisted

Yours is a world of risk management as you oversee the wellness (e.g., integrity, reliability, performance, compliance) of your IT, networking and communications systems (and true CSOs own physical security as well). In addition to managing for threats coming from those directions, in recent years, new threat vectors from service oriented architectures (SOA), Web 2.0 and cloud computing have kept you busy.

Hey, Have you Heard of Smart Grid?
So how much time do you spend on future threats? If you have heard of the Smart Grid, and if you've been reading up on it, then you probably don't need to read further here. You're in the top 10% of your class and get a star on your forehead. If however, you're like some CSOs I've talked with who claim to have never heard the term, then this is your wake up call. There has been little written to guide CSOs through the early stages of preparing to protect their organizations in a world where the power systems they rely on look increasingly like the Internet (and in some cases are the Internet!).

How is it different from today's electrical grid? For starters, it's a 2 x 2-way system. Thanks to advanced metering infrastructure (AMI) and net metering, electricity and usage information will flow from generators to consumers and back again. The total amount of information, which in the beginning will be substantial, will quickly become enormous. Data protection will be crucial, and demand management strategies which could save your organization significant money, could also get you in trouble fast. Water and other services will also be impacted for better and worse. In short, for each benefit a Smarter Grid will bring an organization, there is a commensurate risk to mitigate. And it's your job to know (and plan for) this.

Only CSOs at utilities see this world first hand, and even in the energy and utilities vertical, many of those CSOs work in a balkanized world where their policies touch only IT, and the "rubber meets the road" part of their company, field operations, doesn't want to anything to do with them.

So most CSOs are left to infer what they need to know from a mountain of Smart Grid articles and a multiplicity of Smart Grid conferences. My guess is once they've poked a toe into these confusing waters one time, they soon find their time better spent working on present challenges. The appropriate information has not yet been boiled down for this most important enterprise leadership function ... one that could and would do the right things, proactively, if it had the right knowledge to work with.

CSO Info Resources Not Too Helpful Yet
Where do CSOs turn for expert guidance and to learn from what their successful peers are doing? Why, the journals and other news sources that serve them. Yet from the looks of these two articles from CSO Online and the CSO Roundtable, all they're getting is high level introductory material that in no way considers how Smart Grid trends intersect with CSOs' particular responsibilities. I would advise these orgs to get on the ball: it's their job to see over the horizon and around corners to give their readers the info they need to protect their companies ... and their jobs.

No Answers Yet, But Here are a Few Starter Questions
NIST and other standards bodies are working around the clock to bring appropriate and helpful security standards to this new domain and you don't have to know them yet (however, for a sneak peak, here's the most recent draft edition of Smart Grid Cyber Security Strategy and Requirements from NIST). So much is still in flux that doing too much at present might be as bad as doing too little. But that doesn't mean you shouldn't start getting your head around this challenge and thinking through some of the scenarios. Here's a handful:

1. Supply Chain - Similar to Y2K preparation in some respects, even if you get your house in order for the arrival of the Smart Grid, if the companies yours depends on are not prepared it may affect you. It's time to talk about this with them.
2. Vehicle Fleet - More choices are coming, including hybrid electric, full electric, natural gas, etc. Are you thinking about the challenges and opportunities that present themselves in beginning to move away from gasoline and diesel? What are the security implications of your enterprise depending on these new transportation technologies?
3. Local utilities - All utilities are under guidance to prepare for Smart Grid standards and technologies. What are your providers doing in your different locations and how soon will their actions begin to affect you? What do you need to do to not get blind sided?
4. Smart Grid pilots - With stimulus help from the Fed Gov, pilots are springing up everywhere. Related to number 3 above, are there any pilots going on you could participate in? While this might take resources away from more proximate concerns, the education might more than pay for the time invested.
5. Centralized policy and control - If yours is a geographically distributed operation, to what extent will you attempt to define and enforce Smart Grid-related security policy in a uniform way, versus allowing disparate facilities and offices to determine their own best approaches?

That's all for now, but on each of these and many more there's a ton of thinking and planning to be done. While in most cases it's too early to implement, it's certainly not too early to imagine.

And Then There was None

News from the Smart Grid Investment Grant program

Early Birds win again! Looks like the interest and enthusiasm for Smart Grid Programs has rapidly outstripped even the Government's own $3.4B largess. In an amendment dated September 21, the DOE announced that:

The Department of Energy has received a significant number of high quality applications and our review continues. The dollar value of applications far exceeds the funding available under this Funding Opportunity Announcement. As a result, Phase III is canceled.

and

Given the facts cited above, the Department may decide to cancel Phase II following final selection decisions made on applications currently under review.

So, what was intended to be a three phase investment program in new approaches to energy and grid management has become at best a two-phase program, and likely a single shot of stimulus into the Grid. Taking the amendment on its face, that the dollar value of applications already received far exceeds the funding available, we can conclude:

In the planned Phase I application period, running from the initial solicitation date of June 25th, 2009, to August 6th, 2009, there were requests for grants FAR EXCEEDING $3.4B. This means that, on average, the DOE received grant requests FAR EXCEEDING $113M every business day of the Phase I application period.

Each of these applications was expected to include many things, not least among them a well-articulated security plan. You will remember, from the cyber security requirements description:

Submitted Project Plans are also required to include a section on the technical approach to cyber security. Cyber security should be addressed in every phase of the engineering lifecycle of the project, including design and procurement, installation and commissioning, and the ability to provide ongoing maintenance and support. Cyber security solutions should be comprehensive and capable of being extended or upgraded in response to changes to the threat or technological environment.

Yikes. And more specifically must include:

• A summary of the cyber security risks and how they will be mitigated at each stage of the lifecycle (focusing on vulnerabilities and impact).
• A summary of the cyber security criteria utilized for vendor and device selection.
• A summary of the relevant cyber security standards and/or best practices that will be followed.
• A summary of how the project will support emerging smart grid cyber security standards.

In 20ish years of working in security, I have seldom found an organization that could create this level of cyber security detail within six months for an existing system, much less create it in 30 business days for a brand new project.

The infusion of SGIG capital has definitely gotten things moving, but we should all hang on. This looks to be a bumpy ride.

( Photo thanks to http://www.flickr.com/photos/pleasantpointinn/ / CC BY 2.0 )

Conference Alert: SCADA and Control Systems Security Summit

Just the facts, M'am:

• What: a gathering of like minded individuals intent on learning more about threats posed by systems not well known or understood by IT and Internet security crowd. Similar to mainframes in that they were originally conceived to run in an utterly disconnected world, early SCADA implementations (many still performing critical roles today) were designed with little thought to access control and authentication. Yet SCADA and other types of electronic control systems are as much a part of the emerging Smart Grid as will be the latest hardware and software offerings from CISCO, GE and SilverSpring. Because they have remained relatively obscure outside the operational utilities domain, developing strategies to secure them is now the order of the day as development of the Smart Grid leaps ahead.
• When: 7-9 December 2009
• Who: DHS, DOE, NERC and NIST will be there, joined by others from government and industry
• Where: Washington DC (venue to be named)
• How: For more info and to register, click here

Preparatory Resources

• The Open SCADA Security Project
• The Center for SCADA Security - Sandia National Labs
• Wikipedia's SCADA entry
• Blackhat Powerpoint: SCADA Security and Terrorism (PDF)

Photo courtesy of: Ian David Blum on Flickr

Surge Protection: The New Smart Grid Data Challenge

As has often been written, the advancements of the Smart Grid are founded in information. Data is used to inform consumption, to make rates more dynamic, and to enable the next-generation power prosumer. In reading a recent piece on potentially mandated Smart Metering in the UK, the Telegraph raises the issue of data handling relative to today's data management. In short strokes, 44 million homes were typically measured twice a year, making for 88 million entries for data. In the new system, every home is measured twice a day, meaning that those 88 million entries have now become over 32 billion. Now this sounds like a lot, and let's quickly look at the new challenges that arise for organizations seeing this kind of increase:

Data Center Expansion

The types and volume of data associated with Smart Grid use will mean a new need to bring Internet-style data centers into the complex mesh of Utility control systems

Data Organization and Retention

With Time of Use pricing and user charge recovery for power generated, a sizable subset of this data will no longer be simply transient and used in the aggregate. Individual elements will need to be captured and tagged for later retrieval over whatever period is chosen by regulators as appropriate for looking back.

Data Privacy

While there may be dubious benefit to stealing the private data from individual citizen's Smart Meters, it is naive to think that privacy concerns will not find their way into regulation, meaning that data will need as well, to be partitioned when needed longer term, destroyed when transient, and never left in an unknown state.

I led with the UK piece, because it does a relatively non-threatening analysis of data gathering trends from a Smarter Grid.

The US Smart Grid, however, has a series of challenges that expand on this by many times. Back in May, Beth Pariseau did a piece on Smart Grid storage for SearchStorageChannel.com where she interviewed a variety of players, including Austin Energy's CIO, Andres Carvallo. The data usage trends described are nothing short of mind-boggling.

In the Austin Energy data, for phase one of the roll-out which included 500,000 meters, the increase in yearly data storage went from 20TB to 200TB, with disaster recovery redundancy. This is for 15 minute sampling, and first stage (appears to be largely home-oriented) integration. Ignoring smaller sampling frequencies (resulting in much higher data storage) necessary for some Smart Grid functionality, this presents a model of about 400 MB per meter per year. ( 200,000,000,000,000/500,000 ).

While this sounds mind-numbing, there is substantiation (and a reasonably close ratio) in the same piece, this from Pacific Gas and Electric, who added 1.2PB of memory (and growing) to support 700,000 meters, or over 170MB per meter per year. (This was sampling only twice per day).

What conclusions can we draw from all of this?

Massive Data is about to swamp existing infrastructure, requiring some hard thinking about how to architect, secure, segment, and deploy, the data centers that will accommodate it.

There is striking variability in the amount of data that organizations are expecting, seeing, and preparing for. Work is needed on what information should be gathered, what needs to be stored long-term, what needs to be tagged with user information, and what needs to be treated as private.

This is a new area for providers. The storage, record keeping, and maintenance of all of this data, particularly that which needs to be help for longer regulated periods, is unlikely to be a current function of the provider budget and functional organization. The steps to rationalize this area financially is critically important. Any plan to advance smart metering should include these costs in justification or grant request.

Every new idea for the Smart Grid, particularly those in the soft grid investment space must detail the additional burden they are likely to place on providers from a data acquisition, data management perspective.

Like so much of our economy, these advancements are changing the Grid from a Power economy to a Data and Power economy. To survive and thrive these new requirements must be considered. In the medium and long term, those organizations which consider, and then capitalize on, all of this data acquisition, will find themselves in a much better position to add services, ensure satisfaction levels, and find new ways to make the Smart Grid even Smarter.

[ And by the Way: In their August 2009 report on "Assessment of Demand Response and Advanced Metering", FERC presented a partial scenario (80M meters) and a full deployment scenario (140M meters) by 2019. Assuming that we feel comfortable in the midrange of the data descriptions used earlier, this would imply the need for the creation of infrastructures necessary to organize and manage roughly 100PB of information within the next ten years. Good luck to us all. ]

(SmartGrid diagram courtesy of US D.O.E).

Smart Money on the Smart Grid?

The Venture Capital business is a brutal one. The process can appear to be like Darwinian Natural Selection on speed, as venture dollars drive multiple entrants into an emerging space in hopes that as the weak are weeded out, their own investments will survive and thrive. At worst, there is cold comfort in the fact that the compressed timeframes will help them to identify their own latent failures more quickly so that they can cut their losses.

I was discussing this mechanism of investment acceleration yesterday with a colleague who does some later stage (profitable stable companies) cleantech investing, and he was remarking on the Klondike Gold Rush-like movement by some Venture firms into cleantech, and into Smart Grid startups particularly. The Smart Grid boom, in his view, is the first and closest child of the Internet boom. Biotech (another area of large investment) has been a very different model, with its long lead-times and eight or nine digit price tags. I had to agree. So much of the Smart Grid is looking like Soft Grid, and successful startups are bringing in management software, efficiency software, upgraded infrastructure and communications. It really does feel like the early days of the Internet, where technology startups faced relatively low costs to enter into a new market, where the existing infrastructure needed evolutionary enhancements pretty regularly, and where the established players were unlikely to step outside of the box to make those changes happen. In the Internet era it was telecommunications companies who provided both the enabling backbone and the lack of groundbreaking higher-level innovation that created the opportunity for entrepreneurs. Now it is the utilities' turn.

In sheer numbers, the investment is amazing. The Cleantech Group reported yesterday that the cleantech sector accounted for 27% of venture investing in the second quarter, which shows how enormous this wave is, totaling over $1.5B for that period. They also reported that many of the largest investments went to firms which were also leveraging Government funding dollars. So, what does this foretell?

It foretells a glut of new technologies, advancements, approaches, and failures. Larger organizations will be able to invest their own time and money on comprehending and capitalizing on the meaty part of the wave, while these new entrants stay at the crest, and either find the ride or the rocks as the industry approaches the first winnowing stages. Ordinarily, this kind of furious growth yields rapid progress, and markets and nations benefit from the rapid determination of good and stable solutions. Whether this will work for the Smart Grid is yet to be seen. The nature of power, and the economics of traditional utility finances can make this tumult and its turbulence a disaster.

Venture investors expect to see failures, their models assume them. The Government investors expect to see, well, whatever. The government is funding policy through technology.

Power providers and customers, however, can not be tolerant of too much instability, and so we hope that adoption of these technologies will remain proactive but prudent, regardless of the "energy" that all this investment may put into the grid.


Image courtesy of flickr :

http://www.flickr.com/photos/flashpackinglife/ / CC BY 2.0

Smart Grid Startups and Security: Round 2 from GridWeek

This post picks up up where we left off last week during GridWeek 2009, examining patterns that emerged from our talks with Smart Grid startup booth reps. Jack and I noticed that few of the startups are staffed with a dedicated security professional, and had tasked an existing player (CTO, Application Engineer, etc.) with the responsibility. Other exhibiting companies (Cap Gemini, Cisco, GE, ABB, Siemens, etc.) had booths too, but it seemed crazy to ask them if they employed dedicated security pro's, because of course they do, both for their internal operations as well as for their client-facing products and services.

Back to the startups. As you know, we like to pose questions ... so here are a few:

• In a domain where security rigor is universally regarded as essential, how much security thinking is going on within these start-ups, and how long will the present level be enough?
• Put another way, when you're a small but growing company in the Smart Grid software or hardware space, how long can you hold out before adding a full time security professional to your team?
• Do you hire a security staffer once your development team reaches a certain size, say a headcount of ten, or should you put the security pro in place up front to help define the development process before you start writing real code?
• Given the amount of innovation required in most of these companies, how reasonable is it to expect that the CTO can juggle all the technology balls he/she is responsible for, and do a good job on security tasks (which will often seem like a distraction) at the same time?

I liken this to the situation that faced large and medium companies approximately ten years ago, when it was becoming clear that as they embraced the Internet for new capabilities, they were inadvertently bringing a whole host of new risks and vulnerabilities on board. This is from CSO Magazine in 2001 on why to hire Corporate Security Officer and what he or she can do for you:

... a core responsibility of the CSO will be vulnerability assessment and risk management. Therefore the CSO should report to the COO or CEO. After all, the CSO will evaluate the technology environment and audit the security measures implemented by the CIO. It is in the company's and the CIO's best interest to have the CSO perceived as an impartial assessor of the technology environment instead of a possible rubber stamp .... Think of the CSO as the head of quality assurance for security.

In startup-land, there is no real need for C-level titles beyond CEO. But ignoring the titles, the functional benefits of a dedicated security staffer are clear, no matter what they're called. In other markets we have seen them labeled: Security Architects, Information Security Officers, Security Managers, Security Officers, Information Security Managers, etc. Depending on the offering and the market strategy, there's a mix of roles that these folks may fill, including ensuring the security of the company (its systems, processes and people) and the security characteristics of its products; hardware, software or both.

Hyperbole aside, we all know that the Smart Grid is an area of growing and inevitable security risk. If I'm a utility, and as such am a prospective new customer for a startup, and I'm held accountable to the highest security standards by those who regulate me, I'm going to be damned sure that I put prospective vendors through the ringer before bringing their technology in house. And if I'm a startup, while having a qualified security person on my staff is no silver bullet, our guess is they'll be more than worth their salary as the regulators press their security cases and the utilities/customers get more and more savvy about risk.

Smart Grid: Greener but no Greenfield

It is good to see the attention that the new NIST draft directives for the Smart Grid are getting in the press. Ordinarily, this type of draft release is not interesting enough to the general public to merit any real press, and ends up being a conversational target to the few who arrive interested in the space. Any mainstream attention comes much later in the cycle, as affected parties either applaud or complain.

One impression that I would like to correct is that the Smart Grid itself, and therefore, the challenges of Smart Grid security, is something being developed from scratch.

In Federal Computer Week, Bill Jackson calls out the following:

Deployment of a Smart Grid offers a greenfield opportunity because the existing grid, parts of which are 50 years old or older, was not designed to support alternative energy sources such as wind and solar power, and the two-way flow of energy and data. But this wholesale upgrade also makes it imperative that security be built in now, because the grid lifecycle is measured in decades rather than years, as it is for much of the rest of our information infrastructure. Equipment being designed for deployment now might not be replaced for decades.

There are so many capabilities within the Smart Grid that are new, and there is so much investment going into it, that it is completely understandable to conceive of the Smart Grid as the "new" grid, as opposed to the evolution of the "old" grid. The Smart Grid as a replacement is a misperception that we have seen often in our work on evangelizing smart grid security. The Smart Grid is not a greenfield, not a replacement infrastructure, and most definitely not a new grid. We always have to remember that the Smart Grid is a new way of leveraging, stabilizing, advancing, and enhancing, the OLD Grid.

The billions that have been made available through the Smart Grid Investment Grant Program, the additional billions that are pouring into development of renewables, transmission and distribution advancements, PEV's, and storage, are only a small fraction of the total picture when the nation's power infrastructure is viewed in its eventual entirety. As a result, when we are considering the security of the Smart Grid, we must always consider (as the NIST work does ) the existing grid. Whether we work to create more secure means to connect to it, or to actual revisit the older technologies and improve their protections, those challenges will likely be the most pressing, and the most complicated, that we need to solve.