Not the Lead Dog? Get Used to the View

"Audentis Fortuna iuvat"

(Fortune favors the bold)

-Virgil

Last week Andy led with his thoughts on the risks to the organizations who are acting as the sharp end of the stick as the Smart Grid begins to expand and mature. There is a long tradition of danger for these early movers, whether the front row of sarissa carrying soldiers in Alexander's army, or the unhappy few searching for new titles on their Discovision LaserDisc players.

That said, some upfront thinking and informed planning with built-in checkpoints can make "early mover" a winning proposition, not a eulogy. Not all pioneers take the arrows. Some get the land.Andy notes that the SGIG winners may find themselves regretting their good fortune, as the influx of Government funding for shovel-ready projects is driving the installation of tens of thousands of meters, none of which can possibly have met the federal standards for such meters, since those standards don't really exist yet. Last Wednesday (January 20, 2010) NIST released the first non-draft version of their Interoperability Guide, and which tried to make more manageable the release of a wide variety of standards with which it was charged:

Some are needed more urgently than others. To prioritize its work, NIST chose to focus initially on standards needed to address the priorities identified in the Federal Energy Regulatory Commission (FERC) Policy

Statement, plus additional areas identified by NIST. The eight priority areas are:

• Demand Response and Consumer Energy Efficiency
• Wide-Area Situational Awareness
• Energy Storage
• Electric Transportation
• Advanced Metering Infrastructure
• Distribution Grid Management
• Cyber Security
• Network Communications

These guidelines are providing adopters with plenty of direction for the likely coming regulations, and every syllable (there are many and we will do some more overviewing for this audience soon), is articulating the siren song of reduced likelihood of stranded hardware, incompatible systems, and inappropriate security.

So much of this early growth within the Smart Grid community was already foreseen and planned, I think that the idea of waiting interminably for more data is an overly conservative strategy. To my viewpoint, much of the SGIG funding is the government writing checks to get out in front of a parade that is already moving. Looking for "Shovel-ready" projects is a way for the government to locate initiatives that had already been thought through, that were likely already justifiable from simple cost-savings on labor and system downtime, and which were unlikely to be anything particularly ground-breaking or risky. No bureaucrat is ever looking for the headline, "SGIG Tax Dollars Burnt to Heat Up Smart Grid Market".

For those who have intentionally hung back, I would encourage a little more briskness in their steps. There are risks, as well, to being overly cautious:

• There truly is a land grab ongoing in the leadership space for Smart Grid adoption, and/or
• There are appreciable cost-savings one can see today with AMR/AMI implementations, and/or
• The standards to come will likely be generated from the experiences of those actually moving the Smart Grid forward, therefore naturally favoring them

Waiting for the decisions to be made and for the risk to be gone may be comfortable, but it is unlikely to spell success for organizations and leaders who take the easy way out.

img thanks to letmakerobots.com

Is the Smart Grid Inducing Labor?

"The fight is never about grapes or lettuce. It is always about people."

- Cesar Chavez

It seems there's a wire crossing happening amid the hard-working folks who are helping create and manage the Smart Grid. In spite of positive initial reactions to federal investment in the creation of the Smart Grid, the law of unintended consequences is bringing some consternation among the ranks of organized labor as Smart Grid programs move from philosophy to reality.

While the introduction of the Smart Grid Investment Grant (SGIG) program was applauded by many in the labor community as the beginning of a new market for skilled technicians (see: this AFL-CIO blog post, or this IBEW promotional video), some actual deployments are not being greeted as favorably.

On January 19th, 2010, the Kennebec Journal reported that IBEW Local 1837 was "speaking out against" a new Smart Meter installation project by Central Maine Power (CMP). It was funded to the tune of $96M through the SGIG with a total cost of roughly $190M. Seems that the project would likely eliminate some 141 positions over time, and that did not sit well with the union.

The tensions at CMP, however, are not unique. In October, a plan by the board of Memphis Light, Gas and Water Division (MLGW) received similar criticism from the IBEW, which noted that roughly 400 meter reading jobs would be lost in that plan.

So how can there be such a disconnect?
The Smart Grid is comprised of much more than just smart metering. It involves redundancy, resiliency, and quality of power, and ease of integrating renewables and storage, and more. Today's unfortunate reality, however, it that investment has been overwhelming skewed to Smart Metering. Smart meters, and the improvements in automating, and "remotifying" the reading, turn-on, and cut-off of power, are seen as early wins. They do not appear to jeopardize the delivery of power, and can very quickly demonstrate cost efficiency by decreasing truck rolls. This is both a reaction to the government's emphasis on "shovel-ready" projects to fund, and to the ease with which a utility can justify projects to regulators as cost-savers, paying off capital costs in short order through reductions in labor costs. As a result, the union teams, originally anxious to generate skilled labor to drive the construction of the next generation of transmission and distribution, is left, instead, with a short-term need for installers who will be wiring up the elimination of hundreds of jobs for their meter reading brethren.

What to do?
One of the factors underlying the development of the Smart Grid is very much people-related. We have written of it here briefly in the past, but it deserves another shot. An aging workforce manages the existing Grid, and it is retiring at a rapid pace, even in this tough economy. In an article from April of 2008 in "Power" magazine, the percentage of retiring workers is pretty daunting:

When we then look to the remediative measures that folks are taking, we see impacts related to new technologies:

Rightly or wrongly, about 90 percent of utilities are looking to use new technologies to augment the diminishing staffing, while they continue to employ traditional staff supplementation techniques.

New Smart Grid technologies are creating a raft of new opportunities for a new generation of skilled labor. Whether it is the implementation and management of transmission and distribution technologies within utility infrastructure, as already looked to by the unions, or the creation of new skills and laborers to assist in residential, commercial, and public construction of power systems that will leverage these new capabilities, the opportunities are many.

Within the greater IBEW, there are already efforts underway to help to address this need, including the "National Utility Training Trust", reported on here in "The Electrical Worker", and it looks like they are moving forward to capitalize on the growth in the Smart Grid.

As we have written about previously in the areas of IT adoption and data usage, well-trained personnel are vital to the security of the infrastructure as it grows, and these new resources can integrate security considerations into their own interactions and behaviors with the Grid and its computational components. New workers have the opportunity to advance their careers, their marketability, and their value, with a focus on these additional skills.

This growth though, is not going to come for free. The Smart Grid and the market for power will benefit from this new wave of skilled professionals, but some of that market and advancement will need to be cost-justified through transitioning and reeducating existing personnel. I hope that as the Smart Grid grows, the labor pool increases to fuel it, and organizations such as the IBEW Local Chapters become champions of that change and growth ... not adversaries, who could introduce impediments that might hobble the same workers they seek to protect.

First Mover Disadvantage in Smart Gridland

It's been proven that it works in chess and as everyone knows, like a charm in tic tac toe. In the business world, according to Wikipedia, first mover advantage: "... is the advantage gained by the initial occupant of a market segment. This advantage may stem from the fact that the first entrant can gain control of resources that followers may not be able to match."

Well, as you know, in the heavily regulated utility sector, it's not exactly a cut-throat competition. In fact, it's not a competition at all. But that doesn't mean it's not worth watching who's out of the gate first with AMI and Smart Meter deployments, who's received Smart Grid Investment Grant (SGIG) funds and is now obligated to deploy something of significant size, and who's holding back, keeping their powder dry.

The earliest of early movers (you know who you are in that big state just north of the Rio Grande) who began their own experimenting long before the SGIGs were a twinkle in the current administrations' eyes are probably best positioned to make the right Smart Grid technology deployment decisions at the times and places of their choosing. But the new first movers, the 100 or so SGIG grantees, who are making deployments now of thousands or millions of residential Smart Meters, are, IMHO, in a less than optimal position.

They are choosing hardware, software and communications tech well before most of the relevant standards (including security) have settled. Are moving before their customers, in some cases, are fully in tune with what's going on and how it will impact their bills or their service. They've often asked for rate relief to fully fund these deployments and may well be asking for more in an unfortunately short amount of time when it turns out they've placed bets on the wrong vendor and standards horses.

From speaking with analysts, utilities, and some of their providers, my sense is: laggards may have a real advantage here. How's that you say? Here's how:

• As long as they are active and attentive laggards, waiting, watching and learning, they may come to thank their lucky stars that their SGIG proposals were not selected
• They can tinker with residential pilots that number in the tens or hundreds of meters, vs. thousands and millions
• They'll have a longer lead time to educate and prepare their customers for coming changes
• And laggard utilities will be able to select and deploy, with far more confidence than they can in early 2010, technologies based on a more mature, settled standards landscape

As the Latin proverb says, "Fortune favors the bold". Or maybe Bill Shakespeare has the words most appropriate here: "Discretion is the better part of valor." For the moment, hold your course laggards, but watch, learn, and get ready for your turn.

Photo Credit: CarbonNYC / David Goehring @ Flickr

How we got here: Insecurity, the Grid, and Getting Smart

In a recent series of conversations with people versed in the space of evolving the existing Grid into the Smart Grid, I was initially frustrated by the apparent disconnect that exists between the accepted standard practices among the IT and Internet security communities and the current state of the art, or education, or experience, among many of the implementors and advocates of Grid advancement.

It really made little sense to me, in as much as we have been working on these challenges and their resolution for more than 20 years. How is it possible that the most critical of all of our infrastructures, the US electrical power system, was not leading the charge for more and better IT security? It only made sense that the builders of the world's largest, most complex, and most important system, would be the titans to tackle the most thorny challenge: securing it.

The past several months, though, have been eye-openers for me on the historical reasons behind this disconnect, this lag, and I think it is useful to take a look at those causes and conditions. By looking at the reasons for the current insufficient state of security, we can first stop blaming the industry for its vulnerability, and can begin to conceive of methods and motivators for changing those behaviors.

At this point, I ask any superior-feeling IT security personnel to check their egos at the door. There is little to gain from rock throwing and facetiousness, and a thoughtful perspective can help to inform the right steps to hardening these systems. Secondly, I would ask the valued-but-vanishing IT and Control folks from the Utility community to similarly stand-down on their defensive rhetoric. I believe there has been a lack of common history and heritage between them, and it is through sharing information that we can help to bridge these two communities.

So. No bullies allowed.

"Why Are Utilities so Behind Banks and Retailers and Even the Government (gasp) in IT Security?"

This is a question we have seen published openly, and heard as an undertone in examinations of cyber incidents on the Grid. While it feels like the truth, this type of characterization is not really fair. Utilities are very different from most businesses because their smooth running is not a differentiator, it is a requirement. You can see this in the regulations which drive utility policies, most of which state clearly that "reliability" is the goal, and "security" is usually, conspicuously, absent. Most commercial concerns, and even the government, are investing constantly in new information technology to connect and capitalize on their relationships with clients and communities, with goals of scale, or sharing, or speed. Leading or "bleeding" edge adopters are making an educated bet that new technologies will bring them new goodness in terms of revenue, image, cost-savings, or growth, and security is a necessary drag-along to implement them. We need to remember that many industries, like banks, are mainly software and software operations firms now, since the money, or the transaction, or the data, is largely stored in 1's and 0's, not in vaults. Retailers or the Registry of Motor Vehicles are trying to find ways to increase the ease and speed of your transaction while reducing the cost of executing it. Again, security comes as a cost for these groundbreaking changes in the customer/provider relationship.

Utilities are very different. They are still responsible for keeping the lights on, first, and foremost. It sounds strange, but in the pre-Smart Grid period, there was strikingly little focus on differentiated services, and even marketing, from the perspective of most utilities. Many Americans can't name their electrical provider, and certainly have nothing like a close relationship with them or their plans and data. This means that the investment and the payout on new technologies are not easily understood, measured, or desired, in the way that they are in other industries. This becomes more obvious as we look more closely at some of these differences:

Mother May I?

First off, because it is such a basic and foundational commodity in our lives, and one that is so expensive to create in bulk, electricity is a highly regulated institution. If not, years ago the unscrupulous would have capitalized on and bankrupted the base. In the period before the creation of the Rural Electrification Administration by Franklin Roosevelt in 1935, rural farmsteads were extremely underserved because of the prohibitive cost and lack of profitability. Individual farmers would be forced to pay for their own connections, to the tune of $20,000 in today's dollars, after which the utility would own the constructed lines. The REA changed this, but it also introduced a group of new federal and local regulating bodies. Even today, if a utility wants to institute a new program or policy, it needs to justify that investment to regulators, who represent the rate payers who will ultimately have to bear the upfront and operational costs of any improvements. While this clearly complicates any major investment, it makes more granular and speculative investments (like securing grids against attackers that haven't been widely seen yet), become down right impossible, as ratepayers would be asked to pay more money for the same power that they have been receiving right along, and will likely see only minimal positive impact over a long period of time.

Stability versus Agility

At this point, it is useful to think about another rationale for the lack of progress on some of these more advanced IT fronts, prior to the Smart Grid's introduction. The question is a simple one. "Why?" Why should they have been integrating new technologies over the previous decades? Frankly, the power has stayed on pretty well in the main. Each year has brought its occasional black-outs, but nothing so significant that the world could find substantial fault in the currently underlying architectures and tools. Given that, once again, how would one justify any massive funding to achieve growth and cost-savings? Lacking this, there is no substantial pull in the market to incorporate ground-breaking IT, and there is certainly nothing like the competitive technical blood-letting that has defined the competition between retailers, between banks, between media firms, and among government organizations. No pull, no motion. Like a train.

Experts and Expertise

There is a lack of knowledge about utility implementations that is rife outside of the E&U market, and a comparable lack of comprehensive knowledge of the coming overlaps with advance IT within the E&U market. The complex and largely proprietary systems that have evolved to service the growing market for power has bred its own priests and priestesses who can conjure the connections between sensors and centralization, and between remote units and controllers. This is a very different skill than weaving a consistent pattern of routers, hubs, and access controls. These control networks are the "backbones" that create the possibility of reliable power, and while security is most definitely a requirement, it has meant something very different until recently. Where Internet and IT teams are looking at understanding likely breaches, utility teams have sought out likely failures. Where utilities are focused on uptime and reliability, Internet and IT are concerned with fraud, theft, and corruption. So it is understandable that there are not many who are expert in one area who have also had the time, inclination, and opportunity, to be similarly skilled in the other. No money for the new technology, no one asking for the new technology, means that there is unlikely to be any organic development of resources with the overlapping skill set

Bringing it all together

So what does all this mean? One thing it means to me, and likely to other readers sensitized to the space, is that we can stop looking for some native incapacity or reticence on the part of utilities professionals to learn the techniques and technologies of security in their new and/or looming IT/Internet-based infrastructures. Another thing is that the influx of funding, from governmental and private buckets is creating the opportunity to attract both new skilled resources from elsewhere in the market, and to provide support for the development of those personnel from the inside out.

Understanding that the need for pervasive internetworking is being driven by advancements in energy generation and energy technology, not by a more base desire to "catch up" with mainstream IT, will help to create a much more attractive playing field and mission. Previously-resisting utility teams can acknowledge that there is an important role for these newer and sometimes less stable technologies. And incoming IT professionals can take the lessons they have learned by interconnecting other industries to create a smoother and more successful path forward to the Smart Grid.

Massoud's Smart Grid 2010 Update

My Smart Grid Security blogging career began shortly after meeting Dr. Massoud Amin of the University of Minnesota and the Electric Power Research Institute (EPRI), and attending his presentation at an MIT/Sandia Labs Smart Grid workshop in early 2009.

For those who haven't yet met him, Massoud is a force of nature. In case you're not sure which type of force he is, I'll tell you: he is of the brilliant, massively energetic, patriotic and fortunately, quite benevolent kind. This video (6 mins) just released by U of Minn provides a nice Smart Grid introduction for generalists while making two security points along the way:

1. as much as possible, security has got to be built in from the start, and
2. broad and creative use of now super-cheap sensor systems can play a big role in keeping bad guys at bay

Point number one is of course correct in principle from where Jack and I stand, though will be sometimes tough to implement given how much of the Smart Grid is really the regular grid with retrofits and overlays of new tech on top. Point number two is excellent, as we hold that creative use of orthogonal data and the powerful analytics now available can help us identify malicious behavior early and rapidly adapt systems to keep 'em safe, if not always totally secure.

Here's Massoud's home page with some great material to mine. Recommend you check it out.

Photo Credit: Wikimedia Commons