Surge Protection: The New Smart Grid Data Challenge

As has often been written, the advancements of the Smart Grid are founded in information. Data is used to inform consumption, to make rates more dynamic, and to enable the next-generation power prosumer. In reading a recent piece on potentially mandated Smart Metering in the UK, the Telegraph raises the issue of data handling relative to today's data management. In short strokes, 44 million homes were typically measured twice a year, making for 88 million entries for data. In the new system, every home is measured twice a day, meaning that those 88 million entries have now become over 32 billion. Now this sounds like a lot, and let's quickly look at the new challenges that arise for organizations seeing this kind of increase:

Data Center Expansion

The types and volume of data associated with Smart Grid use will mean a new need to bring Internet-style data centers into the complex mesh of Utility control systems

Data Organization and Retention

With Time of Use pricing and user charge recovery for power generated, a sizable subset of this data will no longer be simply transient and used in the aggregate. Individual elements will need to be captured and tagged for later retrieval over whatever period is chosen by regulators as appropriate for looking back.

Data Privacy

While there may be dubious benefit to stealing the private data from individual citizen's Smart Meters, it is naive to think that privacy concerns will not find their way into regulation, meaning that data will need as well, to be partitioned when needed longer term, destroyed when transient, and never left in an unknown state.

I led with the UK piece, because it does a relatively non-threatening analysis of data gathering trends from a Smarter Grid.

The US Smart Grid, however, has a series of challenges that expand on this by many times. Back in May, Beth Pariseau did a piece on Smart Grid storage for SearchStorageChannel.com where she interviewed a variety of players, including Austin Energy's CIO, Andres Carvallo. The data usage trends described are nothing short of mind-boggling.

In the Austin Energy data, for phase one of the roll-out which included 500,000 meters, the increase in yearly data storage went from 20TB to 200TB, with disaster recovery redundancy. This is for 15 minute sampling, and first stage (appears to be largely home-oriented) integration. Ignoring smaller sampling frequencies (resulting in much higher data storage) necessary for some Smart Grid functionality, this presents a model of about 400 MB per meter per year. ( 200,000,000,000,000/500,000 ).

While this sounds mind-numbing, there is substantiation (and a reasonably close ratio) in the same piece, this from Pacific Gas and Electric, who added 1.2PB of memory (and growing) to support 700,000 meters, or over 170MB per meter per year. (This was sampling only twice per day).

What conclusions can we draw from all of this?

Massive Data is about to swamp existing infrastructure, requiring some hard thinking about how to architect, secure, segment, and deploy, the data centers that will accommodate it.

There is striking variability in the amount of data that organizations are expecting, seeing, and preparing for. Work is needed on what information should be gathered, what needs to be stored long-term, what needs to be tagged with user information, and what needs to be treated as private.

This is a new area for providers. The storage, record keeping, and maintenance of all of this data, particularly that which needs to be help for longer regulated periods, is unlikely to be a current function of the provider budget and functional organization. The steps to rationalize this area financially is critically important. Any plan to advance smart metering should include these costs in justification or grant request.

Every new idea for the Smart Grid, particularly those in the soft grid investment space must detail the additional burden they are likely to place on providers from a data acquisition, data management perspective.

Like so much of our economy, these advancements are changing the Grid from a Power economy to a Data and Power economy. To survive and thrive these new requirements must be considered. In the medium and long term, those organizations which consider, and then capitalize on, all of this data acquisition, will find themselves in a much better position to add services, ensure satisfaction levels, and find new ways to make the Smart Grid even Smarter.

[ And by the Way: In their August 2009 report on "Assessment of Demand Response and Advanced Metering", FERC presented a partial scenario (80M meters) and a full deployment scenario (140M meters) by 2019. Assuming that we feel comfortable in the midrange of the data descriptions used earlier, this would imply the need for the creation of infrastructures necessary to organize and manage roughly 100PB of information within the next ten years. Good luck to us all. ]

(SmartGrid diagram courtesy of US D.O.E).