IE11 Not Supported

For optimal browsing, we recommend Chrome, Firefox or Safari browsers.

New Relationship Will Help State Agency Move into Big Data

The agency, whose mission centers on energy policy and planning, needed a cost-effective way to do more with its data storage.

dataphoto.jpg
One of the key state agencies charged with charting California’s energy future is moving into big data and gaining efficiency through a new private-sector collaboration.

The California Energy Commission (CEC), the state’s primary energy policy and planning agency, is working with San Mateo-based in-cloud data warehouse provider Snowflake Computing to become more data-driven. Jason Harville, CEC assistant executive director for energy data and analytics, told Techwire that Gov. Gavin Newsom's administration has “made it clear down the line that we want to be data-driven as a priority,” and noted that the state also maintains a cloud-first policy. CEC officials have plans to receive more data starting next year “than we have in the entire existence of the organization.” The commission, which procures on an annual basis, will use its relationship with Snowflake to handle exponentially more data.

“We have a broad range of responsibilities at the commission, and a lot of them are analytical in nature, forecasting and modeling to form the basis of the state’s energy planning process. We’re really scaling up into this world of big data and so, we need to scale up our technology in response to that,” Harville said. Among the takeaways:

• The CEC will bring in 30 to 40 terabytes of data annually on top of its existing data collection efforts. It needed a provider capable of warehousing that data and keeping it readily available at an effective price. Snowflake proved the most competitive, Harville said, because it has “decoupled storage from compute,” positively affecting cost. CEC also saves because its new service is managed, eliminating the need to hire database administrators.

“It’s a massive cost savings relative to the alternatives that we considered, and we’re also looking to essentially pay for itself out of modernizing existing data flows,” Harville said. The CEC already has around $60,000 of existing data flows it will migrate to Snowflake, essentially paying for its $50,000 cost.

• The relationship got underway at the end of June, but officials had had “plenty of contact” with the company previously, conducting a proof of concept over two to three months earlier this year to identify all the tech pieces needed to stand up a data pipeline at scale. Snowflake, Harville explained, is “the end of that pipeline,” from where officials will be “serving” data to use cases including visualization and analysis.

“We spend a good amount of time on the front end doing requirements, gathering, discussing success criteria, and what ‘success’ looks like to the CEC team," said Zach Oxman, Snowflake’s sales director for government sector. "We engaged with stakeholders across multiple groups within CEC to solicit feedback and discuss outcomes and the Snowflake solution.” 

• The company’s involvement, Harville said, offers two key advantages to CEC: flexibility and accessibility. Snowflake is a relational database structure, that presents itself as a standard SQL database, the assistant executive director said — making it easily accessible to staff with existing skills; easy to plug into data sharing; and as a foundation for applications. Being able to bring together data not previously combined, Harville said, enables officials to screen for quality and reduce any redundancies or inconsistencies.

The teaming also gives the CEC more flexibility on access, enabling “finer-grain control” of data to ensure its security.

“As CEC grows into a large source of truth for aggregated energy data, with Snowflake they can simply scale and pay as they grow and, more critically, CEC can now share data in real time in a fully secure and governed environment without having to move data,” Oxman said.

• Asked what state and local governments should keep in mind as they move toward doing more with data, Oxman emphasized understanding “use cases and outcomes” — the problems an agency is trying to solve, and how it currently operates. He recommended that officials consider how they share data now; how much time they spend moving data around – and how risky that is; how they use analytical tools; and how much time and money they’re spending on legacy storage or warehouses.

Theo Douglas is Assistant Managing Editor of Industry Insider — California.