HomeGuidesAPI ReferenceChangelog
Log In
Guides

DataHub Foundational Tables - Getting Started Guide

Learn about DataHub's Preview Tables & Capabilities

Suggest Edits

🚧

Preview Feature

Arcadia’s new meter centric utility data product, DataHub, is officially in a Preview Release! Arcadia is excited to announce that we support Plug meter level details in Snowflake through this new offering.

Introduction

  1. What is DataHub?
    1. DataHub’s Foundational Tables provide utility data across three discrete tables: Meters by Statement, Meter Usages, and Meter Charges. Meters by Statement table records display a specific utility meter’s total measured usage, total cited usage, total demand usage, bidirectional in, bidirectional out, and several additional fields to tell the full story about this meter and the energy consumed at the meter’s service location. To validate the meter level totals available in Meters by Statement table, the Meter Usages and Meter Charges tables provide the ability for a user to search on the relevant Meter ID, Site ID, Meter Data ID, or another field to dig into the meter’s line item usage or charge details. DataHub also provides a link to the statement source file (e.g. PDF) on all records in the three tables.
    2. DataHub enables you to access utility data through two new ingestion options: incremental daily zipped CSV files for all three tables delivered to your SFTP server or Snowflake Data Share direct table access. In the daily incremental files option, the customer receives historical records for all tables for the initial compressed file delivery and also receives new and updated records for each table every day moving forward as compressed files. For the Snowflake Data Share option, your Snowflake account must be hosted on AWS in us-east-1 availability zone. You receive access to a private Snowflake Data Share where you can query in SQL or Python against the full dataset in all three tables. The Snowflake Data Share table paths are:
      1. datahub.shares.uc_meters_by_statement_current
      2. datahub.shares.uc_usages_current
      3. datahub.shares.uc_charges_current
  2. Prerequisites
    1. [Required Steps] Enable Proration and Inference Features & Onboard Utility Credentials and Utility Files onto Plug
      1. DataHub utilizes Plug’s database as the system’s source of truth. As an initial step, ensure that your organization is enabled for proration and inference features. As a next step, you will onboard your utility data onto Plug. You will submit utility credentials through Plug’s Connect experience or Create Credential API endpoint. If you do not have utility credentials and instead have the source statement PDFs, you will onboard these statements through Plug’s Bill Uploader module or Add File API endpoint.
    2. [Recommended Optional Step] Create sites and assign meters to site
      1. In order to group your meters by site ID in DataHub, Arcadia recommends that you create sites for your relevant geographical locations and map utility meters to the site containers to further organize your utility data in Plug.

DataHub - Standardize Unit of Measure Features

  1. Standardizing units of measure on meter usages enables DataHub to better serve the carbon accounting use case by providing more accurate usage totals in the same unit of measure. Arcadia derives energy conversion factors from a combination of Energy Star’s US conversions table, Nist.gov conversions factors, and assumes that gallons are measured in US Liquid Gallons. For conversions that are volume to energy (e.g. natural gas) or mass to energy (e.g. steam), Arcadia is leveraging Energy Star’s conversion factors. Arcadia now supports standardizing units of measure where possible for the following service types:
    1. Electric/Lighting: kWh for consumption and kW for demand
    2. Natural Gas: therms
    3. Water/Sewer/Irrigation: Gallons
  2. DataHub does not support unit of measure conversions for reactive max measured demand and reactive total consumption values. This includes the following units of measure: kvar, kva, kvarh, kvah, mvarh, undefined, unit, and days.
  3. DataHub also does not support standardizing the following units of measure: horsepower, residential cooling hecta liters, nm3/h, sm3, m3/h, kgh, and kg.

SFTP Incremental Daily Delivery Files Setup

In order to set up your SFTP connection with Arcadia, you need to share the information below with your Arcadia contact. Reach out to your Arcadia contact to receive Arcadia’s public key (2048-bit RSA). Please work with your IT or infrastructure department to configure and provide these details. Please work with your Arcadia representative for handling any issues around initial setup.

  • SFTP server hostname/IP
  • SFTP server public key.
    • e.g. ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXdVM9rD7wE1vBcDpfJszq3Hs7F...
    • Note that Arcadia supports RSA only
  • The username of the user you have set up for us to access your SFTP server.
    • e.g. arcadia
    • Note that you will need to associate Arcadia’s public key (2048-bit RSA) with this user, so Arcadia may authenticate to your server. Reach out to your Arcadia contact to receive Arcadia’s public key.
  • Full path in the above user’s directory that you would like your Arcadia data to be stored.
    • e.g. / or /datahub/reports
    • You should create this folder and provide the created user write access to it before submitting this information to Arcadia.

Snowflake Data Share Tables Access

  1. To access DataHub’s Snowflake private Data Share, you will need to host your Snowflake account in AWS us-east-1 availability zone. Once you confirm that your Snowflake account is available on this cloud provider and availability zone, run the command below in Snowflake and provide your current organization name and your current account name to your Arcadia contact to setup the Data Share. 

    select current_organization_name(), current_account_name();

  2. Once Arcadia sets up your Snowflake Account for Data Share, Arcadia database is available in the ‘Ready to Get’ section. In Snowflake’s Snowsight UI, you will:

    1. Sign in to Snowsight
    2. Select Data » Private Sharing
    3. Select the Shared with You tab
    4. In the Ready to Get section, select the share that you want to create a database for
    5. Set a database name and the roles that are permitted to access the database
    6. Select Get Data

  3. Once the Data Share is available, you can query all the data in one of the three tables with:

    SELECT * FROM datahub.shares.<table_name>;

  4. For additional information on Snowflake Data Shares, review Snowflake’s overview guide.

Data Dictionary

Recipes

Updated 8 months ago