DataHub Overview

What is DataHub?

DataHub is Arcadia’s new utility data access product, providing transformed and normalized reports to power your downstream systems. DataHub offers an alternative bulk access mechanism to access your utility data without integrating with Plug API. DataHub provides access to your utility data through two new ingestion options: zipped CSV files delivered to your SFTP server or access to a Snowflake Direct Share to query your data directly in Snowflake export tables.

DataHub Reports

Currently, DataHub offers two distinct reports: ESG and Meter Level Summary.

The ESG tables prepare calendarized, standardized, and aggregated meter-level utility data at monthly, quarterly, and annual timeframes to support ESG scope 2 reports. For example, the Calendarized Monthly Usages table aggregates your meters’ daily usage values and provides monthly level totals for your meters’ usages. This table enables you to understand how much energy the meter uses for a specific month to calculate how much greenhouse gas emissions occur at this service location. Learn more about DataHub’s ESG Tables.

The Meter Level Summary table presents meter measurement period utility data, including consumption categories (e.g., measured total consumption, measured max demand, bidirectional in, etc.), total costs, meter-level custom data, and site-level custom data fields. The Meter Level Summary table solves for duplicate statement scenarios in deregulated markets where the supplier and delivery utility providers report overlapping data for the same meter. This new format includes standardizing units of measure for electric, water, and natural gas service types. Meter Level Summary supports insights like highest peak demand per meter, total consumption per site, and meters enrolled in budget billing to address energy management and carbon accounting use cases. Learn more about DataHub's Meter Level Summary Table.

Prerequisites

Onboard Utility Credentials and Utility Files onto Plug

As a second required step, you will onboard your utility data onto Plug. You will submit utility credentials through Plug’s Connect experience or Create Credential API endpoint. If you do not have utility credentials and instead have the source statement PDFs, you will onboard these statements through Plug’s Bill Uploader module or Add File API endpoint.

Create sites and assign meters to site

In order to group your meters by site ID in DataHub, Arcadia recommends that you create Sites for your relevant geographical locations and map utility meters to the site containers to further organize your utility data in Plug.

Data Access Options

DataHub provides access to your utility data through two new ingestion options: zipped CSV files delivered to your SFTP server or access to a Snowflake Direct Share to query the data directly in Snowflake.

SFTP Delivery

For the SFTP option, set up your own SFTP server. The DataHub dashboard page allows you to select DataHub's ESG or Meter Level Summary reports and provide your SFTP configuration details. If you select the ESG Usage Quarterly report, you receive aggregated meter-level utility data for all statements and the Statement Availability Report upon initial successful setup and on the first of each month moving forward. For example, you will receive two files to your SFTP: datahub_quarterly_usages_02_21_2024-10_13_15.csv.gz and datahub_statement_availability_report_02_21_2024-10_13_15.csv.gz.

For SFTP delivery files, null values are represented as ,"\n", for VARCHAR-like data types and as ,"", for empty string values. Null values are represented as ,\n, for NUMBER-like data types.

SFTP Setup

To set up your SFTP connection with Arcadia, access the Plug Dashboard, navigate to the DataHub Settings page, and click ‘Set up SFTP delivery’. You need to be an admin user to access this dashboard page.

In the modal, you need to share the information below. Please work with your IT or infrastructure department to configure and provide these details. Please work with your Arcadia representative for handling any issues around initial setup.

SFTP server hostname/IP
- e.g. 192.158.1.38 or sftp.example.com
SFTP server host key. This key must be in OpenSSH’s authorized_keys format.
- e.g. ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXdVM9rD7wE1vBcDpfJszq3Hs7F...
The username of the user you have set up for Arcadia to access your SFTP server.
- e.g. arcadia
- Note: you will need to associate Arcadia’s OpenSSH format public key (2048-bit RSA) with this user, so Arcadia may authenticate to your server. Most SFTP servers have an authorized_keys file that you should add this key to. This key can be found under the “Arcadia public key value” field in the SFTP setup modal on your Arc dashboard.
Full path in the above user’s directory that you would like your Arcadia data to be stored.
- e.g. / or /datahub/reports
- You should create this folder and provide the created user write access to it before submitting this information to Arcadia.

Snowflake Data Share

For additional information on Snowflake Data Shares, review Snowflake’s overview guide. For the Snowflake Data Share option, your Snowflake account must be hosted on AWS in us-east-1 availability zone. After receiving access to the private Snowflake Data Share, you can query in SQL or Python against the full dataset in all tables. The Snowflake Data Share table names are datahub_monthly_usages, datahub_quarterly_usages, datahub_annual_usages, datahub_statement_availability_report, and datahub_meter_level_summary.

Snowflake Data Share Setup

To access DataHub’s Snowflake private Data Share, you will need to host your Snowflake account in AWS us-east-1 availability zone. Once you confirm that your Snowflake account is available on this cloud provider and availability zone, you access the Plug Dashboard, navigate to the DataHub Settings page, and complete the Snowflake Data Share form. You need to be an admin user to access this dashboard page.

In your Snowflake environment, run the command below and provide your current organization name, your current account name, and account locator into the form.

select current_organization_name(), current_account_name(), current_account();

Once Arcadia sets up your Snowflake Account for Data Share, Arcadia database is available in the ‘Ready to Get’ section. In Snowflake’s Snowsight UI, you will:

Sign in to Snowsight
Select Data » Private Sharing
Select the Shared with You tab
In the Ready to Get section, select the share that you want to create a database for
Set a database name and the roles that are permitted to access the database
Select Get Data

Once the Data Share is available, you can query all the data in one of the four tables with:

SELECT * FROM datahub.shares.<table_name>;

For additional information on Snowflake Data Shares, review Snowflake’s overview guide.