HomeGuidesAPI ReferenceChangelog
Log In

Time Normalization

Suggest Edits

Why do the dates of my consumption reads not match the account billing dates?
Why aren't all of the charge dates in my invoice dated for the same month?
Why do the charge dates on the meter not match the usage dates on the same meter?

Observations

When modeling utility service data from invoice documents, one of the most critical aspects in creating a new statement object (the default invoice data transaction) involves determining the time span(s) a given invoice encompasses. This could be a straightforward 30-day billing period for a single electric meter, common to most utility invoice accounts, or it could involve an invoice ‘re-billing’ statement that entails several months' worth of activity.

The invoice in question may contain several distinct meters or services and each of the meters may have been measured on different reading schedules. Moreover, the extreme statement may only contain a single date stamp and fail to explicitly outline the full service period. In even more extreme scenarios, an invoice may fail to indicate the period year because the utility billing platforms take for granted that years will assume or infer the calendar year when receiving any invoice.

It is of paramount importance to have a definitive answer to these and other date inquiries prior to interpreting the new invoice’s structure and inserting it into core Arcadia products. This process of interpreting and reconciling the dates within an invoice is known as TimeNormalization.

Regardless of the invoice scenario, Arcadia’s goal is to grant users the ability to measure and manage all expenses and commodity consumption.

TimeNormalization helps promote consistency across these billing formats, and helps customers achieve their reporting goals.

Time Normalization Policy

Mechanics

The mechanics of the TimeNormalization process involve surveying the totality of time series information we have available from a given invoice and then ranking and reconciling any inconsistencies. Because each provider chooses the format of their bills, these inconsistencies are very common.

When natural gas is billed together with electricity at a given property, the meter reading may occur on different schedules even while managed by a single utility. In other cases, multiple meters of the same service type may still schedule readings at different times during the month due to differences in tariffs. Regardless of the cause, at scale users will inevitably encounter invoices containing date stamps omitting the year, datestamps for multiple meters that were not read in synchronization, and invoices containing only a statement for the entire document.

What to do if you disagree with a date policy

After reconciling any of the above irregularities, a secondary goal of time normalization is to offer traces and logs that explain when and where certain corrections, updates, and edits were committed. Customers can then utilize these attributes to verify the date inconsistencies and take their own independent steps to reconcile if they disagree with the methods Arcadia has deployed.

Irrational dates and date omissions have to be resolved to maintain integrity within our data model, but some users may wish to adopt their own strategy. These secondary attributes include the ‘origin’ of each measurement period and the original date source components.

The origin signifies when any meter measurement period was edited or overwritten by the account dates or child usages dates, respectively. And the date source attribute declares if the original measurement period was based upon explicit ‘Start Date’ and ‘End Date’, based upon a ‘SIngle Date’, based upon a ‘declared number of days’, or any combination of these scenarios.

Methods

Establishing rational and consistent invoice structures means first declaring the major modeling concepts in all invoice document graphs. In the Arcadia model, an invoice is made up of several levels or elements: statement elements, account elements, meter elements, usage elements and charge elements. Each of these elements has a distinct relationship to its siblings and these relationships then govern how different date-related data points at different levels interact and impact one another.

One invoice can often send mixed signals with respect to time periods. An exact start and end date may exist at the account level, with only a single date on a usage value and a period duration e.g. 28 days. Evaluating the invoice is accomplished by traversing the statement graph, often multiple times during processing. The first time, we will attempt to record a time interval at each element or level, using only the information at that element, whether it be partial or complete. Candidate data points for capture include current read dates, previous read dates, number of days in period, statement dates, name of calendar month, etc. Preceding from the trunk element this survey moves outward towards the leaf elements, beginning from the Statement element, then to account elements, meter elements, and finally usage and charge elements.

Then, as a secondary pass, time normalization makes decisions about which time periods are the most ‘reliable’ or most appropriate based upon the value and completeness of the data targets at each element. The most reliable time periods will be promoted to adjacent elements, if those elements have a less reliable period or an incomplete period. This reconciliation process may repeat several times if necessary, going up and down the invoice tree while progressively determining what the most reliable time period should be at each level. At the end of this process, every element in the statement graph will have a start and end date for its time period and a corresponding set of flags indicating where those dates were derived (their origin).

Illustrated Example

A simple rule of thumb is that Arcadia places more value on the leaf-nodes of invoice data elements and less value or emphasis on trunk-nodes. As an example, imagine an invoice declares the time period as September 30th, 2023 on page one, proximal to the Amount Due for payment. Subsequent pages in the invoice then introduce a natural gas service and meter containing meter reads for 09/07/2023 and previous meter reads from 08/06/2023. Finally, the same gas meter posts line item usages and charges for 08/06/23 through 08/31/2023 and then separately posts line item usages and charges for 09/01/2023 through 09/10/2023.

Data Model Technical Walkthrough for Illustrative Example

The data model for the hypothetical invoice above would appear as follows:

  • Statement Level measurement period
    • The ORIGIN of the measurement period was ‘single date’, which is a LOW_VALUE origin.
    • That ‘single date’ was 09/30/2023, which was inferred from the data point “September 2023”
      • The initial rendering then places the statement level measurementPeriod as 09/01/2023 through 09/30/2023.
  • Meter Level measurement period
    • The ORIGIN of the measurement period was ‘start date and end date’, which is a HIGH-VALUE origin.’
    • The ‘start date’ was 08/06/2023 and the ‘end date’ was 09/07/2023.
      • The initial rendering then places the meter level measurementPeriod as 08/06/2023 through 09/07/2023.
  • Usage Level measurement period(s)
    • The ORIGIN of usage[1] measurement period was ‘start date and end date’, which is a HIGH-VALUE origin.’
    • The ‘start date’ was 08/06/2023 and the ‘end date’ was 08/31/2023.
      • The initial rendering then places the meter level measurementPeriod as 08/06/2023 through 08/31/2023.
  • The ORIGIN of usage[2] measurement period was ‘start date and end date’, which is a HIGH-VALUE origin.’
  • The ‘start date’ was 09/01/2023 and the ‘end date’ was 09/10/2023.
    • The initial rendering then places the meter level measurementPeriod as 09/01/2023 through 09/10/2023.
  • Charge Level measurement period(s)
    • The ORIGIN of charge[1] measurement period was ‘start date and end date’, which is a LOWEST-VALUE origin.’ Charge items are given the least influence across the entire TimeNormalization graph.
    • The ‘start date’ was 08/06/2023 and the ‘end date’ was 08/31/2023.
      • The initial rendering then places the meter level measurementPeriod as 08/06/2023 through 08/31/2023.
  • The ORIGIN of usage[2] measurement period was ‘start date and end date’, which is a LOWEST-VALUE origin.’ Charge items are given the least influence across the entire TimeNormalization graph.
  • The ‘start date’ was 09/01/2023 and the ‘end date’ was 09/10/2023.
    • The initial rendering then places the meter level measurementPeriod as 09/01/2023 through 09/10/2023.

After the initial rendering for each of the respective elements, the next step is to reconcile the inconsistencies that exist in the invoice.

  • Inconsistencies observed:
    • The statement level was 09/01/2023 through 09/30/2023 (LOW-VALUE)
    • The meter level overlapped the statement with 08/06/2023 through 09/07/2023 (HIGH-VALUE)
    • The usage level overlapped the statement with 09/01/2023 through 09/10/2023 (HIGH-VALUE)
    • The charge level overlapped the statement with 09/01/2023 through 09/10/2023 (LOW-VALUE)
      • The other usage and charge did not introduce any inconsistencies as the period 08/06/2023 through 08/31/2023 was contained by the meter period.
  • Reconciliation steps
    • The highest value leaf-node is granted highest priority. This means the usage item declaring the end date as 09/10/2023 is allowed to overrule the parent meter’s end date of 09.07/2023.
      • The usage end date now overwrites the meter 09/07/2023 with 09/10/2023
      • And the ORIGIN of the meter period is also updated now to “usage”
    • The updated meter period also carries higher priority than the statement period (which is the trunk-node). This means the meter end date (09/10/2023) is allowed to overrule the parent statement’s end date of 09/30/2023.
      • The meter end date now overwrites the statement 09/30/2023 with 09/10/2023
      • And the ORIGIN of the statement period is also updated now to “meter”

Time Normalization is now complete with inconsistencies removed.

  • The statement level is 08/06/2023 through 09/10/2023
  • The meter level overlapped the statement with 08/06/2023 through 09/10/2023 (HIGH-VALUE)
  • The usage level overlapped the statement with 09/01/2023 through 09/10/2023 (HIGH-VALUE)

Conclusion

Time Normalization strives to reconcile inconsistencies in invoice dates, and provide users with a clear and accurate timeline for utility consumption and charges. All individual date targets in an invoice are submitted to audits, determining the most reliable time periods, and adjusting less reliable or incomplete periods accordingly. This process promotes consistency across different billing formats from different utility providers, allowing users to construct calendar based reporting and accurately measure their expenses and consumption. A reliable method of date reconciliation is crucial in providing a clear and accurate rendering of utility invoice data.

Updated 3 months ago