Data Lake Transfer Gateway

Architecture

Interval and file structure

The export interval can be either once an hour or once a day. The data format can be the following options:

/yyyy/mm/dd/HH/MM/site/{data-type}.csv // For hourly exports

The date/time of the files are against the update-at date of the record.

Files are generated in CSV format. All tables carry a unique primary key (ID). This key can be used to merge the data when ingesting since the same primary ID record may appear in multiple files in case the record changes.

Export Schema

Activity Instance

Activity Instance is the header of a form submitted. We will only export data for forms that have been submitted or superseded.

You will find here the reference to the order and asset in which this activity was submitted.

Activity Instance Item

This table contains the answers to the questions submitted by the user.

Events

This represents detailed records of events, including their timestamps, types, priorities, associated assets, orders, and statuses and additional metadata. All dates are in UTC formatted as ISO.

Order instance

Represents detailed information about orders, including timestamps, quantities, and associated assets and materials.

Order Performance

Tracks performance metrics of orders on a hourly bucket. It includes the sum of quantities completed and rejected, active and downtime minutes, and rate data. They provide the source data to calculate OEE.

Order Progress

Monitors the minute-by-minute progress of the orders with timestamps, statuses, counts, and associated assets and shifts.

Shift

Includes the shift instances with its Start TIme, and foreign keys.