Transformation

A node in the Workspace lineage graph represents a Transformation, which is essentially a Python function that accepts input data frames and outputs one data frame.

While the main purpose of a Transformation is to transform data, the system is flexible enough to also serve as Ingests, Load, Artifact Storage, Plots, and Dashboards.

On a technical level, a Transform is a Docker container running user code and generating the following resources under the path:

/data/{TRANSFORM_ID}/

• code/

  • Contains the full git repository.

• datasets/

  • Gets populated automatically by turning the user-returned LazyFrame into a Parquet file.

• logs/

  • Stores the log.txt file containing logging data from executing the user code.

• meta/

  • Contains columns.json file describing the column-level relationship to other Transforms.

• artifacts/

  • Can be used to store intermediate resources or HTML files.