Code

The code repository can have two types of code files: regular Python scripts and transformations. For a transformation to be identified as such, a function called transform returning a LazyFrame (or DataFrame) needs to be present. The function will be called automatically when appropriate.

import polars as pl

def transform(source_1, source_2):
    return source_1.join(source_2, on="id")

The transform function can have multiple parameters. A parameter to the transform function needs to have the same name as one of the transforms. By declaring a transform as a parameter, the dependency graph will be updated automatically, the transform will be read, and the DataFrame will be passed in as a parameter to the transform function.

The following example shows how a join transform declares a dependency on source_1 and source_2 by specifying those transform names as parameters for the transform function.