Configuration Files

Every workspace has two files at its root next to your transforms. _config.json describes the environment your transforms run in, and _connections.json records which transforms exist and what they read.

_config.json

_config.json sets up the Python environment of every build in the workspace.

{
  "confVersion": "1.0",
  "python": "3.11",
  "dfType": "polars",
  "draftReadLimit": 1000000,
  "packages": [
    "plotly",
    "requests",
    "git+https://github.com/dataspaceswiss/pyrunner-lib@v0.6.1#egg=pyrunner-lib"
  ]
}

Field	Required	Description
`confVersion`	Yes	Version of the file format, written as `major.minor`. Currently `1.0`.
`python`	Yes	The Python version builds run on, for example `3.11`.
`dfType`	Yes	`polars` or `pandas`. Decides what your transform’s parameters receive: a `polars.LazyFrame` or a `pandas.DataFrame`.
`packages`	Yes	The Python packages to install, one entry per package in the format `pip` accepts: a name, a pinned version (`requests==2.32.3`) or a Git URL.
`draftReadLimit`	No	The number of rows a Draft Build reads from each input. Defaults to 1,000,000; `-1` reads every row.

The Python version and the packages make up the build image. When you change either of them, the next build first prepares a new image, so it takes longer than usual. Builds after that reuse the image.

_connections.json

_connections.json lists every transform of the workspace with its file and its inputs. DataSpace builds the lineage graph from it and uses it to decide the order of builds.

[
  {
    "id": "rid.transform.c4d45b58-3f1a-432e-aacc-b792075ba14f",
    "path": "raw_events.py",
    "inputs": []
  },
  {
    "id": "rid.transform.69773696-0234-4af4-bfbe-383ab02aaf2b",
    "path": "clean_events.py",
    "inputs": ["rid.transform.c4d45b58-3f1a-432e-aacc-b792075ba14f"]
  }
]

Field	Description
`id`	The transform’s RID. It stays the same when the file moves.
`path`	The transform’s file, relative to the workspace root.
`inputs`	The RIDs of the transforms whose datasets this transform reads.

DataSpace keeps this file up to date, so you do not edit it by hand:

Creating, renaming, moving or deleting a node updates its entry.
Saving a transform’s code updates its inputs from the transform function’s parameters. A parameter named after a transform reads that transform, and saving refuses a parameter that names no transform. To read a transform by its RID instead, for example one from another workspace, annotate the parameter:

from typing import Annotated

import polars as pl
from pyrunner_lib import Transform


def transform(
    clean_events,
    customers: Annotated[pl.LazyFrame, Transform("rid.transform.442c7deb-2e1e-42e2-b8ff-57bf1bc72e56")],
):
    ...