YAML Configuration Reference

The foundation model is configured entirely via a single YAML file. This file is passed to pretrain() as the config_path parameter.

Top-Level Structure

data_sources: [...]                # Required — your data tables
shared_entities: [...]             # Optional — unify an entity across data sources
data_params: {...}                 # Required — date ranges and splits
data_loader_params: {...}          # Optional — batch loading settings
training_params: {...}             # Optional — training hyperparameters
memory_constraining_params: {...}  # Optional — model size and memory limits
query_optimization: {...}          # Optional — query parallelization
calibration_params: {...}          # Optional — automatic DataLoader tuning

`data_sources`

A list of data source definitions. At least one event data source is required.

Data Source Fields

Field	Type	Required	Description
`type`	`str`	Yes	Source type: `"event"`, `"main_entity_attribute"`, or `"attribute"`.
`name`	`str`	Yes	Unique name for this data source. Referenced in target functions and joins.
`data_location`	`object`	Yes	Database type and connection parameters. See below.
`main_entity_column`	`str`	Yes (event, main_entity_attribute)	Column identifying the main entity (e.g., `customer_id`).
`date_column`	`object`	Yes (event only)	Timestamp column configuration. See Date Column Formats.
`allowed_columns`	`list[str]`	No	Only these non-reserved columns will be used. Do not include the source's `main_entity_column` or `date_column`; keep them under their dedicated keys. This may be combined with `disallowed_columns` to exclude a raw `extra_columns` field while keeping an explicit fitted shortlist.
`disallowed_columns`	`list[str]`	No	These columns will be excluded. Auto-populated with bijection columns in `suggested_config.yaml`.
`column_type_overrides`	`dict`	No	Override inferred column types with `categorical`, `categoricalCompressed`, `decimal`, `time_series`, `text`, or `image` (for example, `{article_id: categoricalCompressed}`). Auto-populated with time-series candidates in `suggested_config.yaml`.
`joined_data_sources`	`list`	No	Attribute tables to join. See Joining Tables.
`sql_lambdas`	`list`	No	SQL-based computed columns. Each alias is added as a model input automatically; do not repeat an alias in `allowed_columns`.
`where_condition`	`str`	No	SQL WHERE clause to filter rows at query time.
`partition_column`	`str`	No	Event and `main_entity_attribute` sources. Column used to split queries into chunks. Defaults to `main_entity_column`. From 1.14, it may also be listed in `disallowed_columns`: it is then used for splitting but not as a model input. See Data Partitioning.
`partition_values_transformation`	`str`	No	Event and `main_entity_attribute` sources. How `partition_column` values are assigned to chunks. `hash_mod` (default): the value is hashed and taken modulo the number of chunks. `exact_int`: with more than one chunk, only for integers numbered consecutively from 0 to the number of chunks minus 1; with a single chunk, all rows are read whatever their values (in 1.13 and earlier, only rows with the value `0`). `exact_int_mod`: integers, assigned by value modulo the number of chunks. `exact_group`: distinct values are divided into chunks by number of occurrences; avoid on high-cardinality columns.
`num_groups`	`int`	No	Event sources only. Number of disjoint groups, found from frequent combinations of categorical column values, into which events are divided when computing decimal features and histograms. See Automatic Event Grouping.

`data_location`

Field	Type	Description
`database_type`	`str`	One of: `parquet`, `snowflake`, `bigquery`, `databricks`, `hive`, `clickhouse`, `synapse`.
`connection_params`	`object`	Database-specific parameters. See Data Connectors.
`table_name`	`str`	Table name in the database.
`schema_name`	`str`	Schema name (required for some databases).

`date_column`

Field	Type	Description
`name`	`str`	Column name containing the event timestamp.
`format`	`str`	Date format string (e.g., `"%Y-%m-%d"`, `"unix"`).

The declared date_column supplies event order and temporal decomposition, but the public contract does not promise a separate weekday feature. When calendar weekday must be an explicit model input for DuckDB/Parquet, derive an alias with sql_lambdas, as in the event source below; DuckDB numbers Sunday as 0 and Saturday as 6.

Source Type: `event`

Timestamped behavioral data. Multiple event sources are supported. Each row represents a single event for an entity.

- type: event
  name: transactions
  data_location:
    database_type: parquet
    connection_params:
      path: "/data/transactions.parquet"
      cache_path: "/basemodel/db_cache/"
    table_name: transactions
  main_entity_column: customer_id
  date_column:
    name: t_dat
    format: "%Y-%m-%d"
  sql_lambdas:
    - alias: event_is_weekend
      expression: "CASE WHEN dayofweek(CAST({{ resolve_fn('t_dat') }} AS DATE)) IN (0, 6) THEN 1 ELSE 0 END"
  column_type_overrides:
    event_is_weekend: categorical
  disallowed_columns: ["order_id"]
  joined_data_sources:
    - name: articles
      join_on:
        - [article_id, article_id]

Source Type: `main_entity_attribute`

Static or slowly-changing entity properties (one row per entity). Optional.

- type: main_entity_attribute
  name: customers
  data_location:
    database_type: parquet
    connection_params:
      path: "/data/customers.parquet"
      cache_path: "/basemodel/db_cache/"
    table_name: customers
  main_entity_column: customer_id

Source Type: `attribute`

Dimension tables used in joins with event data sources. Optional.

- type: attribute
  name: articles
  allowed_columns: ["product_type_name", "product_group_name",
                    "department_name", "section_name",
                    "colour_group_name", "perceived_colour_master_name"]
  data_location:
    database_type: parquet
    connection_params:
      path: "/data/articles.parquet"
      cache_path: "/basemodel/db_cache/"
    table_name: articles

SQL Lambdas

Compute derived columns using SQL expressions:

sql_lambdas:
  - alias: price_time_series
    expression: price
  - alias: total_price
    expression: "CAST({{ resolve_fn('price') }} * {{ resolve_fn('quantity') }} AS FLOAT)"

`shared_entities`

A top-level list, defined alongside data_sources, that unifies a real-world entity appearing in more than one data source into a single representation.

Field	Type	Required	Description
`name`	`str`	Yes	Identifier for the shared entity.
`id_columns`	`dict`	Yes	Maps each data source name to the column identifying this entity in that source. Must list every data source that contains the entity, not just a subset.
`columns`	`list`	No	Columns to unify. Each item has a `name` and a `sources` map (data source → column name).

shared_entities:
  - name: product
    id_columns:
      product_buy: article_id
      page_visit: article_id
      articles: id
    columns:
      - name: category
        sources:
          articles: category

See Shared Entities for a full walkthrough.

`data_params`

Controls the data range and how data is split for training/validation/testing.

Field	Type	Required	Description
`data_start_date`	`str`	Yes	Earliest timestamp to include (e.g., `"2018-09-20 00:00:00"`).
`split`	`object`	Yes	Split configuration. See below.

Entity Split

Splits entities randomly into training and validation sets. Recommended for most use cases.

data_params:
  data_start_date: "2018-09-20 00:00:00"
  split:
    type: entity
    training: 90
    validation: 10
    training_validation_end: "2020-09-04 00:00:00"
    test:
      start_date: "2020-09-05 00:00:00"
      end_date: "2020-09-22 00:00:00"

Field	Type	Description
`type`	`"entity"`	Entity-based split.
`training`	`float`	Percentage of entities for training (0.01–99.99). Accepts whole numbers (`90`) or fractional values (`90.5`).
`validation`	`float`	Percentage of entities for validation (0.01–99.99). `training` + `validation` must be ≤ 100.
`training_validation_end`	`str`	End date for training and validation data. Must be strictly before `test.start_date` — equal boundaries are rejected as a data-leakage check.
`test.start_date`	`str`	Start of the test window. Entities are scored at a single point in time, not across the whole window — see the note under Time Split.
`test.end_date`	`str`	End of the test window. Bounds the data used to build test targets; it does not add further scoring dates.

Time Split

Splits data by time ranges. Each mode has a specific date window.

data_params:
  data_start_date: "2018-09-20 00:00:00"
  split:
    type: time
    training:
      start_date: "2018-09-20 00:00:00"
      end_date: "2020-06-30 00:00:00"
    validation:
      start_date: "2020-07-01 00:00:00"
      end_date: "2020-09-04 00:00:00"
    test:
      start_date: "2020-09-05 00:00:00"
      end_date: "2020-09-22 00:00:00"

How validation and test entities are scored

With a time split, each entity is scored at a single point in time — the start_date of the validation or test window. The window's end_date only bounds the data used to build targets for that window; it does not spread scoring across the window. A "three-month" test window is therefore evaluated at one date (its start_date), not across the whole quarter.

With an entity split, validation instead reuses the training split-point generator, so each validation entity can have multiple split points. The test and predict sets are always scored at a single fixed timestamp, regardless of split type.

Extra Columns

Columns preserved for the target function via .extra. Declaring an extra_columns entry does not by itself remove that source column from feature inference. For target-only data, also list the same column in the source's disallowed_columns; a column may appear in both settings.

data_params:
  extra_columns:
    - data_source_name: transactions
      columns:
        - order_id

Use distinct names for target and model inputs

fit_columns and extra_columns must be disjoint. If one raw value is needed by both the target and the model, keep the raw column target-only with extra_columns plus the source's disallowed_columns, then expose the same value to the model through a differently named source sql_lambdas alias. A sql_lambdas alias is already a fitted model input, so it must be absent from allowed_columns as well.

# The generated alias is intentionally absent from allowed_columns.
data_sources:
  - type: event
    name: transactions
    main_entity_column: customer_id
    date_column:
      name: event_date
      format: "%Y-%m-%d"
    allowed_columns: [price, sales_channel_id]
    disallowed_columns: [article_id]
    sql_lambdas:
      - alias: article_id_model
        expression: "{{ resolve_fn('article_id') }}"
    column_type_overrides:
      article_id_model: categoricalCompressed
    data_location:
      database_type: parquet
      connection_params:
        path: "/data/transactions.parquet"
        cache_path: "/workspace/cache/transactions"
      table_name: transactions

data_params:
  extra_columns:
    - data_source_name: transactions
      columns: [article_id]

For a recommendation target, .extra["article_id"] is a raw NumPy array and cannot be passed to sketch(). Keep it only for raw target logic. Pass the fitted alias instead: future["transactions"]["article_id_model"]. See Target Function → Extra Columns.

Additional Parameters

Optional fields on data_params that tune split-point generation, example weighting, and sampling. All have sensible defaults — set them only when you need to change the default behavior.

Field	Type	Default	Description
`target_sampling_strategy`	`str`	`"random"`	How split points are sampled per entity: `"random"`, `"valid"`, or `"existing"`. Foundation model training supports only `"valid"` and coerces another value to `"valid"`; classification and regression scenario training can use `"random"`.
`maximum_splitpoints_per_entity`	`int`	`20`	Maximum number of split points generated per entity. With `target_sampling_strategy="random"` the count is `min(√(event count), this value)`; with `"valid"` and `"existing"` it is capped directly at this value.
`split_point_data_sources`	`list[str]`	`null`	Event sources whose timestamps may become split points. Defaults to all event sources.
`minimum_splitpoint_date`	`str`	`null`	Discard split-point candidates earlier than this date.
`ignore_entities_without_history`	`bool`	`true`	Drop entities that have no history before their split point.
`dynamic_events_sampling`	`bool`	`true`	Dynamically sample events per example to reduce overfitting.
`apply_event_count_weighting`	`bool`	`false`	Weight training examples by the entity's event count.
`apply_recency_based_weighting`	`bool`	`false`	Weight training examples by recency.
`window_shuffling_buffer_size`	`int`	`100000`	Size of the buffer used to shuffle event windows.
`limit_entity_num_events`	`int`	`null`	Keep only the N most recent events per entity.
`cache_path`	`str`	`null`	Cache queried data at this path (used by the fit and target-verification steps).
`history_future_split_overrides`	`object`	`{}`	Per-source split-edge and history-limit overrides. See Split Overrides.

Split Overrides

history_future_split_overrides fine-tunes, per data source, which events fall on the history vs. future side of a split point and how far back history reaches:

data_params:
  history_future_split_overrides:
    split_point_inclusion_overrides:
      product_buy: future
      page_visit: history
    entity_history_limit:
      page_visit: 2592000   # seconds (30 days)
    max_data_splits_per_split_point: 200

Field	Type	Default	Description
`split_point_inclusion_overrides`	`dict`	`{}`	Per event source, which side of the split point its events belong to: `history`, `future`, `one-future`, or `one-future-all-variants`. The schema documents an unset source as `future` for recommendation tasks and `history` otherwise. In Monad 1.14, an unset source assigns split-timestamp events to `future` for every task type. Set an explicit override whenever that edge affects the label.
`entity_history_limit`	`dict`	`{}`	Per event source, how far back (in seconds) history reaches — useful to simulate a data-retention policy.
`max_data_splits_per_split_point`	`int`	`200`	Maximum number of data splits generated per split point.

For a question whose history includes the cutoff day and whose outcome starts after it, pin the event source instead of relying on the default:

data_params:
  history_future_split_overrides:
    split_point_inclusion_overrides:
      transactions: history

`data_loader_params`

Controls how data is batched and loaded during training.

Field	Type	Default	Description
`batch_size`	`int`	`256`	Number of entities per batch. Increase for faster training (watch GPU memory).
`val_batch_size`	`int \\| None`	`None`	Batch size for validation. Defaults to `batch_size` if not set. Increase for faster validation without affecting training memory.
`num_workers`	`int`	`5`	Number of data-loading worker processes. Increase for faster data throughput.
`pin_memory`	`bool`	`True`	Whether to copy tensors into pinned host memory so that CPU-to-GPU copies can run asynchronously. The default changed from `False` to `True` in 1.14.
`drop_last`	`bool`	`False`	Whether to drop the last incomplete batch.
`prefetch_factor`	`int \\| None`	`2`	Number of batches to prefetch per worker.
`pin_memory_device`	`str`	`""`	Device to pin memory to (e.g., `"cuda"`). Only relevant when `pin_memory=True`.
`worker_init_fn`	`Callable \\| None`	`None`	Custom initialization function called in each data-loading worker.

data_loader_params:
  batch_size: 256
  num_workers: 4

`training_params`

Training hyperparameters. For the full reference, see Training Parameters.

training_params:
  learning_rate: 0.0003
  epochs: 1
  strategy: "ddp"
  devices: [0, 1]            # Default is "auto" (picks least-occupied GPU)
  precision: "bf16-mixed"
  limit_train_batches: 5    # Remove after validating setup
  limit_val_batches: 5      # Remove after validating setup

`memory_constraining_params`

Control model size and memory usage.

Field	Type	Default	Description
`hidden_dim`	`int`	`2048`	Model hidden dimension. Reduce if GPU memory is insufficient.
`num_layers`	`int`	`4`	Number of hidden layers in the transformer model.
`emde_quality`	`float`	`1.0`	Feature density estimation quality. Lower values reduce memory at cost of accuracy.

memory_constraining_params:
  hidden_dim: 2048
  num_layers: 4

Tip

If you encounter CUDA Out of Memory errors, try reducing hidden_dim to 1024 or 512, or reducing batch_size in data_loader_params.

`query_optimization`

Optimize database query performance for large datasets.

Field	Type	Default	Description
`cleora_num_query_chunks`	`int`	`1`	Split the fit/embedding (Cleora) query into N chunks for parallelization. Renamed from `num_query_chunks`.
`data_loading_num_query_chunks`	`int`	`1`	Split the data-loading query (train, validation, test, predict) into N chunks. Mid-epoch resume is unsupported when greater than 1.
`num_cpus`	`int`	`4`	Number of CPU cores for parallel query execution.
`num_concurrent_features`	`int`	`4`	Number of feature columns processed concurrently.
`sampling_params`	`SamplingParams`	`SamplingParams()`	Entity and history sampling limits for large datasets. Fields: `num_entities`, `history_limit`.

query_optimization:
  cleora_num_query_chunks: 4
  data_loading_num_query_chunks: 1
  num_cpus: 8

Breaking change in 1.7

num_query_chunks was removed. Configurations that set it must migrate to cleora_num_query_chunks (fit phase) and data_loading_num_query_chunks (train/validation/test/predict). Unknown fields are rejected, so an old configuration will fail validation until updated.

`calibration_params`

Automatically finds the optimal DataLoader num_workers and prefetch_factor before foundation model training. When enabled, BaseModel benchmarks multiple configurations and applies the most efficient one.

Field	Type	Default	Description
`enabled`	`bool`	`false`	Enable DataLoader calibration before training.
`candidate_workers`	`list[int]`	`[0,2,4,6,8,10,12,16,32]`	Worker counts to benchmark. `0` is always tested as baseline.
`prefetch_factors`	`list[int]`	`[2, 4]`	Prefetch factor values to sweep for non-zero workers.
`warmup_batches`	`int`	`10`	Batches to skip before measuring (JIT/cache warm-up).
`timeout_seconds`	`float`	`20.0`	Max wall-clock seconds per configuration.
`max_measure_batches`	`int`	`200`	Max batches to measure per configuration.
`cap_workers_by_ram`	`bool`	`true`	Estimate per-worker buffer memory and cap workers to avoid OOM.
`ram_safety_margin`	`float`	`0.2`	Fraction of available RAM to keep free when computing the worker cap.
`memory_probe_count`	`int`	`10000`	Max unique entities to sample for memory estimation.
`cpu_margin`	`float`	`0.2`	Fraction of CPUs to reserve (e.g., 0.2 = use 80 % for workers).
`efficient_threshold`	`float`	`0.90`	Selects cheapest config achieving ≥ 90 % of peak throughput.
`plateau_threshold`	`float`	`0.05`	Relative improvement below which a config is considered flat (early stopping).
`plateau_min_configs`	`int`	`3`	Min successful configs before plateau stopping can trigger.
`max_consecutive_failures`	`int`	`3`	Abort sweep after this many consecutive failures.
`seed`	`int`	`42`	Seed for reproducibility of calibration runs.

calibration_params:
  enabled: true
  candidate_workers: [0, 4, 6, 8, 12, 14]
  prefetch_factors: [1, 2]
  warmup_batches: 10
  timeout_seconds: 15.0
  cap_workers_by_ram: true
  ram_safety_margin: 0.2
  memory_probe_count: 20000

Tip

Start with just enabled: true — the defaults work well for most setups. Customize candidate_workers only if you know your hardware limits, and adjust ram_safety_margin if you need tighter or looser memory headroom.

Complete Example

A full, annotated configuration using Parquet with joins and entity split:

# ---- Data Sources ----
data_sources:

  # Events (mandatory, at least one)
  - type: event
    name: transactions
    data_location:
      database_type: parquet
      connection_params:
        path: "/data/transactions.parquet"
        cache_path: "/basemodel/db_cache/"
      table_name: transactions
    main_entity_column: customer_id
    date_column:
      name: t_dat
      format: "%Y-%m-%d"
    disallowed_columns: ["order_id"]
    sql_lambdas:
      - alias: price_time_series
        expression: price
    column_type_overrides:
      price_time_series: time_series
    joined_data_sources:
      - name: articles
        join_on:
          - [article_id, article_id]

  # Main entity attributes (optional)
  - type: main_entity_attribute
    name: customers
    data_location:
      database_type: parquet
      connection_params:
        path: "/data/customers.parquet"
        cache_path: "/basemodel/db_cache/"
      table_name: customers
    main_entity_column: customer_id

  # Attribute / dimension table (optional)
  - type: attribute
    name: articles
    allowed_columns: ["product_type_name", "product_group_name",
                      "department_name", "section_name",
                      "colour_group_name", "perceived_colour_master_name"]
    data_location:
      database_type: parquet
      connection_params:
        path: "/data/articles.parquet"
        cache_path: "/basemodel/db_cache/"
      table_name: articles

# ---- Data Parameters ----
data_params:
  data_start_date: "2018-09-20 00:00:00"
  split:
    type: entity
    training: 90
    validation: 10
    training_validation_end: "2020-09-04 00:00:00"
    test:
      start_date: "2020-09-05 00:00:00"
      end_date: "2020-09-22 00:00:00"

# ---- Data Loader ----
data_loader_params:
  batch_size: 256
  num_workers: 4

# ---- Training ----
training_params:
  learning_rate: 0.0003
  epochs: 3
  precision: "bf16-mixed"
  strategy: "ddp"
  devices: [0, 1]

# ---- Memory Constraints ----
memory_constraining_params:
  hidden_dim: 2048

# ---- DataLoader Calibration (optional) ----
calibration_params:
  enabled: true

YAML Configuration Reference

Top-Level Structure

data_sources

Data Source Fields

data_location

date_column

Source Type: event

Source Type: main_entity_attribute

Source Type: attribute