Enrich & Transform

With your sources connected and organized, you can augment the data before it enters training — compute new columns, override how BaseModel interprets existing ones, let it auto-detect event subgroups, or unify entities that appear across multiple tables.

Computed Columns

sql_lambdas let you derive new columns using SQL expressions evaluated by the query engine. Each lambda needs an alias (the new column name) and an expression.

yaml

sql_lambdas:
  - alias: price_float
    expression: "TO_DOUBLE(price)"
  - alias: discount_pct
    expression: "(list_price - price) / list_price"
  - alias: tenure_years
    expression: "date_diff('year', enrolment_date::DATE, DATE '2024-12-01')"

Lambdas run at query time and can use any function your backend supports — casts, arithmetic, date functions, CASE WHEN, etc.

Make use of multiple date columns as features

Apart from the event timestamp (which BaseModel handles automatically), raw date columns are not embedded. Use sql_lambdas to transform them into numeric values — such as tenure in years or days since an event — so the model can learn from them.

Column Type Overrides

BaseModel auto-detects feature types (see Managing Data), but you can override the detection when the default is not ideal.

yaml

column_type_overrides:
  loyalty_tier: categorical
  quantity: decimal

Check integer columns that represent quantities

An integer column like num_items (a count of purchased items) represents a continuous quantity where ordering and magnitude matter. However, BaseModel may classify it as categorical because it contains a small set of distinct whole numbers. Override it to decimal so the model treats it as a numerical feature.

Standard overrides

Override value	When to use
`categorical`	Force a numeric column to be treated as categories (e.g., status codes, tiers)
`decimal`	Force a column to be treated as continuous numeric (e.g., integers that represent quantities)

Rich encoding overrides

These unlock richer feature representations but require explicit opt-in.

Time Series

For numeric columns with meaningful temporal patterns — price history, account balances, sensor readings. Best practice is to create a derived column via sql_lambdas and mark it as time_series, so the model gets both the raw value and the temporal encoding:

yaml

sql_lambdas:
  - alias: price_ts
    expression: price
column_type_overrides:
  price_ts: time_series

BaseModel will extract sequence-based features rather than treating values independently.

Text

For free-form string columns with an average of more than ~5 tokens per record — product descriptions, reviews, customer feedback:

yaml

column_type_overrides:
  description: text

BaseModel generates tokenised semantic embeddings instead of treating the column as a categorical token.

Text columns are skipped by default

BaseModel skips text columns during fitting unless you explicitly opt in with column_type_overrides. If you expect a text column to contribute features but it appears under "Skipped columns" in the columns analysis report, add the override shown above.

Image

For columns containing a path or URL to a visual asset — product photos, property listings, travel destinations:

yaml

column_type_overrides:
  image_url: image

BaseModel accepts local paths, cloud URIs, or web URLs and generates visual embeddings from the image content.

Automatic Event Grouping

When an event table contains diverse event patterns that are hard to separate manually with where_condition, BaseModel can detect groups automatically and process them separately.

When to consider auto-grouping

Use num_groups when your event table contains latent event types characterized by combinations of metadata columns — for example, visits to different store formats, banking transactions that mix card payments with wire transfers, or medical encounters that range from routine checkups to emergency admissions. If you can't easily express the split with a single where_condition, auto-grouping will find the natural boundaries for you.

yaml

- type: event
  name: transactions
  num_groups: 3
  ...

num_groups triggers pattern recognition that divides events into disjoint groups based on frequent combinations of categorical column values. If the data cannot support the requested number of groups, training will fail with a message indicating the maximum allowed value.

Auto-grouping only for event sources

num_groups is only valid for event sources.

Shared Entities

If the same real-world entity (e.g., a product) appears in multiple data sources, BaseModel normally treats each occurrence independently. shared_entities lets you unify them into a single representation, enriching the model's understanding of that entity across all sources.

When to consider shared entities

Use shared_entities when an entity like a product, store, or content item appears across multiple event tables — for example, the same article_id in both a purchases table and a page-views table. Without sharing, BaseModel builds a separate representation of that entity in each source and misses cross-source signals. Sharing unifies those representations, so the model learns that the product a customer browsed is the same one they later bought.

A top-level configuration key

shared_entities is defined at the top level of the config, alongside data_sources — not nested inside individual sources. Each entry declares, per data source, which column identifies the entity (id_columns) and, optionally, which columns to unify (columns).

Basic usage — the same entity in multiple sources

yaml

data_sources:
  - type: event
    name: product_buy
    ...
  - type: event
    name: page_visit
    ...

shared_entities:
  - name: product
    id_columns:
      product_buy: article_id
      page_visit: article_id

Both sources identify the product by article_id, so BaseModel treats those IDs as the same entity across the two tables. The keys of id_columns are data source names; the values are the ID column within each source — the column names may differ from one source to another.

List every source that contains the entity

id_columns must include every data source in which the entity appears — not only the ones you want to unify or pull extra columns from. If a data source contains the entity but is left out of id_columns, BaseModel raises an error. For example, if article_id is present in three event tables but you list only two, the run fails.

Adding extra columns

When to add extra columns

Add extra columns when different event tables hold complementary information about the same entity. For example, a purchases table might have a product name while a page-views table has a product_description. Pulling both into the shared entity gives BaseModel a richer picture of each product than either source provides alone.

Each entry in columns has a unified name and a sources map that says which column feeds it in each data source:

yaml

# product_buy has a "name" column; page_visit has a "product_description" column
shared_entities:
  - name: product
    id_columns:
      product_buy: article_id
      page_visit: article_id
    columns:
      - name: title
        sources:
          product_buy: name
          page_visit: product_description

Here a single shared column title is populated from name in product_buy and from product_description in page_visit. A column that exists in only one source simply lists that one source.

Combining with joined attributes

When an attribute table is joined to the event sources, add it to id_columns (mapped to its own key column) and reference it under each column's sources:

yaml

# "articles" is an attribute table joined to the event sources, keyed by its own "id" column
shared_entities:
  - name: product
    id_columns:
      product_buy: article_id
      page_visit: article_id
      articles: id
    columns:
      - name: category
        sources:
          articles: category
      - name: description
        sources:
          articles: description

No overlapping columns

A given (data source, column) pair may belong to only one shared entity. BaseModel rejects configs that assign the same source column to two different entities.

Parameters

Parameter	Type	Description
`name`	`str`	Identifier for the shared entity.
`id_columns`	`dict`	Maps each data source name to the column identifying this entity in that source. Include every source — event or joined attribute — the entity appears in.
`columns`	`list`	Columns to unify. Each item has a `name` (the shared column name) and a `sources` map from data source name to the column name in that source. Optional — omit to share only the entity's identity.

Updated schema

shared_entities is now a top-level key using id_columns (a map) and {name, sources} column entries. The older style — shared_entities nested inside each data source, with a single id_column and a plain columns list — is no longer accepted. Update existing configs before upgrading.