Data Connectors

BaseModel reads data from multiple database types. This page documents the connection_params block for each supported database within the YAML configuration.

Overview

Parquet

Read Parquet files from a local or mounted filesystem, or directly from S3. Supports single files, directories, and glob patterns.

data_location:
  database_type: parquet
  connection_params:
    path: "/data/transactions.parquet"
    cache_path: "/basemodel/db_cache/"
  table_name: transactions

Glob pattern example:

connection_params:
  path: "/data/transactions/*.parquet"
  cache_path: "/basemodel/db_cache/"

Reading Parquet from S3

From 1.14, path accepts an s3:// URI. BaseModel passes the URI to DuckDB unchanged, and DuckDB reads the objects through its httpfs extension.

data_location:
  database_type: parquet
  connection_params:
    path: "s3://my-bucket/transactions/*.parquet"
  table_name: transactions

• URI form. Write the URI as a plain string with two slashes after the scheme: s3://bucket/key. A value in which the double slash has collapsed to one (s3:/bucket/key, which is what wrapping the URI in a Python pathlib.Path produces) is rejected with a validation error. gs:// and az:// URIs are kept as they are and passed to DuckDB in the same way; this section covers S3 only.
• Credentials. BaseModel does not pass S3 credentials to DuckDB. The httpfs extension reads them from the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, and AWS_DEFAULT_REGION. config_overrides is passed to DuckDB unchanged, so DuckDB S3 settings such as s3_endpoint (for S3-compatible storage) can be set there. The storage configuration file does not apply to Parquet path.
• Caching. Without cache_path, queries read straight from the bucket. With cache_path, an s3:// source is loaded into the local cache once and reused on later runs; BaseModel does not check the bucket for changes, so new or modified objects are ignored. Delete the cache_path directory to force a reload.

DuckDB temporary files

When a query needs more memory than its limit allows, DuckDB writes temporary (spill) files to disk. From 1.14, each DuckDB instance that BaseModel opens gets its own subdirectory for these files, so concurrent instances do not overwrite each other's files. The subdirectory is created under the first of:

1. temp_directory in config_overrides, if set;
2. the MONAD_DUCKDB_TEMP_DIRECTORY environment variable, if set;
3. monad_duckdb in the system temporary directory ($TMPDIR, usually /tmp).

By default, spill files therefore go to the system temporary directory. If the container's /tmp is small, point temp_directory or MONAD_DUCKDB_TEMP_DIRECTORY at a volume with enough free space. Set temp_directory: "" in config_overrides to disable spilling.

Snowflake

data_location:
  database_type: snowflake
  connection_params:
    user: "${SNOWFLAKE_USER}"
    password: "${SNOWFLAKE_PASSWORD}"
    account: "${SNOWFLAKE_ACCOUNT}"
    warehouse: "${SNOWFLAKE_WAREHOUSE}"
    database: "${SNOWFLAKE_DATABASE}"
    schema: "${SNOWFLAKE_SCHEMA}"
    role: "${SNOWFLAKE_ROLE}"
  table_name: transactions

BigQuery

data_location:
  database_type: bigquery
  connection_params:
    filename: "/secrets/bigquery-user.json"
    project_id: "my-bigquery-project"
  table_name: transactions
  schema_name: my_dataset

Databricks

data_location:
  database_type: databricks
  connection_params:
    host: "${DATABRICKS_SERVER_HOSTNAME}"
    warehouse_id: "${DATABRICKS_WAREHOUSE_ID}"
    token: "${DATABRICKS_TOKEN}"
    extra_connect_params:
      _socket_timeout: 300   # seconds, passed straight to databricks.sql.connect
  table_name: transactions

# Personal access token authentication
data_location:
  database_type: databricks
  connection_params:
    host: "${DATABRICKS_SERVER_HOSTNAME}"
    warehouse_id: "${DATABRICKS_WAREHOUSE_ID}"
    token: "${DATABRICKS_TOKEN}"
  table_name: transactions

# Service principal authentication
data_location:
  database_type: databricks
  connection_params:
    host: "${DATABRICKS_SERVER_HOSTNAME}"
    warehouse_id: "${DATABRICKS_WAREHOUSE_ID}"
    client_id: "${DATABRICKS_CLIENT_ID}"
    client_secret: "${DATABRICKS_CLIENT_SECRET}"
  table_name: transactions

Hive

HiveParamsConfig

KerberosParamsConfig

# DSN-based configuration
data_location:
  database_type: hive
  connection_params:
    hive_params:
      DSN: "MyHiveDSN"
    ini_file: "/etc/odbc.ini"
    kerberos_params:
      user: "hive_user@REALM"
      kinit_realm: "REALM"
      kerberos_host: "10.0.0.1"
      kerberos_service_name: "hive"
      kerberos_fqdn: "hive-server.example.com"
      keytab_path: "/etc/security/keytabs/hive.keytab"
  table_name: transactions

ClickHouse

data_location:
  database_type: clickhouse
  connection_params:
    host: "${CLICKHOUSE_HOST}"
  table_name: transactions

Synapse

data_location:
  database_type: synapse
  connection_params:
    server_name: "my-synapse.sql.azuresynapse.net"
    database_name: "my_pool"
    user: "${SYNAPSE_USER}"
    password: "${SYNAPSE_PASSWORD}"
  table_name: transactions

Storage Configuration

The storage configuration file holds credentials and options for the cloud filesystems that BaseModel reads and writes through fsspec. Pass the file with --storage-config-path on the command line, or as storage_config_path in the Python API (for example in pretrain). It does not configure DuckDB: for Parquet path values on S3, see Reading Parquet from S3. It also does not let you keep the pretraining configuration YAML on object storage: the path passed to pretrain or --config-path must be a local or mounted file, because BaseModel copies it into the output directory with a plain file copy before reading it.

The file is YAML. Its top-level keys are storage protocols; include only the ones you use:

• Every key in the table except store_name falls back to the environment variable of the same name when it is missing from the block or written without a value.
• ${VAR} references in string values are replaced with the value of the environment variable when the file is loaded, as in connection_params.
• Any other key in a block is passed to the underlying fsspec filesystem unchanged.

s3:
  AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
  AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}

az:
  AZURE_STORAGE_ACCOUNT_NAME: my-account
  AZURE_STORAGE_SAS_TOKEN: ${AZURE_STORAGE_SAS_TOKEN}

JSON Schema

From 1.14, the image ships a JSON Schema for this file, storage-config.schema.json, which editors that support JSON Schema for YAML can use for autocompletion and validation. To extract it from the image:

docker run --rm -e MONAD_LOGGING_DIRECTORY=/tmp/monad-logs \
  --entrypoint python <monad-image> \
  -c "import importlib.resources as r; print(r.files('monad.config').joinpath('schemas', 'storage-config.schema.json').read_text())" \
  > storage-config.schema.json

Both options are needed: --entrypoint python skips the image's startup banner, which would otherwise end up in the output file, and MONAD_LOGGING_DIRECTORY points logging at a writable directory. BaseModel does not validate the storage configuration file against the schema at run time; the schema is an aid for writing the file.

To check a file from Python, load it with StorageConfig.from_yaml(), which raises a validation error if the file does not match the schema — for example, an unknown top-level key, or AWS_ACCESS_KEY_ID without AWS_SECRET_ACCESS_KEY:

from pathlib import Path
from monad.config import StorageConfig

StorageConfig.from_yaml(Path("storage-config.yaml"))

Column Selection

Control which columns are included in training using allowed_columns or disallowed_columns on any data source.

# Drop columns that add no signal
disallowed_columns: ["order_id", "internal_id"]

# Or explicitly select columns
allowed_columns: ["customer_id", "article_id", "t_dat", "price", "sales_channel_id"]

Date Column Formats

The date_column block specifies the event timestamp column and its format.

date_column:
  name: t_dat
  format: "%Y-%m-%d"

Joining Tables

Use joined_data_sources on event data sources to join attribute (dimension) tables.

# On the event data source
joined_data_sources:
  - name: articles           # Name of the attribute data source
    join_on:
      - [article_id, article_id]  # [event_column, attribute_column]

Multiple joins are supported:

joined_data_sources:
  - name: articles
    join_on:
      - [article_id, article_id]
  - name: stores
    join_on:
      - [store_id, store_id]