Troubleshooting

Use this guide to diagnose foundation-model training in the BaseModel Python App. Match the failing stage and error message to the relevant section below.

Before Training Starts

If an import or CLI command fails with LoggingSetupError, check whether the configured log directory is writable by the container user. If necessary, select another directory with MONAD_LOGGING_DIRECTORY before retrying; see Log Output Directory. The variable selects a location but does not change its permissions.

For input errors, check the paths specified in your configuration from inside the training container. For a database source, check the connection and access to the configured tables.

Repeated Runs

Before retrying, keep the error message and logs, and note the image version and failing stage. Before re-running pretrain in the same container, restart the training container to clear the previous Ray session. Check that the previous job has stopped and that no other job needs the container:

# Replace basemodel with the name of the training container.
docker restart basemodel

Ray manages background processing for training. Restarting the container clears processes and state left by the previous session, which can otherwise interfere with another run. It does not delete saved outputs on mounted volumes. Use this cleanup before another pretrain run; overwrite replaces output files and does not replace session cleanup.

After cleanup, choose the action appropriate to the run:

If the error persists after restart, use the error message to investigate the remaining cause. Correct configuration and path errors as well, and avoid deleting output directories while a job is using them.

Phase 1: Fitting Behavioral Representation

Configuration & Initialization

Log signature:

INFO - monad.run: Storing config in output directory...
INFO - monad.run: Processing n columns concurrently.
INFO - monad.run: Ray available resources {...}

What can go wrong:

• Invalid config schema — missing ID or date columns, duplicated column definitions, conflicting settings. Raises ValueError or ValidationError with a description of the problem.
  • Fix: correct the flagged entries in your YAML.
• Broken data source logic — non-unique source names, incorrect or cyclic joins, inconsistent type overrides. Raises ValueError pointing to the conflicting fields.
  • Fix: review joins and source names.
• Joining on two SQL Lambda columns — not supported. Results in database errors like "missing table" or "column does not exist".
  • Fix: precompute one of the join keys in the source data or create a SQL view.
• Invalid data parameters — reversed date ranges, overlapping test and training periods, negative window sizes. Raises ValueError.
  • Fix: correct the dates and split config.

Data Sampling

Log signature:

INFO - monad.fit.monad_analyzer: Creating connector for data_source abc
INFO - monad.fit.monad_analyzer: Retrieving data sample for abc
INFO - monad.fit.monad_analyzer: Data sample shape: (x, y)

What can go wrong:

• Out-of-memory on database side — the sample query loads too much data at once. You may see engine-specific errors like Resources exceeded during query execution.
  • Fix: reduce the number of features with allowed_columns / disallowed_columns, or increase database-side quotas with your platform administrator.
• Query timeouts — long-running queries exceed execution limits.
  • Fix: increase database-side retry limits or concurrency settings with your platform administrator.

Column Analysis

Log signature:

INFO - monad.fit.preprocessing: Analyzing abc type...
INFO - monad.fit.preprocessing: Column bcd, type categorical
INFO - monad.fit.preprocessing: Column cde, type decimal
INFO - monad.fit.preprocessing: Column def, type categoricalCompressed

What can go wrong:

• Misinterpreted column types — e.g., a numeric field inferred as categorical because it lacks a decimal part.
  • Fix: use column_type_overrides or correct the type in the source database.
• Suboptimal encoding — text or time-series columns defaulting to categorical or decimal. BaseModel may log a hint like Column 'price' appears to be a time series.
  • Fix: add the appropriate column_type_overrides entry. See Enrich & Transform.
• Text columns missing from features — text columns are skipped by default during fitting.
  • Fix: explicitly opt in with column_type_overrides: {description: text}. See Enrich & Transform → Column Type Overrides.

Computing Representations

Log signature:

INFO - monad.fit.features.tasks: Computing embeddings fusion
INFO - monad.fit.features.tasks: Running EMDE
INFO - monad.fit.features.calculators.cleora_calculator: Will train cleora for column abc
INFO - monad.utils.feature_stats: Saving feature stats

What can go wrong:

• Out-of-memory (local machine) — worker processes crash. Typical error: A worker died or was killed while executing a task by an unexpected system error.
  • Fix: review the automatic resource estimation report to check whether concurrency is within the recommended limit. Tune parallelism with num_concurrent_features and num_cpus in query_optimization (see Scaling & Memory). Allocate more memory to the Docker container if needed.
• Excessively long runtime — heavy feature computation, especially with many sketch-type columns.
  • Fix: remove redundant sketch features — e.g., if both product_id and product_name represent the same entity, keep only one. Reduce cardinality where possible — if product IDs encode size/color/volume, restructure so the ID represents the core product and pass variants as separate columns.
• Out-of-memory (database side) — same as during sampling.
  • Fix: set cleora_num_query_chunks in query_optimization to split the fit-stage Cleora (embedding) queries into smaller parts.

When the fit stage completes, a columns analysis report summarizes what was processed — column types, skipped columns, and action recommendations. See Run Training → Columns Analysis Report.

Redundancy Report

During the fit stage, BaseModel checks for redundant columns — pairs whose values form a bijection (one-to-one mapping), meaning they carry the same information. Redundant pairs are flagged in a warning log and in the columns analysis report:

WARNING - Detected 1 redundancy groups in provided columns.
  Columns groups of categorical type `[('product_code', 'product_id')]`
  provide the same information.

The columns analysis report also shows them under Action recommendations with a hint:

Redundant categorical columns: product_code, product_id Hint: These columns form a bijection (one-to-one mapping). Keeping both adds no new information. Remove the redundant column via disallowed_columns.

Redundant columns add computation time without contributing extra signal. BaseModel does not remove them automatically — you must update the config yourself. However, the suggested_config.yaml generated in the output path already includes one column from each bijection pair in disallowed_columns and adds detected time-series columns as sql_lambdas with matching column_type_overrides entries.

• Fix: review the flagged pairs to confirm they genuinely represent the same dimension. Check the suggested_config.yaml — bijection columns are already excluded and time-series candidates are already configured there. Verify the suggestions, then adopt the file or copy the relevant entries into your config. See also Select & Organize → Filtering Columns.

Phase 2: Training the Model

Loading Representations

Log signature:

INFO - monad.modalities.modality_artifact: Available datasets [names]
INFO - monad.modalities.modality_artifact: Data sources to load: [names]
INFO - monad.datasets.utils: Set fit columns [aliases] for data source abc.

What can go wrong:

• Corrupted or manually modified checkpoint files — leads to hard-to-debug loading failures.
  • Fix: do not modify checkpoint contents manually. Re-run the fit stage to produce a clean checkpoint.

Initializing the Trainer

Log signature:

INFO - monad.run: Training foundation model...
INFO - monad.core.fm.model_provider: Number of model parameters: n

Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True

You should also see a model summary with parameter counts and estimated size.

What can go wrong:

• Long delay before training begins — memory-heavy data preparation, especially with large Parquet sources or multi-GPU setups.
  • Fix: exclude low-signal columns with allowed_columns / disallowed_columns. Lower window_shuffling_buffer_size in data_params.
• Out-of-memory — too many columns or aggressive buffer settings.
  • Fix: lower window_shuffling_buffer_size in data_params. Exclude low-signal columns. Increase available memory for the container.

Training Loop

Log signature:

Train Epoch 0/2 Entities ━━━━━━━━━━╸━━━━━━━━━━  45%  4,500/10,000  0:03:12  0:03:55
Train Epoch 0/2 Entities ━━━━━━━━━━━━━━━━━━━━ 100% 10,000/10,000  0:07:08  0:00:00
Validation Entities      ━━━━━━━━━━━━━━━━━━━━ 100%  1,000/1,000   0:00:23  0:00:00
`Trainer.fit` stopped: `max_epochs=n` reached.
INFO - monad.run: Training foundation model finished.
INFO - monad.run: Pretraining finished.

The Entities lines appear only when show_entity_progress: true is set in training_params (the default is false). The Rich entity progress bar shows percentage complete, entities processed, elapsed time, and estimated time remaining. It supplements the standard PyTorch Lightning training output.

What can go wrong:

• GPU out-of-memory — model doesn't fit. Typical error: torch.OutOfMemoryError: CUDA out of memory.
  • Fix: check that the GPU isn't occupied by other processes (nvidia-smi or nvtop). If a previous run was interrupted with Ctrl+Z, the process is only suspended — use kill %1 or kill <PID> to free the memory. Otherwise, enable multi-GPU training via devices and strategy (see Scaling & Memory), reduce model size with hidden_dim or emde_quality in memory_constraining_params, or remove redundant features.
• Training is very slow — long pauses between batches or epochs.
  • Fix: enable multi-GPU with devices and strategy. Increase batch_size (in data_loader_params) if memory allows. Increase num_workers in data_loader_params.
• Database-side OOM during training — queries during the training loop exceed database memory.
  • Fix: increase num_workers to split queries into more parts. Set data_loading_num_query_chunks in query_optimization. Enable cache_path in data_params to avoid re-querying.