Validation & Best Practices

This page covers how to validate your target function before training, common mistakes to avoid, and performance tips.

Validating with verify_target()

Before launching a full training run, use verify_target() to check that your target function executes without errors and returns the values you expect. It runs the function against a sample of entities from your foundation model data and reports problems early:

• Executes target_fn on a sampled subset of entities
• Checks return type and shape match the task
• Tracks None rate and fails if it exceeds the allowed threshold
• Surfaces exceptions with a stack trace and sample entity context

Basic Usage

from monad.ui.module import BinaryClassificationTask
from monad.ui.target_function import verify_target

verify_target(
    target_fn=my_target_fn,
    fm_checkpoint_path="./foundation_model",
    task=BinaryClassificationTask(),
)

If validation passes, verify_target() returns a NumPy array containing all non-None targets produced by the checked sample. Its first dimension is the number of accepted entities, not one example. Inspect result.shape or aggregate counts before printing it when labels are sensitive. If something is wrong, it raises one of:

For the full parameter reference, see verify_target().

Full Example

from typing import Dict
from datetime import timedelta

import numpy as np

from monad.batch import SPLIT_TIMESTAMP
from monad.ui.module import RegressionTask
from monad.ui.target_function import (
    Attributes, Events, verify_target,
    has_incomplete_training_window,
)

def ltv_target(
    history: Events, future: Events, entity: Attributes, ctx: Dict
) -> np.ndarray:
    if history["transactions"].count() < 2:
        return None
    if has_incomplete_training_window(ctx, timedelta(days=30)):
        return None

    future_30d = future.interval_from(ctx[SPLIT_TIMESTAMP], timedelta(days=30))
    return np.array([future_30d["transactions"].sum("price")], dtype=np.float32)

result = verify_target(
    target_fn=ltv_target,
    fm_checkpoint_path="./foundation_model",
    task=RegressionTask(num_targets=1),
    num_percentage_entities=5,        # check 5% of entities
    percentage_nones_allowed=70,      # allow up to 70% None
    log_every_n_steps=100,            # progress logging
)

print("Accepted target array shape:", result.shape)

Debugging Failures

• RuntimeError
  Wrap risky sections in try/except, print counts, guard empty data sources with if txns.count() == 0
• TypeError
  Print type(result), result.shape, and result.dtype — verify they match the task
• ValueError
  Your function returns too many None values. Log your eligibility conditions, temporarily relax filters, use log_every_n_steps to see which entities are skipped

Validation Tips

• Start small.
  Leave num_percentage_entities at 1 (the default) for initial debugging. Increase it once the function runs cleanly to get a better picture of your None rate.

• Watch your None rate.
  A high None rate isn't necessarily wrong — cold-start filtering and incomplete windows are legitimate reasons to exclude entities. But if the rate is unexpectedly high, it often signals a bug in your filtering logic. Adjust percentage_nones_allowed to match your expectation.

• Reuse checkpoint data parameters when possible.
  Validation loads the stored DataParams, including extra_columns, from the foundation model checkpoint. Supply data_params_overrides only to change those settings or support a legacy checkpoint that lacks them.

• Enable progress logging.
  Set log_every_n_steps (e.g., 100) to print a status line every N entities — useful for spotting where failures cluster or confirming the run is progressing.

• Use limit for large datasets.
  If your foundation model covers millions of entities, set limit to cap the number of evaluations and keep iteration fast.

Common Mistakes

• Forgetting to check for empty data
  On an empty DataSourceEvents, sum() returns 0.0 and mean() returns NaN; min() and max() raise ValueError. Guard mean(), min(), max() and any division with if txns.count() == 0. For churn, count or spend targets an empty future is a valid label (churn = 1, spend = 0), so returning None there drops exactly those entities from training. Return None only when there is no label without future events (for example, which brand or item comes next).

• Missing .events or .attribute
txns["price"] returns a ModalityEvents object, not an array. Use txns["price"].events to get the np.ndarray. Similarly, attributes["customers"]["age"] returns a ModalityAttribute; use .attribute to get the value.

• Leaking future information into eligibility checks
  Use history (or attributes) to decide whether an entity qualifies. Use future only to compute the label. If you filter entities based on future data, the model receives a signal it cannot reproduce at inference time.

  Wrong — filtering on future

  future_30d = future.interval_from(ctx[SPLIT_TIMESTAMP], timedelta(days=30))
  if future_30d["transactions"].count() == 0:
      return None  # removes all churners from training!
  

  Correct — filtering on history, labeling from future

  # eligibility: entity must have history
  if history["transactions"].count() == 0:
      return None
  
  # ensure a complete target window
  if has_incomplete_training_window(ctx, timedelta(days=30)):
      return None
  
  # label: use future only to compute the target
  future_30d = future.interval_from(ctx[SPLIT_TIMESTAMP], timedelta(days=30))
  return np.array([1 if future_30d["transactions"].count() == 0 else 0],
                  dtype=np.float32)
  

• Skipping has_incomplete_training_window
  Without this check, split points near the end of your data produce windows shorter than intended, creating inconsistent labels.

• Forgetting to unpack .groupBy
  All 'groupBy operations return (values, names). Always unpack: counts, names = txns.groupBy("category").count().

• Wrong timedelta direction
  Positive timedelta moves forward (into future), negative moves backward (into history). A common error is using a positive timedelta on history, which selects nothing.

Performance Tips

Rare target cohorts

For a rare eligible cohort that is fixed for the whole run and selected without future target-window outcomes, TrainingParams.entity_ids can restrict scenario training before target materialization. Monad 1.14 has no automatic, target-aware prefilter that can infer which target_fn calls will return None. Pass entity_ids=EntityIds(subquery=..., matching=True) to TrainingParams; see the EntityIds reference.

Do not apply a static ID list when eligibility changes at each sampled split point: that changes the training population and can leak later history. In that case, return None from target_fn after the cheapest eligibility checks.

Put the cheapest history-only eligibility checks first, before joined-column reads or future-window aggregations. Scenario training reuses data_loader_params.num_workers from the foundation checkpoint, so tune that setting for loading throughput. Use limit_train_batches and limit_val_batches to bound a trial run. These controls reduce per-entity work, improve throughput, or cap the run; they do not improve the eligible-entity ratio.

predictions_to_include_fn and predictions_to_exclude_fn restrict prediction classes or items for supported tasks. They do not filter training entities.

• Use built-in aggregations
  txns.sum("price") is faster than manual loops over .events.
• Exit early
  Return None as soon as you know an entity is invalid.
• Cache data source access
  Assign txns = future["transactions"] once and reuse.

• Avoid manual loops
  Use the built-in aggregation and filtering methods rather than iterating over .events arrays yourself.

  Python

  # ❌ Slow
  total = sum(future["transactions"]["price"].events)
  
  # ✅ Fast
  txns = future["transactions"]
  total = txns.sum("price")
  

Putting It Together

Once your target function is validated, wire it into a training run:

# --- Imports ---
from pathlib import Path
from datetime import timedelta
import numpy as np
from monad.batch import SPLIT_TIMESTAMP
from monad.config import TrainingParams
from monad.ui.module import load_from_foundation_model, BinaryClassificationTask
from monad.ui.target_function import Events, Attributes, has_incomplete_training_window

# --- Task & target function ---
task = BinaryClassificationTask()

def churn_target(history: Events, future: Events, entity: Attributes, ctx: dict):
    ...
    return np.array([1 if future["transactions"].count() == 0 else 0],
                    dtype=np.float32)

# --- Load from foundation model ---
module = load_from_foundation_model(
    checkpoint_path="./foundation_model",
    downstream_task=task,
    target_fn=churn_target,
)

# --- Execute training ---
training_params = TrainingParams(
    checkpoint_dir=Path("./churn_model"),
    ...
)

module.fit(training_params=training_params)

Once task and target are defined, move on to Model Configuration to set up and run training.