Skip to content

feat(train): add dry_run=True to train()#6027

Open
amazeAmazing wants to merge 2 commits into
aws:master-nova-follow-upsfrom
amazeAmazing:feature/dry-run-validation
Open

feat(train): add dry_run=True to train()#6027
amazeAmazing wants to merge 2 commits into
aws:master-nova-follow-upsfrom
amazeAmazing:feature/dry-run-validation

Conversation

@amazeAmazing

@amazeAmazing amazeAmazing commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Add dry_run=True parameter to all trainers (SFT, DPO, RLVR, RLAIF) and evaluators (BenchMark, CustomScorer, LLMAsJudge). When dry_run=True, all validation runs but no job is submitted and no compute is consumed.

What it does

  • All existing inline validation runs as normal (IAM role, hyperparameters, recipe constraints, model resolution)
  • validate_data_path_exists() validates S3 URIs and DataSet ARNs unconditionally before submission
  • dry_run=True short-circuits before the API call, returning None
  • Raises with a clear error message on validation failure

Changes

Commit 1: feat(train): add dry_run=True to train()

  • data_utils.py — new validate_data_path_exists() (S3 via list_objects_v2, DataSet ARN via describe_hub_content)
  • base_trainer.py — add dry_run to train(), _train_serverful_smtj(), _train_hyperpod()
  • sft/dpo/rlvr/rlaif_trainer.py — add dry_run, validate data paths, short-circuit
  • Trainer notebook examples (SFT, DPO, RLVR, RLAIF)
  • Unit tests in existing test files
  • Integration test (test_dry_run_integration.py)

Commit 2: feat(evaluate): add dry_run=True to evaluate()

  • base_evaluator.py — add dry_run to evaluate() signature
  • benchmark/custom_scorer/llm_as_judge_evaluator.py — add dry_run, validate dataset paths (custom scorer + LLM-as-judge), short-circuit before _start_execution()
  • Evaluator notebook examples (benchmark, custom_scorer, llm_as_judge)

Testing

  • Integration test uploads sample data to default bucket, validates S3 path failures, confirms result is None, verifies no job created via SageMaker API

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@amazeAmazing amazeAmazing force-pushed the feature/dry-run-validation branch 2 times, most recently from 164a14c to 7b06162 Compare July 15, 2026 17:55
@amazeAmazing amazeAmazing reopened this Jul 15, 2026
@amazeAmazing amazeAmazing force-pushed the feature/dry-run-validation branch 9 times, most recently from 008794d to 16c783a Compare July 15, 2026 22:18
@amazeAmazing amazeAmazing changed the title feat(train): add dry_run=True to train() and deploy() feat(train): add dry_run=True to train() Jul 15, 2026
@amazeAmazing amazeAmazing marked this pull request as ready for review July 15, 2026 22:30
Add dry_run parameter to all trainers (SFT, DPO, RLVR, RLAIF).

When dry_run=True:
- All existing validation runs inline (IAM role, hyperparameters,
  recipe constraints, infrastructure availability)
- Returns None without submitting a job or consuming compute
- Raises with clear error message on validation failure

Additionally, validate_data_path_exists() is called unconditionally
(regardless of dry_run) before job submission to catch non-existent
S3 paths or dataset ARNs early.

Design follows nova-forge-sdk pattern: validation always runs as part
of the normal code path, dry_run short-circuits before the actual
TrainingJob.create API call.

Changes:
- data_utils.py: add validate_data_path_exists() utility (S3 + DataSet ARN)
- base_trainer.py: add dry_run to abstract train(), _train_serverful_smtj(),
  and _train_hyperpod()
- sft/dpo/rlvr/rlaif_trainer.py: add dry_run param, pass through to
  shared methods, short-circuit serverless path
- Notebook examples added to SFT, DPO, RLVR, RLAIF notebooks
- Unit tests added to existing test files
- Integration test added
@amazeAmazing amazeAmazing force-pushed the feature/dry-run-validation branch 5 times, most recently from 9f89498 to f6bd43d Compare July 15, 2026 23:35
Add dry_run parameter to BaseEvaluator.evaluate() and all subclasses
(BenchMarkEvaluator, CustomScorerEvaluator, LLMAsJudgeEvaluator).

When dry_run=True:
- All existing validation runs (IAM role, model resolution, recipe,
  pipeline rendering)
- Dataset S3 path / DataSet ARN validated via validate_data_path_exists()
- Returns None without submitting a pipeline execution
- Raises on validation failure

Dataset validation runs unconditionally (not just during dry_run) for
CustomScorerEvaluator and LLMAsJudgeEvaluator which accept user datasets.

Changes:
- base_evaluator.py: add dry_run to evaluate() signature
- benchmark_evaluator.py: add dry_run, short-circuit before _start_execution()
- custom_scorer_evaluator.py: add dry_run, validate dataset, short-circuit
- llm_as_judge_evaluator.py: add dry_run, validate dataset, short-circuit
- Notebook examples added to benchmark, custom_scorer, llm_as_judge notebooks
@amazeAmazing amazeAmazing force-pushed the feature/dry-run-validation branch from f6bd43d to 8a3936b Compare July 15, 2026 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant