feat(train): add dry_run=True to train()#6027
Open
amazeAmazing wants to merge 2 commits into
Open
Conversation
164a14c to
7b06162
Compare
008794d to
16c783a
Compare
Add dry_run parameter to all trainers (SFT, DPO, RLVR, RLAIF). When dry_run=True: - All existing validation runs inline (IAM role, hyperparameters, recipe constraints, infrastructure availability) - Returns None without submitting a job or consuming compute - Raises with clear error message on validation failure Additionally, validate_data_path_exists() is called unconditionally (regardless of dry_run) before job submission to catch non-existent S3 paths or dataset ARNs early. Design follows nova-forge-sdk pattern: validation always runs as part of the normal code path, dry_run short-circuits before the actual TrainingJob.create API call. Changes: - data_utils.py: add validate_data_path_exists() utility (S3 + DataSet ARN) - base_trainer.py: add dry_run to abstract train(), _train_serverful_smtj(), and _train_hyperpod() - sft/dpo/rlvr/rlaif_trainer.py: add dry_run param, pass through to shared methods, short-circuit serverless path - Notebook examples added to SFT, DPO, RLVR, RLAIF notebooks - Unit tests added to existing test files - Integration test added
9f89498 to
f6bd43d
Compare
Add dry_run parameter to BaseEvaluator.evaluate() and all subclasses (BenchMarkEvaluator, CustomScorerEvaluator, LLMAsJudgeEvaluator). When dry_run=True: - All existing validation runs (IAM role, model resolution, recipe, pipeline rendering) - Dataset S3 path / DataSet ARN validated via validate_data_path_exists() - Returns None without submitting a pipeline execution - Raises on validation failure Dataset validation runs unconditionally (not just during dry_run) for CustomScorerEvaluator and LLMAsJudgeEvaluator which accept user datasets. Changes: - base_evaluator.py: add dry_run to evaluate() signature - benchmark_evaluator.py: add dry_run, short-circuit before _start_execution() - custom_scorer_evaluator.py: add dry_run, validate dataset, short-circuit - llm_as_judge_evaluator.py: add dry_run, validate dataset, short-circuit - Notebook examples added to benchmark, custom_scorer, llm_as_judge notebooks
f6bd43d to
8a3936b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
dry_run=Trueparameter to all trainers (SFT, DPO, RLVR, RLAIF) and evaluators (BenchMark, CustomScorer, LLMAsJudge). Whendry_run=True, all validation runs but no job is submitted and no compute is consumed.What it does
validate_data_path_exists()validates S3 URIs and DataSet ARNs unconditionally before submissiondry_run=Trueshort-circuits before the API call, returningNoneChanges
Commit 1:
feat(train): add dry_run=True to train()data_utils.py— newvalidate_data_path_exists()(S3 vialist_objects_v2, DataSet ARN viadescribe_hub_content)base_trainer.py— adddry_runtotrain(),_train_serverful_smtj(),_train_hyperpod()sft/dpo/rlvr/rlaif_trainer.py— adddry_run, validate data paths, short-circuittest_dry_run_integration.py)Commit 2:
feat(evaluate): add dry_run=True to evaluate()base_evaluator.py— adddry_runtoevaluate()signaturebenchmark/custom_scorer/llm_as_judge_evaluator.py— adddry_run, validate dataset paths (custom scorer + LLM-as-judge), short-circuit before_start_execution()Testing
result is None, verifies no job created via SageMaker APIIssue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.