Skip to content

feat: Add third-party evaluator CLI integration (DeepEval + Autoevals)#1779

Open
stone-coding wants to merge 4 commits into
aws:mainfrom
stone-coding:deepeval-cli-integration
Open

feat: Add third-party evaluator CLI integration (DeepEval + Autoevals)#1779
stone-coding wants to merge 4 commits into
aws:mainfrom
stone-coding:deepeval-cli-integration

Conversation

@stone-coding

@stone-coding stone-coding commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

  • Add --param key=value (repeatable) and --parameters-file for metric constructor kwargs
  • Registry-driven architecture: adding a new library = one config entry + one template directory
  • Library-specific defaults: DeepEval (300s/1024MB), Autoevals (60s/512MB)
  • Per-metric warnings for metrics requiring retrieval_context or expected_output
  • Templates generate: lambda_function.py + pyproject.toml + execution-role-policy.json
  • E2E lifecycle test: create → add 3P evaluator → deploy → invoke → run eval

Test plan

  • 62 unit tests passing (npx vitest run src/cli/primitives/__tests__/EvaluatorPrimitive.test.ts)
  • E2E test: e2e-tests/third-party-eval-lifecycle.test.ts (DeepEval + Autoevals full lifecycle)
  • Manual E2E: 4/4 evaluators deployed and scored successfully against live AgentCore service

Customer experience

# Before: ~40 lines of handler code + SAM/CDK template + manual deployment
# After: 2 commands
agentcore add evaluator --3p-library deepeval --metric FaithfulnessMetric --param threshold=0.7 --name faithfulness --level TRACE
agentcore deploy --yes

@haomiao037
Add deepeval CLI integration: flags, templates, warnings, memory, uni…
6fba5c8 
…t tests
@haomiao037
Refactor CLI to registry-driven third-party evaluator framework (Deep…
d44430a 
…Eval + Autoevals)
@haomiao037
fix: Update 3P evaluator templates and add --param/--parameters-file …
485000e 
…support

- DeepEval template: add /tmp workaround + DEEPEVAL_TELEMETRY_OPT_OUT for Lambda
- Autoevals template: fix scorer= → metric= to match SDK API
- Rename --from-3p-library → --3p-library
- Add --param key=value (repeatable) for metric constructor kwargs
- Add --parameters-file for JSON file of kwargs
- Add parseParamFlags() utility + unit tests
- --param and --parameters-file are mutually exclusive
@haomiao037
feat: Add 3P evaluator E2E test + fix autoevals template
0bff7a8 
- Add e2e-tests/third-party-eval-lifecycle.test.ts: full lifecycle test for
  DeepEval + Autoevals 3P evaluators (create → add → deploy → invoke → run eval)
- Fix autoevals template: params go to AutoevalsAdapter (not metric constructor)
- Add openai>=1.0.0 to autoevals pyproject.toml (required by LLM-based scorers)
@stone-coding
stone-coding requested a review from a team July 16, 2026 23:17
@github-actions github-actions Bot added the size/xl PR size: XL label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

size/xl PR size: XL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stone-coding @haomiao037