fix(tracing): capture span body exceptions and export SGP status=ERROR#460
Merged
Merged
Conversation
Contributor
|
can we target next instead of main here and also fix the diff to pass lint? |
a2fa8d5 to
76af59f
Compare
declan-scale
approved these changes
Jul 10, 2026
Spans exported to SGP always showed status=SUCCESS even when the operation they represent failed. The SGP span defaults status to "SUCCESS" and only flips to "ERROR" inside its own __exit__ context manager, but the agentex processor builds SGP spans via create_span(...) and flushes them directly, so __exit__ never runs. The Trace.span()/AsyncTrace.span() context managers also ended spans in a bare finally, so a body exception was never recorded. Capture the exception in both context managers and carry it on the span so the SGP processor can map it: - add span_error.py with set_span_error/get_span_error, storing the failure under the reserved span.data["__error__"] key (the Span model is generated from the OpenAPI spec and has no status/error field; data is a real field that survives model_copy(deep=True) and round-trips to both stores) - Trace.span()/AsyncTrace.span(): except -> set_span_error(span, exc); raise - _build_sgp_span(): when an error is present, set sgp_span.status = "ERROR" plus error/error.type/error.message metadata (matching SGP's native __exit__ shape) Exceptions still propagate; asyncio.CancelledError/KeyboardInterrupt are not flagged (control flow, not failures). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
76af59f to
3f6fb40
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Spans exported to SGP always show
status=SUCCESS, even when the operation they represent failed.Root cause, in three parts:
scale_gp_beta) defaultsstatus="SUCCESS"and only flips to"ERROR"inside its own__exit__context manager. The agentex processor builds SGP spans viacreate_span(...)and flushes them directly, so__exit__never runs.Spantype (generated from the OpenAPI spec) has nostatus/errorfield.Trace.span()/AsyncTrace.span()ended spans in a barefinally, so a body exception was never recorded before the span was flushed.Net effect: a failing operation is indistinguishable from a successful one in SGP.
Fix (SDK-only, no schema/backend change)
Capture → carry → map:
except Exception as exc: set_span_error(span, exc); raise.span_error.pystores the failure under the reservedspan.data["__error__"]key, following the existing__span_type__/__source__convention.datais a real field, so it survivesmodel_copy(deep=True)and round-trips to both the SGP and agentex-native stores. (A first-classstatus/errorfield would require the OpenAPI → Stainless → backend → migration chain; this achieves the same SGP outcome without it.)_build_sgp_span()setssgp_span.status = "ERROR"+error/error.type/error.messagemetadata when an error is present, matching SGP's native__exit__shape.Exceptions still propagate unchanged.
asyncio.CancelledError/KeyboardInterruptare intentionally not flagged (control flow, not failures) — broaden theexcepttoBaseExceptionif that changes.Tests
tests/lib/core/tracing/test_span_error.py:set_span_error/get_span_errorhelpers (incl. list-shapeddatano-op)._build_sgp_spanmaps error →status=ERROR+ metadata; no-error path staysSUCCESS.ruff checkclean; new + existing tracing tests pass (40 passed).🤖 Generated with Claude Code
Greptile Summary
This PR fixes a bug where all spans exported to SGP always showed
status=SUCCESS, even when the operation they represented failed. The root cause was that the agentex processor builds SGP spans viacreate_spanand flushes them directly, bypassing the__exit__context manager that normally flips status to ERROR.span_error.pymodule — introducesset_span_error/get_span_errorhelpers that store exception info under the reservedspan.data["__error__"]key, following the existing__span_type__/__source__convention. HandlesNone,dict, andlist-shapeddatacorrectly.trace.py— bothTrace.span()andAsyncTrace.span()now catchException, record it on the span viaset_span_error, then re-raise before thefinallyblock flushes the span;asyncio.CancelledError/KeyboardInterruptare intentionally excluded.sgp_tracing_processor.py—_build_sgp_spancallsget_span_errorafter building the span and invokessgp_span.set_error(error_type, error_message)to setstatus="ERROR"on the SGP side, matching SGP's native__exit__shape.Confidence Score: 5/5
Safe to merge. The change only touches the tracing layer, exceptions always re-propagate unchanged, and the success path is unaffected.
The fix is well-scoped: error capture happens before the span is flushed, the error key survives model_copy(deep=True) and recursive_model_dump, and the SGP status mapping is exercised by both the new unit tests and the existing tracing test suite (40 pass). No mutation of span identity fields, no change to public API surfaces, and no risk to callers that don't catch exceptions themselves.
No files require special attention.
Important Files Changed
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant UC as User Code participant CM as Trace.span() / AsyncTrace.span() participant SE as span_error.py participant ES as end_span() participant SGP as _build_sgp_span() UC->>CM: enter context manager CM->>CM: start_span() CM-->>UC: yield span alt Exception raised in body UC-->>CM: raise Exception CM->>SE: set_span_error(span, exc) Note over SE: span.data["__error__"] = {type, message} CM->>CM: re-raise CM->>ES: finally: end_span(span) ES->>SGP: on_span_end(span) SGP->>SE: get_span_error(span) SE-->>SGP: "{type, message}" SGP->>SGP: sgp_span.set_error(error_type, error_message) Note over SGP: sgp_span.status = ERROR SGP->>SGP: sgp_span.flush() else Success UC-->>CM: normal return CM->>ES: finally: end_span(span) ES->>SGP: on_span_end(span) SGP->>SE: get_span_error(span) returns None Note over SGP: sgp_span.status stays SUCCESS SGP->>SGP: sgp_span.flush() end%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant UC as User Code participant CM as Trace.span() / AsyncTrace.span() participant SE as span_error.py participant ES as end_span() participant SGP as _build_sgp_span() UC->>CM: enter context manager CM->>CM: start_span() CM-->>UC: yield span alt Exception raised in body UC-->>CM: raise Exception CM->>SE: set_span_error(span, exc) Note over SE: span.data["__error__"] = {type, message} CM->>CM: re-raise CM->>ES: finally: end_span(span) ES->>SGP: on_span_end(span) SGP->>SE: get_span_error(span) SE-->>SGP: "{type, message}" SGP->>SGP: sgp_span.set_error(error_type, error_message) Note over SGP: sgp_span.status = ERROR SGP->>SGP: sgp_span.flush() else Success UC-->>CM: normal return CM->>ES: finally: end_span(span) ES->>SGP: on_span_end(span) SGP->>SE: get_span_error(span) returns None Note over SGP: sgp_span.status stays SUCCESS SGP->>SGP: sgp_span.flush() endReviews (3): Last reviewed commit: "fix(tracing): capture span body exceptio..." | Re-trigger Greptile