fix: harden metric scrape edge cases#2297
Open
zeitlinger wants to merge 3 commits into
Open
Conversation
Contributor
|
6cc0f6c to
9a7bf86
Compare
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
9a7bf86 to
0020301
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the metrics scrape path across core buffering, query parsing/filtering, and the standalone HTTPServer exporter to address multiple reported scrape-triggered crashes/DoS vectors and information disclosure.
Changes:
- Make
Bufferthread striping overflow-safe and bound spin-wait during collection (replaying buffered observations before failing). - Stop exposing stack traces to HTTP clients; return a generic 500 error body while logging server-side.
- Bound query parsing (length + parameter count) and de-duplicate metric-name filters for more predictable scrape cost.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| prometheus-metrics-model/src/main/java/io/prometheus/metrics/model/registry/MetricNameFilter.java | De-duplicates filter inputs and adds fast-path contains() checks. |
| prometheus-metrics-exporter-httpserver/src/test/java/io/prometheus/metrics/exporter/httpserver/HTTPServerTest.java | Updates tests for new default executor + generic error responses. |
| prometheus-metrics-exporter-httpserver/src/test/java/io/prometheus/metrics/exporter/httpserver/HttpExchangeAdapterTest.java | Verifies generic 500 body and absence of stack traces. |
| prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.java | Switches default executor to bounded queue and changes unauthorized-body handling. |
| prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HttpExchangeAdapter.java | Replaces stack-trace responses with a generic error message + server-side logging. |
| prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/BlockingRejectedExecutionHandler.java | Removes the blocking rejection handler to avoid dispatcher thread stalls. |
| prometheus-metrics-exporter-common/src/test/java/io/prometheus/metrics/exporter/common/PrometheusScrapeHandlerTest.java | Adds tests for rejecting overly-long / overly-many query parameters. |
| prometheus-metrics-exporter-common/src/main/java/io/prometheus/metrics/exporter/common/PrometheusScrapeHandler.java | Catches invalid query parameter parsing and returns HTTP 400. |
| prometheus-metrics-exporter-common/src/main/java/io/prometheus/metrics/exporter/common/PrometheusHttpRequest.java | Implements bounded query parsing for getParameterValues(). |
| prometheus-metrics-exporter-common/src/main/java/io/prometheus/metrics/exporter/common/InvalidQueryParameterException.java | Introduces an internal exception for invalid query parsing. |
| prometheus-metrics-core/src/test/java/io/prometheus/metrics/core/metrics/BufferTest.java | Adds regression tests for stripe indexing and timeout/replay behavior. |
| prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Buffer.java | Uses floorMod for striping; adds spin-wait timeout + replay-on-timeout. |
| prometheus-metrics-config/src/test/java/io/prometheus/metrics/config/UtilTest.java | Updates tests for redacted/escaped invalid-value error messages. |
| prometheus-metrics-config/src/test/java/io/prometheus/metrics/config/OpenMetrics2PropertiesTest.java | Updates expected messages to quote/escape invalid values. |
| prometheus-metrics-config/src/test/java/io/prometheus/metrics/config/ExporterPushgatewayPropertiesTest.java | Updates expected messages to quote/escape invalid values. |
| prometheus-metrics-config/src/test/java/io/prometheus/metrics/config/ExporterPropertiesTest.java | Updates expected messages to quote/escape invalid values. |
| prometheus-metrics-config/src/main/java/io/prometheus/metrics/config/Util.java | Centralizes invalid-value message formatting with escaping + truncation. |
| prometheus-metrics-config/src/main/java/io/prometheus/metrics/config/ExporterPushgatewayProperties.java | Uses Util.invalidValueMessage(...) to avoid leaking raw config values. |
| integration-tests/it-exporter/it-exporter-test/src/test/java/io/prometheus/metrics/it/exporter/test/HttpServerIT.java | Validates generic HTTPServer error body in integration tests. |
| integration-tests/it-exporter/it-exporter-test/src/test/java/io/prometheus/metrics/it/exporter/test/ExporterIT.java | Factors error-body assertions behind an overridable hook. |
| docs/apidiffs/current_vs_latest/prometheus-metrics-exporter-httpserver.txt | Records API-diff output for exporter-httpserver changes. |
| docs/apidiffs/current_vs_latest/prometheus-metrics-exporter-common.txt | Records API-diff output for exporter-common changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
79
to
82
| int idx = pair.indexOf("="); | ||
| if (idx != -1 && URLDecoder.decode(pair.substring(0, idx), "UTF-8").equals(name)) { | ||
| result.add(URLDecoder.decode(pair.substring(idx + 1), "UTF-8")); | ||
| } |
Comment on lines
+41
to
+43
| private static final int DEFAULT_MIN_THREADS = 1; | ||
| private static final int DEFAULT_MAX_THREADS = 10; | ||
| private static final int DEFAULT_QUEUE_SIZE = 100; |
Comment on lines
158
to
160
| } else { | ||
| drainInputAndClose(exchange); | ||
| exchange.getRequestBody().close(); | ||
| exchange.sendResponseHeaders(403, -1); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This addresses the bug reports from #2282, #2283, #2284, #2285, #2286, and #2287 in one PR.
Bufferstripe indexing so large thread IDs cannot produce negative stripe indexes.Bufferspin wait during collection and replay buffered observations before failing the scrape on timeout.PrometheusPropertiesExceptionmessages.Fixes #2282
Fixes #2283
Fixes #2284
Fixes #2285
Fixes #2286
Fixes #2287
Validation
mise run lint:fix./mvnw -pl prometheus-metrics-core,prometheus-metrics-config,prometheus-metrics-exporter-common,prometheus-metrics-exporter-httpserver,prometheus-metrics-model test -Dcoverage.skip=true -Dcheckstyle.skip=truemise run build