Skip to content

fix(request): stop Request.from_url mutating caller user_data and writing empty __crawlee#2067

Open
anxkhn wants to merge 1 commit into
apify:masterfrom
anxkhn:fix/request-from-url-user-data-mutation
Open

fix(request): stop Request.from_url mutating caller user_data and writing empty __crawlee#2067
anxkhn wants to merge 1 commit into
apify:masterfrom
anxkhn:fix/request-from-url-user-data-mutation

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Request.from_url mutates the caller's user_data dict in place, and it always
writes an empty __crawlee block even when no crawlee options are supplied. Both
were introduced by #1603.

  • It took a live reference to the caller's nested block
    (crawlee_data_dict = user_data_dict.get('__crawlee', {})), then mutated it in
    place (crawlee_data_dict['maxRetries' | 'enqueueStrategy'] = ...) and replaced
    the __crawlee key on the caller's dict with a CrawleeRequestData model.
    Reusing a user_data dict therefore leaks enqueueStrategy / maxRetries into
    the caller's nested dict, and passing the same dict object twice raises
    TypeError: 'CrawleeRequestData' object does not support item assignment on the
    second call (the key is no longer a plain dict). The framework's own
    enqueue_links dodges the crash by building a fresh outer dict per link
    ({**base_user_data}), but the shared nested __crawlee dict still accumulates
    enqueueStrategy across links.
  • The if crawlee_data: guard, meant to skip writing __crawlee when there is no
    crawlee data, is always true because an empty pydantic model is truthy
    (bool(CrawleeRequestData()) is True). So Request.from_url('https://example.com')
    writes user_data = {'__crawlee': {}} instead of leaving user_data empty.

Fix: copy both the outer user_data dict and the nested __crawlee block before
mutating them, and compare against CrawleeRequestData() so the write is skipped
when empty.

-        user_data_dict = kwargs.pop('user_data', {}) or {}
-        crawlee_data_dict = user_data_dict.get('__crawlee', {})
+        user_data_dict = dict(kwargs.pop('user_data', {}) or {})
+        crawlee_data_dict = dict(user_data_dict.get('__crawlee') or {})

         if max_retries is not None:
             crawlee_data_dict['maxRetries'] = max_retries
@@
         crawlee_data = CrawleeRequestData(**crawlee_data_dict)

-        if crawlee_data:
+        if crawlee_data != CrawleeRequestData():
             user_data_dict['__crawlee'] = crawlee_data

Issues

  • No existing GitHub issue. This is a self-identified correctness bug found while
    reviewing Request.from_url. Happy to open a tracking issue with the repro
    first if you'd prefer that.

Testing

  • Added tests/unit/test_request.py with four tests:
    • test_from_url_does_not_mutate_caller_user_data: the caller's dict and its
      nested __crawlee block are left untouched (same object, unchanged value).
    • test_from_url_can_reuse_same_user_data_dict: calling from_url twice with
      the same user_data object no longer raises TypeError.
    • test_from_url_omits_crawlee_data_when_empty: a bare from_url does not write
      __crawlee into user_data.
    • test_from_url_keeps_crawlee_data_when_supplied: the positive round-trip,
      max_retries / enqueue_strategy are still serialized into __crawlee and
      reflected on the request.
  • Verified fail-first: reverting only src/crawlee/_request.py (keeping the new
    tests) fails 3 of the 4 tests, each on its targeted defect; all 4 pass with the
    fix.
  • uv run pytest tests/unit/test_request.py -> 4 passed.
  • Regression: uv run pytest tests/unit/crawlers/_basic/test_basic_crawler.py tests/unit/storages/test_request_queue.py tests/unit/test_router.py and the
    enqueue_strategy / max_retries cases -> all pass.
  • uv run poe lint (ruff format + check) passes on the changed files.

Checklist

  • CI passed

…ting empty __crawlee

Request.from_url took a live reference to the caller's user_data dict and its
nested `__crawlee` block, then mutated them in place and replaced the
`__crawlee` key with a CrawleeRequestData model. As a result, reusing a
user_data dict leaked `enqueueStrategy`/`maxRetries` into the caller's nested
dict, and passing the same dict object twice raised
`TypeError: 'CrawleeRequestData' object does not support item assignment`
because the key was no longer a plain dict.

Copy both the outer user_data dict and the nested `__crawlee` block before
mutating them so the caller's input is left untouched.

Also, the `if crawlee_data:` guard meant to skip writing `__crawlee` when there
is no crawlee data was always true, because an empty pydantic model is truthy.
This wrote an empty `{'__crawlee': {}}` for requests with no crawlee options.
Compare against `CrawleeRequestData()` so the write is skipped when empty.

Add regression tests covering input mutation, reusing the same dict, and the
empty-guard case.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants