fix(request): stop Request.from_url mutating caller user_data and writing empty __crawlee#2067
Open
anxkhn wants to merge 1 commit into
Open
fix(request): stop Request.from_url mutating caller user_data and writing empty __crawlee#2067anxkhn wants to merge 1 commit into
anxkhn wants to merge 1 commit into
Conversation
…ting empty __crawlee
Request.from_url took a live reference to the caller's user_data dict and its
nested `__crawlee` block, then mutated them in place and replaced the
`__crawlee` key with a CrawleeRequestData model. As a result, reusing a
user_data dict leaked `enqueueStrategy`/`maxRetries` into the caller's nested
dict, and passing the same dict object twice raised
`TypeError: 'CrawleeRequestData' object does not support item assignment`
because the key was no longer a plain dict.
Copy both the outer user_data dict and the nested `__crawlee` block before
mutating them so the caller's input is left untouched.
Also, the `if crawlee_data:` guard meant to skip writing `__crawlee` when there
is no crawlee data was always true, because an empty pydantic model is truthy.
This wrote an empty `{'__crawlee': {}}` for requests with no crawlee options.
Compare against `CrawleeRequestData()` so the write is skipped when empty.
Add regression tests covering input mutation, reusing the same dict, and the
empty-guard case.
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Request.from_urlmutates the caller'suser_datadict in place, and it alwayswrites an empty
__crawleeblock even when no crawlee options are supplied. Bothwere introduced by #1603.
(
crawlee_data_dict = user_data_dict.get('__crawlee', {})), then mutated it inplace (
crawlee_data_dict['maxRetries' | 'enqueueStrategy'] = ...) and replacedthe
__crawleekey on the caller's dict with aCrawleeRequestDatamodel.Reusing a
user_datadict therefore leaksenqueueStrategy/maxRetriesintothe caller's nested dict, and passing the same dict object twice raises
TypeError: 'CrawleeRequestData' object does not support item assignmenton thesecond call (the key is no longer a plain dict). The framework's own
enqueue_linksdodges the crash by building a fresh outer dict per link(
{**base_user_data}), but the shared nested__crawleedict still accumulatesenqueueStrategyacross links.if crawlee_data:guard, meant to skip writing__crawleewhen there is nocrawlee data, is always true because an empty pydantic model is truthy
(
bool(CrawleeRequestData()) is True). SoRequest.from_url('https://example.com')writes
user_data = {'__crawlee': {}}instead of leavinguser_dataempty.Fix: copy both the outer
user_datadict and the nested__crawleeblock beforemutating them, and compare against
CrawleeRequestData()so the write is skippedwhen empty.
Issues
reviewing
Request.from_url. Happy to open a tracking issue with the reprofirst if you'd prefer that.
Testing
tests/unit/test_request.pywith four tests:test_from_url_does_not_mutate_caller_user_data: the caller's dict and itsnested
__crawleeblock are left untouched (same object, unchanged value).test_from_url_can_reuse_same_user_data_dict: callingfrom_urltwice withthe same
user_dataobject no longer raisesTypeError.test_from_url_omits_crawlee_data_when_empty: a barefrom_urldoes not write__crawleeintouser_data.test_from_url_keeps_crawlee_data_when_supplied: the positive round-trip,max_retries/enqueue_strategyare still serialized into__crawleeandreflected on the request.
src/crawlee/_request.py(keeping the newtests) fails 3 of the 4 tests, each on its targeted defect; all 4 pass with the
fix.
uv run pytest tests/unit/test_request.py-> 4 passed.uv run pytest tests/unit/crawlers/_basic/test_basic_crawler.py tests/unit/storages/test_request_queue.py tests/unit/test_router.pyand theenqueue_strategy/max_retriescases -> all pass.uv run poe lint(ruff format + check) passes on the changed files.Checklist