Skip to content

[python] Introduce 'nested_update' aggregator function#8577

Merged
JingsongLi merged 9 commits into
apache:masterfrom
PyRSA:feature/pypaimon-nested-update-impl
Jul 18, 2026
Merged

[python] Introduce 'nested_update' aggregator function#8577
JingsongLi merged 9 commits into
apache:masterfrom
PyRSA:feature/pypaimon-nested-update-impl

Conversation

@PyRSA

@PyRSA PyRSA commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

Introduce the nested_update field aggregator for pypaimon.

This change adds support for the nested_update aggregation function in the Python reader, aligning its behavior with the Java implementation. It supports nested-key based updates, optional sequence-field based conflict resolution, configurable nested-key null handling, and count limiting.

Dependency

This PR depends on #8561 and reuses several shared utility methods introduced there. Please review and merge #8561 first.

Tests

  • python -m pytest pypaimon/tests/test_field_aggregators.py::FieldNestedUpdateAggTest -q
  • git diff --check

@JingsongLi

Copy link
Copy Markdown
Contributor
  • RowType.get_field_index() returns DataField.id, but the ProjectedRow of nested key/sequence uses it as an array index.
  • The actual nested field ID is a global ID and does not guarantee continuity from 0.
  • Reproducing locally using field IDs 10, 20: Both generate mapping=[10], then directly throw an IndexError: Position 10 is out of bounds for row arity 2.
  • The existing tests use 0, 1, 2, which obscures the issue. The ordinal field should be returned, and non-sequential ID tests should be added.

@PyRSA

PyRSA commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author
  • RowType.get_field_index() returns DataField.id, but the ProjectedRow of nested key/sequence uses it as an array index.
  • The actual nested field ID is a global ID and does not guarantee continuity from 0.
  • Reproducing locally using field IDs 10, 20: Both generate mapping=[10], then directly throw an IndexError: Position 10 is out of bounds for row arity 2.
  • The existing tests use 0, 1, 2, which obscures the issue. The ordinal field should be returned, and non-sequential ID tests should be added.

Good catch, thanks! I've fixed RowType.get_field_index() to return the ordinal field position instead of DataField.id, and added a regression test using non-sequential field IDs to cover this case.

@JingsongLi

Copy link
Copy Markdown
Contributor

nested_update assumes that the elements of ARRAY<ROW> are InternalRow; however, the PyArrow→Polars reader actually outputs a list[dict], and calling it directly results in an AttributeError: ‘dict’ object has no attribute ‘get_field’.

FashGirl added 2 commits July 15, 2026 20:11
…impl

# Conflicts:
#	paimon-python/pypaimon/common/options/core_options.py
#	paimon-python/pypaimon/read/reader/aggregate/aggregators.py
#	paimon-python/pypaimon/tests/test_field_aggregators.py
@PyRSA

PyRSA commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

nested_update assumes that the elements of ARRAY<ROW> are InternalRow; however, the PyArrow→Polars reader actually outputs a list[dict], and calling it directly results in an AttributeError: ‘dict’ object has no attribute ‘get_field’.

Thanks for pointing this out.
I updated the implementation to support both InternalRow and dict representations of nested rows.

Instead of accessing fields directly through InternalRow.get_field(), the aggregator now uses a common FieldProjection helper to extract configured fields from either row representation. This keeps the aggregation logic unchanged while making it compatible with the PyArrow → Polars read path.

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi

Copy link
Copy Markdown
Contributor

Please resolve conflicts.

@PyRSA

PyRSA commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Please resolve conflicts.

Resolved the merge conflicts and updated the branch. Thanks!

@JingsongLi
JingsongLi merged commit 7234e4c into apache:master Jul 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants