fix: Raincloud oracle gains repeated/list column support#267
Merged
Conversation
RaincloudConformanceIntegrationTest's Parquet oracle previously aborted on any column with maxRepetitionLevel > 0 (LIST/repeated groups), which is how Vortex FixedSizeList/List columns round-trip through Parquet — blocking conformance verification for glove-* and osm-germany-relations. Rewrite the oracle to iterate top-level schema fields (one CSV cell per Vortex column, matching CsvExporter's model) instead of leaf columns (which for a nested field would wrongly emit multiple cells for what CsvExporter renders as one JSON array/object). Primitive top-level fields keep the existing physical-type rendering; LIST/STRUCT fields render via hardwood's row-object API (RowReader.getList/getStruct, PqList, PqStruct) walked alongside the SchemaNode tree so unsigned INT32/INT64 detection (#253) still applies at any nesting depth, mirroring CsvExporter's jsonObject/jsonArray/jsonValue rules exactly. Verifying against real hydrated data (glove-6b-50d/100d/200d, osm-germany-relations) surfaced two genuine, separate bugs the previously-aborting oracle had never let this test path reach: - CsvExporter.offsetAt only handled I32/I64 list offsets; osm's members/tags columns use I16 offsets, throwing VortexException. offsets can legally be any integer ptype (the encoder picks the narrowest that fits, same as VarBinArray's offsets) (#263). - ScanIterator.sliceArray had no case for ListArray/FixedSizeListArray, so scanning such a column failed whenever its covering flat chunk spans several scan windows (misaligned per-column chunk grids) — a real reading gap, not only an export-side one (#265). Both fixed; all 4 slugs now conform end-to-end and flip to `ok` in the matrix. Full ./mvnw verify (integration + unit, 304 tests) green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RaincloudConformanceIntegrationTest) previously aborted on anyLIST/repeated Parquet column, blocking verification forglove-6b-50d/100d/200d(FixedSizeList[F32,n]) andosm-germany-relations(List[Struct]).RowReader.getList/getStruct,PqList,PqStruct), mirroringCsvExporter'sjsonObject/jsonArray/jsonValuerules exactly — including unsigned INT32/INT64 detection (covid-world-vaccination-progress: i32 column decoded as u32 (sign bit lost) #253) at any nesting depth.CsvExporter.offsetAtonly handled I32/I64 list offsets;osm-germany-relations'members/tagscolumns use I16 offsets and threwVortexException. Offsets can legally be any integer ptype, same asVarBinArray.ScanIterator.sliceArrayhad no case forListArray/FixedSizeListArray, so scanning such a column failed whenever its covering flat chunk spans several scan windows (misaligned per-column chunk grids) — a real reading gap, not only an export-side one.glove-6b-50d/100d/200dandosm-germany-relationsfromgap:261tookin the conformance matrix.docs/compatibility.mdtriage counts (117 ok / 0 gaps / 130 untriaged) and CHANGELOG.Test plan
./mvnw verify -pl integration -am -Dit.test=RaincloudConformanceIntegrationTestwith all 4 slugs hydrated (up to 700MB cap) and flipped tooklocally — 5/5 pass, 0 skipped./mvnw test(all modules) — green./mvnw verify -pl integration -am(full IT suite, 304 tests) — green🤖 Generated with Claude Code