[Iceberg] Make timestamptz return new Timestamp.MICROS logical type by ahmedabu98 · Pull Request #39344 · apache/beam

ahmedabu98 · 2026-07-15T18:04:49Z

The new Timestamp logical type was added in #36705. This PR integrates it in IcebergIO and makes it the return type when reading timestamptz Iceberg columns. It's a breaking change, but we gate it with upgradeCompatibilityVersion pipeline option

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

ahmedabu98 · 2026-07-15T18:05:12Z

R: @claudevdm can you take a look?

gemini-code-assist · 2026-07-15T18:05:25Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request updates the IcebergIO connector to improve timestamp precision when reading timestamptz columns. By migrating from the legacy Joda-based DATETIME type to the newer Timestamp.MICROS logical type, the connector now preserves microsecond-level data that was previously truncated. To ensure stability for existing pipelines, this change is gated behind the updateCompatibilityVersion pipeline option, allowing users to opt-out of the change by specifying an older version.

Highlights

Logical Type Update: Updated IcebergIO to use the new Timestamp.MICROS logical type for timestamptz columns, replacing the legacy DATETIME type to preserve microsecond precision.
Compatibility Gating: Implemented a version-gated mechanism using the updateCompatibilityVersion pipeline option, allowing users to retain the legacy behavior if needed.
Test Coverage: Added comprehensive tests to verify the new timestamp mapping and ensure that legacy compatibility settings correctly trigger the old behavior.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2026-07-15T18:07:03Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

gemini-code-assist

Code Review

This pull request updates IcebergIO to read timestamptz columns as the Timestamp.MICROS Beam logical type to preserve microsecond precision, while providing backward compatibility via the --updateCompatibilityVersion option. The review feedback highlights two main improvements: first, using OffsetDateTime.parse(strValue).toInstant() in IcebergUtils to avoid DateTimeParseException when parsing strings with non-UTC timezone offsets; second, refactoring the new test in IcebergIOReadTest into two separate test methods to avoid sequential pipeline executions and properly utilize the TestPipeline rule.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

claudevdm · 2026-07-16T14:07:53Z

          .put(SqlTypes.DATETIME.getIdentifier(), Types.TimestampType.withoutZone())
          .put(SqlTypes.UUID.getIdentifier(), Types.UUIDType.get())
          .put(MicrosInstant.IDENTIFIER, Types.TimestampType.withZone())
+          .put(Timestamp.IDENTIFIER, Types.TimestampType.withZone())


Will using timestamp logical type break xlang python reads?

claudevdm · 2026-07-16T14:11:31Z

+   * Converts an Iceberg {@link org.apache.iceberg.Schema} to a Beam {@link Schema}, accounting for
+   * update compatibility.
+   */
+  public static Schema icebergSchemaToBeamSchema(


I see some other call sites that do not use the update compat override

beam/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergCatalogConfig.java

Line 205 in 2cd12e5

return IcebergUtils.icebergSchemaToBeamSchema(table.schema());

beam/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcOutputUtils.java

Line 139 in 2cd12e5

return IcebergUtils.icebergSchemaToBeamSchema(

claudevdm · 2026-07-16T14:15:57Z

          .put(SqlTypes.DATETIME.getIdentifier(), Types.TimestampType.withoutZone())
          .put(SqlTypes.UUID.getIdentifier(), Types.UUIDType.get())
          .put(MicrosInstant.IDENTIFIER, Types.TimestampType.withZone())
+          .put(Timestamp.IDENTIFIER, Types.TimestampType.withZone())


What if it is a Timestamp.NANOS? Should we throw if the logical type is not explicitly MICROS?

claudevdm · 2026-07-16T14:16:50Z

            .map(tr -> BigQueryUtils.toBeamRow(BEAM_SCHEMA, tr))
            .collect(Collectors.toList());

+    System.out.println("returned: " + beamRows);


claudevdm · 2026-07-16T14:17:33Z

            .put("gcp_project", OPTIONS.getProject())
            .put("gcp_location", "us-central1")
            .put("warehouse", warehouse)
+            .put("io-impl", "org.apache.iceberg.gcp.gcs.GCSFileIO")


What is this for?

claudevdm · 2026-07-16T14:51:41Z

        }
      } else if (fieldType.isLogicalType(Timestamp.IDENTIFIER)) {
-        return VAR_PRECISION_FORMATTER.parse(jsonBQString, java.time.Instant::from);
+        try {


This is also used by load jobs and direct reads and both of them use the UTC string version right?

I am a bit concerned about cost of try to parse -> throw exception (expensive) -> try again for every single row in those cases.

Can we either do a conditional (check if ends with UTC or if contains :) to infer which approach we should take?

Maybe a future improvement we can make is the caller can pass the API they used to read so we know what type to expect.

claudevdm · 2026-07-16T14:54:40Z

-    assertEquals(52, actual.atZone(java.time.ZoneOffset.UTC).getMinute());
-    assertEquals(7, actual.atZone(java.time.ZoneOffset.UTC).getSecond());
-    assertEquals(123456000, actual.getNano());
+    String parsableTimestamp = "2024-08-10T16:52:07.123456Z";


Can we add a negative epoch and micros with leading zeros (e.g. .012345) test?

claudevdm · 2026-07-16T15:08:37Z


  @Test
  @SuppressWarnings("JavaInstantGetSecondsGetNano")
  public void testToBeamRow_timestampMicros_utcSuffix() {


Update test name since it doesnt just test UTC anymore?

Ahmed Abualsaud added 2 commits July 15, 2026 13:57

switch to new timestamp logical type

2fb2103

changes and trigger ITs

fa43293

github-actions Bot added java io labels Jul 15, 2026

gemini-code-assist Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergUtils.java

Comment thread sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/IcebergIOReadTest.java

address comments

88064c7

ahmedabu98 mentioned this pull request Jul 15, 2026

[Iceberg CDC] Add Changelog readers and update resolver #38837

Open

3 tasks

Ahmed Abualsaud and others added 3 commits July 15, 2026 14:59

format changes

14d0e04

add link

2ada632

tighten BQ conversion logic

69a4d11

github-actions Bot added the gcp label Jul 16, 2026

spotless

d18d8f8

claudevdm reviewed Jul 16, 2026

View reviewed changes

Uh oh!

Conversation

ahmedabu98 commented Jul 15, 2026

GitHub Actions Tests Status (on master branch)

Uh oh!

ahmedabu98 commented Jul 15, 2026

Uh oh!

gemini-code-assist Bot commented Jul 15, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants