Updates for Snowflake and BigQuery Backends

1. Native Types Internal Upsert Behavior for _timestamp Columns

The distinction between tables with native datatypes and string tables, as described in our documentation regarding the _timestamp column, is no longer applicable.

Key updates:

• The _timestamp column will now be updated only when any value in the row changes, ensuring consistent behavior across all table types.
• This feature can be enabled in Project Settings under Features > Native types incremental _timestamp.

Impact:

Currently, when you load a table incrementally, more rows can be updated than if you were to use the new feature. If you enable this feature, only rows that have changed will be updated, along with the internal _timestamp column. This can affect the amount of data when the change since filters are used in transformations.

Timeline:

• Optional enablement: Users can enable this feature in their project settings starting today.
• Default enablement: This feature will be enabled by default for all projects beginning July 1, 2024.

2. Snowflake Timestamp Export Format

This update enhances the precision of timestamps by adding fractional seconds to the exported data.

Key updates:

• All timestamp columns (the _timestamp column and timestamps in tables with native types) will now be exported in the YYYY-MM-DD HH24:MI:SS.FF9 format, which was previously YYYY-MM-DD HH24:MI:SS. The change is that fractional seconds will be exported.
• This feature can be enabled in Project Settings under the Snowflake export fractional seconds option.

Impact:

When code in a Python transformation parses the date from a timestamp and expects a certain format, the transformation may fail due to a parsing error.

Timeline:

• Optional enablement: Users can enable this feature in their project settings starting today.
• Default enablement: This feature will be enabled by default for all projects beginning July 1, 2024.

Please contact us if you have any queries or comments.