Introducing New Telemetry

There will be a couple of changes to the data structure:

• All platform (kbc_*) tables – Primary Keys will be slightly changed to add stack identifiers, which will help us programmatically manage the script and make implementation of the new stack easier. Example: “kbc_project_id” 9241_us-east-1_aws will be changed to 9241_kbc-us-east-1. This is important only in case you are parsing values from Primary Keys.
• Usage Metrics Values – the usage_breakdown “Snowflake Writer” will be renamed “DWH Direct Query” so that it is less confusing for the user, who is often anticipating queue jobs of the component called Snowflake Writer (Data Destination).
• KBC Project – the column “kbc_project_stack” will be added, which uniquely identifies the stack where the project exists.

The major changes are happening on the backend of the telemetry data processing:

• All our internal Keboola Connection projects related to telemetry are managed by Keboola CLI, which allows us to easily manage updates, use GitHub reviews of the changes or initiate rollbacks in case of an issue.
• All our internal Keboola Connection projects related to telemetry are using Queue V2, so they are able to take advantage of all of the new and future Keboola Connection features.
• Using a new strictly managed multi-project architecture enables better readiness of the entire flow.
• We are addressing minor issues related to product updates that were not covered in the legacy telemetry.

How the release affects our users:

• Snowflake usage of Workspaces and Direct Queries will be adjusted, as the legacy telemetry did not correctly identify all of the Snowflake queries related to these usage breakdowns. The vast majority of users will see little to no impact from these changes and all those who will be most impacted have already been provided with further details by our Customer Success Team.
• We will force FULL load in the Telemetry Data extractor, as changes to Primary Keys would cause duplicates if incremental processing were to be used in your configuration. It also leaves one task for our users – they will need to process the data in full in their subsequent processes after the new telemetry is released in order to align their telemetry with the update.
• Telemetry data won’t be updated for up to two days so that we can ensure that users are not receiving incorrect data during the new telemetry product implementation.

In case you encounter any issues after the new telemetry is released, please contact support through the button in your project.