Telemetry Update: Flow ID and Name in KBC Job Table

This week, we will be releasing an update to our telemetry data, specifically to the kbc_job table, where we will:

• Update the content of the flow column to include the name of the Flow configuration alongside the numeric ID, making it easier to distinguish between different Flows with the same name.
• Add a new flow_id column, which contains a foreign key to the kbc_component_configuration table, allowing users to join all information about the current version of the Flow configuration.

Previous Update

At the beginning of April, when we were addressing an issue with inconsistent Flow names in the column, we removed the Name from the existing column instead of adding a new column containing only the ID. This made it harder for users to use the information effectively.

New Data Structure

The new version of the flow column will contain values in the form of Flow Name (ID: Numeric Flow Configuration ID).

Flow Name represents the name of the Flow at the time the job was run and may differ if the name is changed later. This provides the most accurate representation of the Flow during job data processing.

Numeric Flow Configuration ID shows the numeric ID of the Flow configuration (this is not necessarily unique among all your component configurations).

Example: Process Raw Data (ID: 884004639)

The new flow_id column contains the unique identifier of the Flow configuration and can be joined with the kbc_component_configuration table via the kbc_component_configuration_id column. This allows users to join all jobs to the most recent version of the configuration, providing an up-to-date name for all jobs, along with other configuration metadata.

Example: 36_com-keboola-gcp-us-east4_keboola.orchestrator_884004639