What Is the New Queue and Why Do We Need It?

The new queue introduces many features, such as new transformations, a dynamic backend size, 24-hour job limit, and execution of configuration rows in parallel, among many others.

Queue is a group of services that run jobs in Keboola Connection. The new queue (Queue v2)  implements essential technical changes to our original job execution and orchestration system (including notifications and a scheduler), which was designed back in 2011. The new queue stays conceptually the same as the old one, and since only very few changes were made to the UI, our end users won't notice anything. However, the new queue has a different API, so external integrations have to be modified.

Features of the new queue

The redesign of the queue allowed us to implement various features:

  • Parallel config rows execution
  • Dynamic backend size for Snowflake—making it possible to choose a larger warehouse for your job when needed
  • Versioning and a trash bin for Orchestrator configurations
  • Jobs can now run longer than 24 hours
  • More concise API of the individual services

How do you migrate to the new queue?

To move a project to Queue v2, it must be migrated. This will be done by Keboola.

Note that:

  • The project must not use legacy transformations.
  • The migration cannot be reversed.
  • The encrypted fields (user entered) of some components won't be migrated.

What do you need to do?

When you ask us to activate a new feature, we will need your collaboration on the following:

  • Currently, we will migrate your system only when you want to enable a new feature. To begin the process, submit a ticket through Zendesk or use the feedback button in the platform.
  • Keboola Support will get back to you to check that the components in your project are compatible with the new queue.
  • You will then be asked for final confirmation before we proceed with the migration.
  • Keboola will migrate your project to the new queue for you and enable the requested feature.

What can go wrong?

The migration process modifies orchestration configurations and transfers job records. A job migration is idempotent, so it can be run repeatedly in case it crashes or fails on something. The migration is one-way—we can't revert a project from the new queue to the original queue. This is mainly because the old queue would not see any jobs created in the new queue.

Is there a deadline for the migration?

Not yet. The migration of a project to Queue v2 depends on a prior migration from legacy transformations.

Possible problems connected to a migration

  • External use of the API
  • Components using the forward token feature

External use of the API

Running and listing jobs are now done via a new API (which is the same for both). This has been simplified, so there is no separate debug call. Also, there is no separate branch job—branches are now better integrated. The Docker runner API and Queue API are deprecated (and are actually disabled for projects migrated to Queue v2).

Operation Old New
Create Job POST{component}/run POST
Create Branch Job POST{branch}/{component}/run POST
Create a job with image POST{component}/run/tag/{tag} POST
Create a branch job with image POST{branch}/{component}/run/tag/{tag} POST
List jobs GET GET
Kill Job POST{jobId}/kill POST{jobId}/kill

Components using the forward token feature and other apps

Although there are no changes in the component interface, some components may break, such as any component using a forward token and relying on the Job API.

Component ID Status (Status, Queue v2 Status)
4mile.ex-mysql-next Unpublished, incompatible
bizztreat.ex-segment Published, incompatible Published, incompatible Unpublished, incompatible Unpublished, incompatible Unpublished, incompatible Unpublished, incompatible Unpublished, incompatible Unpublished, incompatible Unpublished, incompatible Unpublished, incompatible
keboola.config-migration-tool Platform Integrated, incompatible
keboola.metadata-gatherer Unpublished, incompatible
keboola.wr-snowflake-workspace Unpublished, incompatible
tde-exporter Published, incompatible Unpublished, incompatible Unpublished, incompatible

UI changes

Although the orchestrator is completely new, the UI is very similar. Here are the main changes:

  1. Task parameters are no longer editable via the UI, although they are still editable via the API. The reintroduction of this feature to the UI is under consideration.
  2. A task action is no longer editable. The reintroduction of this feature is not planned.

IP addresses

If you have a firewall set up, please make sure you're using current IP addresses listed in our documentation.