What Is the New Queue and Why Do We Need It?
The new queue introduces many features, such as new transformations, flows, dynamic backend size, timeouts per configuration, and execution of configuration rows in parallel, among many others.
Queue is a group of services that run jobs in Keboola Connection. The new queue (Queue v2) implements essential technical changes to our original job execution and orchestration system (including notifications and a scheduler), which was designed back in 2011. The new queue stays conceptually the same as the old one, and since only very few changes were made to the UI, our end users won't notice anything. However, the new queue has a different API, so external integrations have to be modified.
Features of the new queue
The redesign of the queue allowed us to implement various features:
- Parallel config rows execution
- Set the size of the Snowflake warehouse used by your transformations when needed
- Dynamic backends ability - set the instance size for Python and R transformations
- Increased overall security
- Better control over the job timeouts
- Shared code snippets and variables within transformations
- Flows - a new way to create data pipelines
- Multiple workspaces (personal or shared)
- Persistent Workspaces (Beta)
- More concise API of the individual services
How do you migrate to the new queue?
Keboola will take care of the migration process.
Note that:
- The project must not use legacy transformations.
- The migration cannot be reversed.
- The encrypted fields (user entered) of a few legacy components can not be migrated.
What do you need to do?
- To begin the migration process, contact your account executive or submit a support ticket.
- We will get back to you to check the prerequisites for the migration to the new queue are met.
- You will then be asked for final confirmation before we proceed with the migration.
- Keboola will migrate your project to the new queue for you and enable the requested features.
What can go wrong?
The migration process modifies orchestration configurations and transfers job records. A job migration is idempotent, so it can be run repeatedly in case it crashes or fails. The migration is one-way. We can't revert a project from the new queue to the original queue mainly because the old queue would not see any jobs created in the new queue.
Is there a deadline for the migration?
By the end of the year 2024.
Possible problems connected to a migration
- External use of the API
- Components using the forward token feature - see below
External use of the API
Running and listing jobs are now done via a new API. This API has been simplified, so there is no separate debug call. Also, there is no separate branch job—branches are now better integrated. The Docker runner API and Queue API are deprecated (and are actually disabled for projects migrated to Queue v2).
| Operation | Old | New |
|---|---|---|
| Create Job | POST https://syrup.keboola.com/docker/{component}/run |
POST https://queue.keboola.com/jobs |
| Create Branch Job | POST https://syrup.keboola.com/docker/branch/{branch}/{component}/run |
POST https://queue.keboola.com/jobs |
| Create a job with image | POST https://syrup.keboola.com/docker/{component}/run/tag/{tag} |
POST https://queue.keboola.com/jobs |
| Create a branch job with image | POST https://syrup.keboola.com/docker/branch/{branch}/{component}/run/tag/{tag} |
POST https://queue.keboola.com/jobs |
| List jobs | GET https://syrup.keboola.com/queue/jobs |
GET https://queue.keboola.com/search/jobs |
| Kill Job | POST https://syrup.keboola.com/queue/jobs/{jobId}/kill |
POST https://queue.keboola.com/jobs/{jobId}/kill |
Components using the forward token feature and other apps
Although there are no changes in the component interface, some components may break, such as any component using a forward token and relying on the Job API.
| Component ID | Status (Status, Queue v2 Status) |
|---|---|
4mile.ex-mysql-next |
Unpublished, incompatible |
fisa.app-gd-custom-colors |
Published, incompatible |
kds-team.app-column-tagger |
Unpublished, incompatible |
kds-team.app-component-configuration-updater |
Unpublished, incompatible |
kds-team.app-data-science-scaffolds |
Unpublished, incompatible |
kds-team.app-data-test-scaffolder |
Unpublished, incompatible |
kds-team.app-fhs-menumix |
Unpublished, incompatible |
kds-team.app-job-storage-trace-cleaner |
Unpublished, incompatible |
kds-team.app-rest-api-deployment-tool |
Unpublished, incompatible |
kds-team.app-scd-transformation-generator |
Unpublished, incompatible |
keboola.config-migration-tool |
Platform Integrated, incompatible |
keboola.metadata-gatherer |
Unpublished, incompatible |
keboola.wr-snowflake-workspace |
Unpublished, incompatible |
tde-exporter |
Published, incompatible |
keboola.app-orchestrator-trigger |
Unpublished, incompatible |
esnerda.app-orchestrator-trigger-mod |
Unpublished, incompatible |
UI changes
The orchestrator UI has been completely redesigned and it's called Flows now.
It's much more visual and offers a better overview of your data flow.
If you prefer the old look - a tabular view of the "orchestrations" is still available.
IP addresses
If you have a firewall set up, please make sure you're using current IP addresses listed in our documentation.