platform

What is New Queue and Why do we need it?

New Queue enables the use of many new features, such as new transformations, a dynamic back-end size, 24-hour job limit, execution of configuration rows in parallel, etc.

The Queue is a group of services that run jobs in Keboola Connection. New Queue (Queue v2) is a replacement for the job execution and orchestration system (including notifications and a scheduler). It was designed back in 2011. The primary goal of the new queue is to implement essential technical changes. Thus, it is conceptually the same as the old one and, in general, the changes will not be noticeable by end users, especially since there are very few changes to the UI. However, New Queue has a different API, so external integrations have to be modified. From an end user’s perspective, there is very little difference.

How can we migrate to the new queue?

To move a project to Queue v2, it must be migrated, which is done by Keboola.

Note that:

  • The project must not use legacy transformations.
  • The migration cannot be reversed.

What do you expect from me?

When you ask us for a new feature to be activated, we need your collaboration with the following:

  • Revise this document and the consequences of a migration.
  • Currently, we will migrate your system only when you want to enable a new feature. To begin the process, submit a ticket through Zendesk or use the feedback button in the platform.
  • Keboola Support will get back to you to check that the components in your project are compatible with New Queue.
  • You will then be asked for final confirmation before we proceed with the migration.
  • Keboola will migrate your project to New Queue on your behalf and enable the requested feature.

What can go wrong?

The migration process modifies orchestration configurations and transfers job records. A job migration is idempotent, so it can be run repeatedly in case it crashes or fails on something. The migration is one way (i.e., we can't revert a project from New Queue to the old Queue) mainly because the old Queue would not see any jobs created in New Queue.

Is there a deadline for the migration?

Not yet. The migration of a project to Queue v2 depends on a prior migration from legacy transformations.

Possible problems connected to a migration

  • External use of the API
  • Components using the forward token feature

External use of the API

Running and listing jobs are now done via a new API (which is the same for both). This has been simplified, so there is no separate debug call. Also, there is no separate branch job (branches are now better integrated). The Docker runner API and Queue API are deprecated (and are actually disabled for projects migrated to Queue v2).

Operation Old New
Create Job POST https://syrup.keboola.com/docker/{component}/run POST https://queue.keboola.com/jobs
Create Branch Job POST https://syrup.keboola.com/docker/branch/{branch}/{component}/run POST https://queue.keboola.com/jobs
Create a job with image POST https://syrup.keboola.com/docker/{component}/run/tag/{tag} POST https://queue.keboola.com/jobs
Create a branch job with image POST https://syrup.keboola.com/docker/branch/{branch}/{component}/run/tag/{tag} POST https://queue.keboola.com/jobs
List jobs GET https://syrup.keboola.com/queue/jobs GET https://queue.keboola.com/jobs
Kill Job POST https://syrup.keboola.com/queue/jobs/{jobId}/kill POST https://queue.keboola.com/jobs/{jobId}/kill

Components using the forward token feature and other apps

Although there are no changes in the component interface, some components may break, such as any component using a forward token and relying on the Job API.

Component ID Status (Status, Queue v2 Status)
4mile.ex-mysql-next Unpublished, incompatible
bizztreat.ex-segment Published, incompatible
fisa.app-gd-custom-colors Published, incompatible
kds-team.app-column-tagger Unpublished, incompatible
kds-team.app-component-configuration-updater Unpublished, incompatible
kds-team.app-fhs-menumix Unpublished, incompatible
kds-team.app-gd-custom-colors Deprecated, incompatible
kds-team.app-job-storage-trace-cleaner Unpublished, incompatible
kds-team.app-rest-api-deployment-tool Unpublished, incompatible
kds-team.app-scd-transformation-generator Unpublished, incompatible
keboola.app-transformation-pattern-scd Published, incompatible
keboola.config-migration-tool Platform Integrated, incompatible
keboola.ex-gooddata Published, incompatible
keboola.gd-user-mgmt Unpublished, incompatible
keboola.gooddata-writer Published, incompatible
keboola.metadata-gatherer Unpublished, incompatible
keboola.oracle-transformation Published, incompatible
keboola.project-backup Platform Integrated, incompatible
keboola.project-restore Platform Integrated, incompatible
keboola.wr-db-snowflake-v2 Unpublished, incompatible
keboola.wr-snowflake-workspace Unpublished, incompatible
tde-exporter Published, incompatible
keboola.app-orchestrator-trigger Unpublished, incompatible
esnerda.app-orchestrator-trigger-mod Unpublished, incompatible

UI changes

Though the orchestrator is completely new, the UI is very similar. Here are the main changes:

  1. There is no list of jobs in an orchestration. This list will return at some point.
  2. The “Job Retry” option is not available. This process retried failed tasks in an orchestration. Its reintroduction is under consideration.
  3. Task parameters are no longer editable via the UI, although they are still editable via the API. The reintroduction of this feature to the UI is under consideration.
  4. A task action is no longer editable. The reintroduction of this feature is not planned.