Dynamic Backend for Python & R
Like Snowflake transformations, the backend size of Python and R transformations can now be changed as well.
This feature allows you to allocate more resources to speed up long-running jobs.
How does dynamic backend scaling work?
Simply go to Transformations in the navigation menu, choose a Python or an R transformation that you want to resize, and pick your candidate for backend resizing.
Before you run the job or transformation, you can manually configure the size of the backend by choosing one of the four options (extra-small, small, medium, and large) in the transformation configuration panel.
The default backend is small. If you decrease or increase the backend size, you will be charged appropriately. More info about credit consumption can be found here.
How do you access the dynamic backend scaling?
Note that this feature is available automatically only for non-PAYG projects migrated to new transformations with the new queue.
How can you determine which jobs would be best optimized by changing the size of the backend?
An excellent way to pick candidates for backend resizing is to navigate to “Jobs” in your navigation bar and optimize the jobs with the longest run time in the column Duration:
