Configuring how many invocations of the script run

Sometimes it’s necessary to limit the number of script invocations of a Kognitio external script whether due to available resources or if the script doesn’t require more than a certain amount determined by the default behaviour.

To set a limit on the number of invocations of an external script that are run across the entire Kognitio server add LIMIT n THREADS before the SCRIPT element of the CREATE EXTERNAL SCRIPT SQL. For full syntax see the external script DDL.

The limit on the number of threads can also be changed after a script is created:

ALTER EXTERNAL SCRIPT myschema.myscript SET LIMIT 1 THREADS;

The specification is a limit because Kognitio automatically manages the available resource allocation through its scheduler. Kognitio yypically allows an single external script query to utilize up to 50% of what’s available but this can be changed by the system administraor.

Note

LIMIT 1 THREADS removes parallelism and most of the potential performance benefits of external scripting.

As well as limiting the total number of threads it is also possible to limit the number of threads per node using LIMIT n THREADS PER NODE or after a script is created:

ALTER EXTERNAL SCRIPT myschema.myscript SET LIMIT 1 THREADS PER NODE;

This limit is useful in a number of different scenarios:

  • An external script may need to use a specific local resource, such as a named file, in which case a limit of 1 script invocation per node would be appropriate
  • A script may be reporting information about the hardware that Kognitio is deployed on. In this case multiple threads per node are not required
  • Some external scripts might be parallel themselves, and so only need one invocation per node to utilise all the available resource. For example using a library of routines for complex computation
  • Software licencing issues (from the providers of 3rd party software) may necessitate an absolute limit on the number of threads that can be used for a given piece of software, see section on configuring where the script is run for more details on this

Example: obtaining a list of R packages installed in Kognitio

Example: Bash script to list files in linux directory

Example: list the files in a linux directory on every node

The default external script invocation behaviour on Kognitio is for the number of threads to be equal to the number of ramstores on the system provided there is enough memory resource available. This default can be specified manually using either:

ALTER EXTERNAL SCRIPT myschema.myscript SET LIMIT 0 THREADS;
ALTER EXTERNAL SCRIPT myschema.myscript SET NO THREADS LIMIT;