Script Environments

Kognitio external scripting lets users run non-SQL scripts (e.g. Python, R, bash, perl etc.) using any Linux compatible executable from the SQL interface. These require an environment to run in and users should ensure they have these set up before creating and invoking external scripts.

Script environments specify the executable such as Python, bash, Perl etc. to run when an external scrip using the environemnt is invoked.

To create a script environment you must have create script environment privilege on the system:

GRANT create script environment ON SYSTEM TO user_name

Setting up script environments is an administrative task on Kognitio if you don’t have administration rights on your Kognitio system you will need to contact an administrator to set up script envirnments for you

Script environments also specify how the executable is invoked, what resources it can utilize and how data is passed between Kognitio and the script environment.

Once created script environments reside in your Kognitio system and can be viewed and managed by granting privileges in a simlar way to schemas.

Creating script environments

The full syntax for creating script environments can be found in the script environment DDL.

Note that to create these script environments, the executables need to be installed on each of the nodes that can be used by your cluster (unless you specify your environment to run on specific nodes). Contact your system admininistrator to have them install it for you.

There are a number of optional configuration settings that can be set in the scripting environment. These are used as the default for all external scripts using the environments. See advanced configuration settings for more details.

Below are some examples of creating commonly used script environments and useful configuration settings when doing so.

Example 1: Creating a bash and perl script environments on Kognitio

Many programs are installed by default during the initial installation of Linux. The programs included depend on the version of Linux but using these as external script environments is extremely straightforward. The syntax for creating a script environment for the bash shell is:

CREATE SCRIPT ENVIRONMENT bash
COMMAND '/bin/bash'

Similarly a script environment for running perl code is:

CREATE SCRIPT ENVIRONMENT perl
COMMAND '/bin/perl'

Example 2: Creating a python script environment on Kognitio

Other programs that have been installed on the system after the initial installation can also be used. A script environment to run python code can be created using:

CREATE SCRIPT ENVIRONMENT python27
COMMAND '/usr/local/bin/python2.7'

Here Python 2.7 has already been installed on all nodes in the directory /usr/local/bin. On Kognitio standalone this installation can be done easily using the Kognitio command utilities wxsync and wxtool.


Example 3: Creating a R script environment on Kognitio

For some executables, such as R it might be necessary to provide additional options controlling how the executable is run:

CREATE SCRIPT ENVIRONMENT rscript
COMMAND '/usr/bin/Rscript --vanilla --slave'

Note that the executable used here is the rscript command. This is provided in all installations of R and is specifically designed for submitting scripts to R from files or the command line. The additional options control how R is executed, i.e. quietly with minimal setup overhead and no saving on exit (see the CRAN document “An Introduction to R, appendix B: invoking R” for more details). Kognitio recommends that these 2 options are always utilised when running R scripts from the Kognitio Analytical Platform.


Example 4: Using a bash script to compile c code on invocation

Within Kognitio it is possible to compile scripts as you invoke them. For example, below is a scripting environment calling a specified bash script that uses gcc to compile c scripts as they are invoked by Kognitio:

CREATE SCRIPT ENVIRONMENT c
COMMAND '/opt/kognitio/wx2/current/examples/c.wxenv'

The file c.wxenv is a bash script stored in the specified location on every node in the Kognitio system. The file contains bash code similar to the following:

#!/bin/bash
# Example wrapper script for C script environment
temp\_bin=\`mktemp\`
gcc -o $temp\_bin -x c <(tail -n +2 $1) || exit 1
$temp\_bin
rc=$?
rm -f $temp\_bin
exit $rc

This bash code makes a uniquely named temporary file then uses gcc to compile the script in c and write any output (i.e. the compiled executable) to the temporary file. The temporary file is then executed, the temporary file is removed and the bash script exits with the appropriate return code $rc.


Example 5: Restricting the numbers of threads in a script environment

The example below creates a bash script environment whose external scripts will only be invoked as a single thread, i.e. all parallelism is removed, unless specifically overridden in an individual script creation. This is useful when external scripts are used to access information about the system, such as from the logs (see section 10.7):

CREATE SCRIPT ENVIRONMENT bash_sng
COMMAND '/bin/bash'
LIMIT 1 THREADS

R scripts may often require more memory than the default 200MB given to external script invocations in Kognitio by default.:

CREATE SCRIPT ENVIRONMENT rscript_1tpn
COMMAND '/usr/local/R/bin/Rscript --vanilla --slave'
REQUIRES 2 GB RAM
LIMIT 1 THREADS PER NODE

Here some parallelism is retained but throttled to one thread per node allowing each invocation to consume more RAM, provided that the system parameters are set up to allow this.

Altering and dropping script environments

Sometimes it’s useful to alter a script environment rather than creating a new one, for example if many objects depend it then it’s quicker to alter than drop and recreate those.

For syntax, usage and notes see the scripting envirnoment which also covers dropping external script environments.

Note

Any alterations made become the default setting for all external scripts using the scripting environnment when the same setting is not set in the external script itself

Some examples of what can be altered:

ALTER SCRIPT ENVIRONMENT script-env SET REQUIRES 1 GB RAM

ALTER SCRIPT ENVIRONMENT script-env SET LIMIT 1 THREADS
ALTER SCRIPT ENVIRONMENT script-env SET LIMIT 1 THREADS PER NODE
ALTER SCRIPT ENVIRONMENT script-env SET RUN ON node-list