Using Python in external scripts
Multiple Poster
User avatar
Posts: 3
Joined: Wed Sep 04, 2013 12:04 pm

Can't get my python script running in parallel

by swhiteside » Wed Sep 04, 2013 12:12 pm

Hi all

I am trying to get a simple python script running in Kognitio. When I run it through a single thread using limit threads command it works fine but I cannot get it running in parallel. I don't understand what I am doing wrong any tips.

Reply with quote Top
Single Poster
User avatar
Posts: 1
Joined: Wed Sep 04, 2013 12:19 pm

Re: Can't get my python script running in parallel

by Tasneef » Wed Sep 04, 2013 12:24 pm

Handling no input into python (2.7.5) external scripts on Kognitio console

When running scripts in Kognitio the default behaviour of the platform is to set up one script invocation per ram store(approx. number CPUS on platform). Whether a script is actually sent data to process is dependent on the hashed values of the fields declared in the partition command. Kognitio uses a similar hashing regime here to that used for handling joins and group by in SQL queries. There are instances (low partition cardinality or small data volumes) where some script invocations may not receive any data. This can cause errors in your python script if you do not test for it and handle it correctly
A simple approach to overcome this problem is flagging. Below is an example of how to test for this in python:

import csv
import sys

p =csv.reader(sys.stdin) #declare stdin as source of data
for row in p : # loop through input
for i in range(len(row)): # for each comma separated element in row change data type
row=float( row )
SAMPLES.append(row) # append row to SAMPLES array

flag = 0 # initialise flag
If SAMPLES == [ ] : # this checks whether the whole data feed into this node is empty or not
flag =1

if flag == 0:
{ run the main code }

If the array SAMPLES is empty the main code is not executed, the script simply exits and you avoid errors being generated by scripts sent no data
Reply with quote Top

Who is online

Users browsing this forum: No registered users and 1 guest