Forum

Discussions specific to version 8.1
Contributor
Offline
User avatar
Posts: 48
Joined: Tue May 28, 2013 1:44 pm

PI0183: Module HADOOP: Failed to fork

by anonymous2 » Tue Nov 29, 2016 2:01 pm

We are trying to setup connectivity between our Kognitio system and data in HDFS.

We created an external table over HDFS, but selecting from that table gives the error "PI0183: Module HADOOP: Failed to fork"

Here are our steps for setting up the hadoop module to access the HDFS data - is there a problem in those steps, or is this a bug:

create module hadoop;
alter module hadoop set mode active;
alter module hadoop set parameter libhdfs to '/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib64/libhdfs.so';
alter module hadoop set parameter hadoop_client to '/usr/lib/hadoop';
alter module hadoop set parameter java_home to '/opt/ibm/java-x86_64-71/jre';
alter module hadoop set parameter libjvm to '/opt/ibm/java-x86_64-71/jre/lib/amd64/default/libjvm.so';

create connector testhdfscon source hdfs target 'namenode XX.XX.XX.XX:8020, user test, wholefile true';

create external table customer (
id int,
name varchar(32),
address varchar(5000)
) from testhdfscon
target 'file /user/test/data.txt FMT_FIELD_SEPARATOR "|" FMT_IGNORE_EXTRA_FIELDS yes';

select * from customer;
Reply with quote Top
Contributor
Offline
User avatar
Posts: 48
Joined: Tue May 28, 2013 1:44 pm

Re: PI0183: Module HADOOP: Failed to fork

by anonymous2 » Tue Nov 29, 2016 2:03 pm

We looked in `wxlogd smd`/serverdbg* and saw the following at the time we were trying to use the external table mentioned before:

...
T_2016-11-27_01:02:41_GMT: PI HADOOP(0x23000000): get_hadoop_settings(): unable to fork: Invalid argument
T_2016-11-27_01:02:41_GMT: PI HADOOP(0x23000000): get_hadoop_settings() failed with ecode 0
T_2016-11-27_01:02:41_GMT: PI HADOOP(0x23000000): hdfs_connect() failed with ecode -41
...
Reply with quote Top
Contributor
Offline
User avatar
Posts: 48
Joined: Tue May 28, 2013 1:44 pm

Re: PI0183: Module HADOOP: Failed to fork

by anonymous2 » Tue Nov 29, 2016 2:05 pm

The error message means that the HDFS connector tried to run "hadoop classpath" but couldn't.

If the "hadoop" executable is not in the PATH, you need to specify where it is with the hadoop_client module parameter. In your case, we can see that module parameter is set to /usr/lib/hadoop, which is probably a directory rather than an executable.

To fix, find out where your hadoop executable is (log on to one of the nodes and run "which hadoop") and set the hadoop_client module parameter to that. Note that the hadoop client software must be available on every node.
Reply with quote Top

Who is online

Users browsing this forum: No registered users and 1 guest

cron