Forum

Information and discussion related to the Kognitio on Hadoop product
Multiple Poster
Offline
User avatar
Posts: 2
Joined: Wed Oct 09, 2013 2:45 pm

Kodoop create_cluster fails to start on Cloudera 5.8.x CDH

by alan » Fri Jun 02, 2017 1:20 pm

I have previously been able to successfully create Kodoop clusters on Cloudera 5.8.x CDH, but I've started getting unexpected create_cluster failure messages?

The Kodoop create_cluster command finishes with the following error:

...
Starting slider cluster for qaapp16u1k1
Waiting for cluster to start up
This may take a few minutes, please be patient.
If this takes more than 5 minutes you may have a problem.
Cluster started, starting local runtime
Starting local management daemon
Waiting for containers to check in.
This may take a few minutes, please be patient.
If this takes more than 5 minutes you may have a problem.
Abandoning as slider cluster in state FINISHED:
kognitio-qaapp16u1k1 FINISHED application_1496166397000_0005
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 2
Joined: Wed Oct 09, 2013 2:45 pm

Re: Kodoop create_cluster fails to start on Cloudera 5.8.x C

by alan » Fri Jun 02, 2017 1:56 pm

A problem has been seen with Cloudera CDH where the version of python used by slider to install the kodoop application is using the Cloudera SCM python "/usr/lib64/cmf/agent/build/env/bin/python" to deploy and not the native installed python on the node "/usr/bin/python".

It is not entirely clear how this happens, but it has been linked to the stopping and starting of Hadoop services via the Cloudera SCM.

To diagnose why the Kodoop application has failed requires access to the NameNode to examine the application container logs. However, by default, those logs will be removed automatically when an application finishes. To keep the logs for debug purposes (advised) make a change to the "yarn-site.xml" to allow 1 hour before application logs are deleted.

<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>3600</value>
</property>

Search under the NodeManager configuration in the Cloudera Manager to change this setting.

Kodoop uses slider to install on the Hadoop cluster. Slider runs a python installer script for each Kodoop container. It gets the python path to use from the associated NodeManager. It has been observed that Cloudera can start a NodeManager with a path environment that picks up the CDH version of python which is used by the SCM agents. The slider Kodoop installer is invoked with the "-S don't imply 'import site' on initialization" and it will subsequently fail to import modules like JSON which are not present leading to the Kodoop cluster failing to start.

This can be resolved by adding the following property to the slider-client.xml file to specify the python to use rather than inherit from the NodeManager path.

Add the following to the “slider-client.xml” on the edge node under ./kodoop/slider/conf

<property>
<name>agent.python.exec.path</name>
<value>/usr/bin/python</value>
</property>

Making the change to the slider-client.xml is the recommended approach to avoid this Cloudera issue impacting on your Kodoop cluster installation.
Reply with quote Top

Who is online

Users browsing this forum: No registered users and 1 guest

cron