Forum

Information and discussion related to the Kognitio on Hadoop product
Multiple Poster
Offline
User avatar
Posts: 7
Joined: Thu Jan 25, 2018 10:33 pm

Waiting for containers to check in. If this takes more than 5 minutes you may have a problem. Never finishes

by hadoop » Fri Jan 26, 2018 1:31 am

I am trying to create a cluster and actually see kodoop_container running but kodoop create_cluster never finishes.
Is there any way to see what is going on?

$ CONTAINER_MEMSIZE=16386 CONTAINER_VCORES=1 CONTAINER_COUNT=1 kodoop create_cluster k6
Kognitio Analytical Platform software for Hadoop ver80200rel171101.
(c)Copyright Kognitio Ltd 2001-2017.

Creating Kognitio cluster with ID k6
=================================================================
Cluster configuration for k6
Containers: 1
Container memsize: 16386 Mb
Container vcores: 1

Internal storage limit: 100 Gb per store
Internal store count: 1

External gateway port: 6550

Kognitio server version: ver80200rel171101

Cluster will use 16 Gb of ram.
Cluster will use up to 100 Gb of HDFS storage for internal data.

Data networks: all
Management networks: all
Edge to cluster networks: all
Using broadcast packets: no
=================================================================
Hit ctrl-c to abort or enter to continue

Creating cluster root in hdfs://.kodoop-clusters/k6
Synchronising package ver80200rel171101 to slider
Creating slider cluster kognitio-k6
Installing local copy of kognitio clients to /home/kognitio/kodoop/clusters/k6/wx2
Kognitio WX2 Software Installer v8.02.00-rel171101
(c)Copyright Kognitio Ltd 2004-2017.

Installing in user mode for administration by a single user.
Checking licences...
Using system ID k6.
Creating base directory structure in /home/kognitio/kodoop/clusters/k6/wx2.

Installing WX2 software:
Wxpkg file: version 3, minver 2.
Package ver80200rel171101, version 8.02.00-rel171101, version_no 80200.
Checksum: 986892737
Created on: 26-01_01:07:48_UTC by dev (Dev User).
Package root directory: ver80200rel171101.
Description: WX2 SQL database server software base package

Installed OK.

Setting current pointer /home/kognitio/kodoop/clusters/k6/wx2/current->ver80200rel171101.
Writing out system configuration.
Server configuration for new cluster k6:
# This file should only be edited with wxviconf or wxconftool!

[general]
system_id=k6

[logs]
hdfs_log_dir=.kodoop-clusters/k6/logs

[mpk]
checksum_enabled=1

[boot options]
external_scripts=yes ## imported from recommended_settings.cfg
external_tables=yes ## imported from recommended_settings.cfg
idle_core_cost=0 ## imported from recommended_settings.cfg
numa_aware=no ## imported from recommended_settings.cfg

[runtime parameters]
ds_ins_batch=0 ## imported from recommended_settings.cfg
Starting slider cluster for k6
Waiting for cluster to start up
This may take a few minutes, please be patient.
If this takes more than 5 minutes you may have a problem.
Cluster started, starting local runtime
Starting local management daemon
Waiting for containers to check in.
This may take a few minutes, please be patient.
If this takes more than 5 minutes you may have a problem.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 384
Joined: Thu May 23, 2013 4:48 pm

Re: Waiting for containers to check in. If this takes more than 5 minutes you may have a problem. Never finishes

by markc » Fri Jan 26, 2018 8:54 am

This can happen if there is insufficient resource (memory or vcores) to start up the required containers.

If you have access to the YARN resource manager, can you check the status of the application attempt? One method is to use a browser and connect to the node which the resource manager runs on.

Once you are there, can you check:
  • the progress bar of the application
  • the maximum allocation in terms of memory and vcores
Increasing the maximum allocations for containers can be done in e.g. Ambari, in the YARN configs tab. Please try this.

If this doesn't work, could you provide some information on what Hadoop distribution you are using, the underlying Linux distribution, the size of your cluster in terms of nodes, RAM per node, resource available for YARN applications, and any other relevant configuration information.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 384
Joined: Thu May 23, 2013 4:48 pm

Re: Waiting for containers to check in. If this takes more than 5 minutes you may have a problem. Never finishes

by markc » Fri Jan 26, 2018 9:29 am

One other way to get the symptom you are seeing is by not having installed the correct 32-bit libraries. https://kognitio.com/documentation/late ... aries.html has information on what libraries are needed, so could you check they are all installed. In particular, not having the 32-bit version of openssl produces the symptom you are seeing.

Kognitio are working on reducing prerequisites for running the software, and providing better diagnosis of problems like the one you are seeing, so in future this sort of diagnostic work either won't be required, or will be much easier.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 7
Joined: Thu Jan 25, 2018 10:33 pm

Re: Waiting for containers to check in. If this takes more than 5 minutes you may have a problem. Never finishes

by hadoop » Mon Jan 29, 2018 8:21 pm

64 nodes, 8 Cores,64GB, Ubuntu 16.04.3 LTS
running HDP-2.6.2.3-1 on MS Azure
Yarn has Maximum Container Size (Memory): 51GB, Maximum Container Size (VCores): 15

It may be the 32-bit libraries but I did run installation on all nodes
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 7
Joined: Thu Jan 25, 2018 10:33 pm

Re: Waiting for containers to check in. If this takes more than 5 minutes you may have a problem. Never finishes

by hadoop » Mon Jan 29, 2018 8:30 pm

looks like all libraries are installed:

# apt-get -q install gcc-6-base:i386 libc6:i386 libgcc1:i386 zlib1g:i386 libssl1.0.0:i386 libncurses5:i386
Reading package lists...
Building dependency tree...
Reading state information...
gcc-6-base:i386 is already the newest version (6.0.1-0ubuntu1).
libgcc1:i386 is already the newest version (1:6.0.1-0ubuntu1).
libncurses5:i386 is already the newest version (6.0+20160213-1ubuntu1).
libc6:i386 is already the newest version (2.23-0ubuntu10).
libssl1.0.0:i386 is already the newest version (1.0.2g-1ubuntu4.10).
zlib1g:i386 is already the newest version (1:1.2.8.dfsg-2ubuntu4.1).
The following packages were automatically installed and are no longer required:
linux-cloud-tools-4.4.0-92 linux-cloud-tools-4.4.0-92-generic linux-cloud-tools-4.4.0-93
linux-cloud-tools-4.4.0-93-generic linux-headers-4.4.0-92 linux-headers-4.4.0-92-generic linux-image-4.4.0-92-generic
linux-image-extra-4.4.0-92-generic
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 7
Joined: Thu Jan 25, 2018 10:33 pm

Re: Waiting for containers to check in. If this takes more than 5 minutes you may have a problem. Never finishes

by hadoop » Mon Jan 29, 2018 11:02 pm

Extract from the commands log:

RUNNING hadoop fs -put -f - .kodoop-clusters/k10/cluster-start-info
Running /home/kognitio/kodoop/clusters/k10/wx2/current/bin/wxsvc -s status
Service System management daemon is not running.
Running /home/kognitio/kodoop/clusters/k10/wx2/current/bin/wxsvc -s start
Starting System management daemon: OK.
Running /home/kognitio/kodoop/clusters/k10/wx2/current/bin/wxtool -R
FATAL ERROR: Failure to send mop packet (Connection refused) - aborted before operation.
FATAL ERROR: Failure to re-send mop packet (Connection refused) - aborted before operation.
Running /home/kognitio/kodoop/clusters/k10/wx2/current/bin/wxtool -R
FATAL ERROR: Timeout on mop packet - aborted before operation.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 7
Joined: Thu Jan 25, 2018 10:33 pm

Re: Waiting for containers to check in. If this takes more than 5 minutes you may have a problem. Never finishes

by hadoop » Tue Jan 30, 2018 12:30 am

I was able to create and access the kognitio cluster after creating symlinks on the head node.
Reply with quote Top

Who is online

Users browsing this forum: No registered users and 1 guest

cron