Forum

Discussions specific to version 8.1
Contributor
Offline
User avatar
Posts: 22
Joined: Tue Dec 10, 2013 6:59 am

CANNOT BOOT: wxbootcheck failed with these errors

by vishwasjione » Thu Jan 02, 2014 11:24 am

We just finished with installation of trial Kognitio and did 2nd stage for DB creation ( using file as disk source).. following is the log that we got .. can anybody help what is the error here ?


-----------------------------------------
About to create a new database on system 'kognitio'.
WX2 system has: 1 node in 1 group.
Disk resources: 50.0G in 1 disk.
System has 1 unique type of node.
System has 1 unique type of disk.
System RAM 5.72G, 5.72G for data processing.
2 CPUs available for data processing.

Detected node classes:
full: 1 node

Detected Operating platforms:
Linux-2.6.32-279.el6.x86_64: 1 node

THIS WILL ERASE ALL EXISTING DATABASE DATA
About to erase database data and reset to defaults.
Enter to continue or ctrl-c to abort.
:
Logging startup to /var/log/wx2/logs-kognitio/startup.T_2014-01-02_16:38:02_IST.
--> Cleaning up unwanted files/processes.
--> Clean up failed: falling back to pname method.
--> Examining system components.
--> Configuring WX2 software.
Generation results:
WARNING: Unable to recover images as no memory images detected.
WARNING: Memory image set not valid. Rebuilding images instead.
WARNING: New boota is the same as new bootb.
--> Initialising disks.
CANNOT BOOT: wxbootcheck failed with these errors:
For node ABCD318396D (ecode -2147483548, 81 bytes):
Test for libraries failed: can't find libraries: libz.so.1
Complete. Hit enter to continue.
:
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Thu Jan 02, 2014 11:34 am

This is indicating that libz.so is required for Kognitio to work - Kognitio is 32-bit software, so it needs some 32-bit libraries to be installed to work.

For instance, on 64-bit CentOS the following steps are required as root to install the required 32-bit libraries and other supporting software (e.g. perl):

yum install glibc.i686
yum install libgcc.i686
yum install zlib.i686
yum install perl
yum install openssl.i686
ln -s /usr/lib/libcrypto.so.10 /usr/lib/libcrypto.so
ln -s /usr/lib/libssl.so.10 /usr/lib/libssl.so


Regards,
Mark.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 22
Joined: Tue Dec 10, 2013 6:59 am

Re: CANNOT BOOT: wxbootcheck failed with these errors

by vishwasjione » Thu Jan 02, 2014 11:40 am

Thanks Mark,

You mean running the below commands would solve the issue -

yum install glibc.i686
yum install libgcc.i686
yum install zlib.i686
yum install perl
yum install openssl.i686
ln -s /usr/lib/libcrypto.so.10 /usr/lib/libcrypto.so
ln -s /usr/lib/libssl.so.10 /usr/lib/libssl.so

??
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Thu Jan 02, 2014 11:49 am

If you are running CentOS in an environment where yum can operate (i.e. your node has access to a repository), then running those commands should ensure all the required software is installed.

Mark.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 22
Joined: Tue Dec 10, 2013 6:59 am

Re: CANNOT BOOT: wxbootcheck failed with these errors

by vishwasjione » Thu Jan 02, 2014 11:54 am

I have red hat Linux ? will it work ?
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Thu Jan 02, 2014 12:12 pm

Yes, that should be OK as CentOS is supposed to be 100% binary compatible with RHEL.

Mark.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 4
Joined: Thu Jan 02, 2014 12:30 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by RKMISRA » Thu Jan 02, 2014 1:20 pm

Hi,

I am working along with Vishwas on the Kognitio installation. As per your advice, i tried executing the commands provided. And below is the log for the same.

# yum install glibc.i686
Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi, presto, refresh-packagekit, security, tmprepo, verify, versionlock
Loading support for CentOS kernel ABI
Loading mirror speeds from cached hostfile
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6 ... 64&repo=os error was
14: PYCURL ERROR 6 - "Couldn't resolve host 'mirrorlist.centos.org'"
Error: Cannot find a valid baseurl for repo: base

Please advice how to get the 32-bit library installed.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Thu Jan 02, 2014 1:43 pm

You will have to enable access to the repository, as that is outside the scope of this forum.

Vishwas indicated you were using RH, but you are trying to access a CentOS site here, so I am guessing you are really using CentOS. In that case, you could try getting support from CentOS.

A quick search did show these URLs with the same error, so I'd suggest reading through them - the second one seemed to result in resolution: https://www.centos.org/forums/viewtopic.php?t=8892 and https://www.centos.org/forums/viewtopic.php?t=4760.

Mark.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 22
Joined: Tue Dec 10, 2013 6:59 am

Re: CANNOT BOOT: wxbootcheck failed with these errors

by vishwasjione » Fri Jan 03, 2014 5:15 am

Thanks Mark,

Sorry about RHD/Centos confusion, it was due to our teams different locations.

We have some network restrictions on connecting to these libraries of centos.
We are planning to get copy of them with the help of network administrator and then place them in appropriate location. But we are not sure where to place once we have those libraries ready with us, could you please help where to place these libraries ?


Thanks
Vishwas
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Fri Jan 03, 2014 8:28 am

Vishwas,

This seems like a generic CentOS issue (patching machines without internet access), so again contacting them or checking their forums is likely to be the best way to resolve this.

http://serverfault.com/questions/218472 ... connection might be a good place to start with this, as it shows how you can setup a local repository having downloaded patches (albeit for RH rather than CentOS).

http://www.tecmint.com/install-software ... ion-dvdcd/ explains how to use a CentOS installation DVD to install relevant packages.

Regards,
Mark.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 4
Joined: Thu Jan 02, 2014 12:30 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by RKMISRA » Mon Jan 06, 2014 1:56 pm

Hi Mark,

I have found that though I cannot execute "yum install *****" on my machine as per organisation policy. So I was able to procure the RPMs manually and can install them with the "rpm -ivh ****" command.

But I would like to know if Kognitio is looking for any particular version/release of the packages mentioned.

Currently the versions available on this machine with CentOS 6.3 are as follows:

# rpm -qa | grep glibc
compat-glibc-2.5-46.2.x86_64
glibc-common-2.12-1.80.el6.x86_64
compat-glibc-headers-2.5-46.2.x86_64
glibc-headers-2.12-1.80.el6.x86_64
glibc-utils-2.12-1.80.el6.x86_64
glibc-2.12-1.80.el6.i686
glibc-2.12-1.80.el6.x86_64
glibc-devel-2.12-1.80.el6.x86_64

# rpm -qa | grep libgcc
libgcc-4.4.6-4.el6.x86_64
libgcc-4.4.6-4.el6.i686

# rpm -qa | grep zlib
zlib-devel-1.2.3-29.el6.i686
zlib-1.2.3-29.el6.i686
zlib-devel-1.2.3-27.el6.x86_64
jzlib-1.0.7-7.5.el6.x86_64
zlib-1.2.3-27.el6.x86_64

# rpm -qa | grep perl
perl-XML-RegExp-0.03-7.el6.noarch
perl-URI-1.40-2.el6.noarch
perl-DBD-Pg-2.15.1-3.el6.x86_64
perl-libwww-perl-5.833-2.el6.noarch
perl-version-0.77-127.el6.x86_64
perl-devel-5.10.1-127.el6.x86_64
perl-Time-HiRes-1.9721-127.el6.x86_64
perl-SGMLSpm-1.03ii-21.el6.noarch
perl-Git-1.7.1-2.el6_0.1.noarch
perl-5.10.1-127.el6.x86_64
perl-Date-Manip-6.24-1.el6.noarch
graphviz-perl-2.26.0-7.el6.x86_64
perl-Test-Pod-1.40-1.el6.noarch
perl-ExtUtils-MakeMaker-6.55-127.el6.x86_64
perl-Compress-Raw-Zlib-2.020-127.el6.x86_64
perl-IO-Compress-Zlib-2.020-127.el6.x86_64
perl-YAML-Syck-1.07-4.el6.x86_64
perl-XML-Parser-2.36-7.el6.x86_64
perl-Module-Pluggable-3.90-127.el6.x86_64
perl-Error-0.17015-4.el6.noarch
perl-Pod-Coverage-0.20-4.el6.noarch
perl-DBD-MySQL-4.013-3.el6.x86_64
perl-Devel-Symdump-2.08-2.el6.noarch
perl-libs-5.10.1-127.el6.x86_64
perl-DBI-1.609-4.el6.x86_64
perltidy-20090616-2.1.el6.noarch
perl-ExtUtils-ParseXS-2.2003.0-127.el6.x86_64
perl-HTML-Tagset-3.20-4.el6.noarch
perl-Pod-Simple-3.13-127.el6.x86_64
perl-Test-Simple-0.92-127.el6.x86_64
perl-Test-Harness-3.17-127.el6.x86_64
perl-CGI-3.51-127.el6.x86_64
perl-Test-Pod-Coverage-1.08-8.1.el6.noarch
perl-HTML-Parser-3.64-2.el6.x86_64
perl-IO-Compress-Base-2.020-127.el6.x86_64
perl-Compress-Zlib-2.020-127.el6.x86_64
postgresql-plperl-8.4.11-1.el6_2.x86_64
perl-XML-DOM-1.44-7.el6.noarch
perl-Pod-Escapes-1.04-127.el6.x86_64

# rpm -qa | grep openssl
openssl-devel-1.0.0-20.el6_2.5.x86_64
openssl-1.0.0-20.el6_2.5.x86_64
openssl098e-0.9.8e-17.el6.centos.2.x86_64


This is in pursuance of the same issue of wxbootcheck error that we got when we selected 10: newsys in Stage 2 install.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Mon Jan 06, 2014 4:15 pm

You should be OK if you do the rpm equivalent of:

yum install glibc.i686
yum install libgcc.i686
yum install zlib.i686
yum install perl
yum install openssl.i686
ln -s /usr/lib/libcrypto.so.10 /usr/lib/libcrypto.so
ln -s /usr/lib/libssl.so.10 /usr/lib/libssl.so


Regards,
Mark.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 4
Joined: Thu Jan 02, 2014 12:30 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by RKMISRA » Tue Jan 07, 2014 1:25 pm

Hi Mark,

I was able to update all the packages manually. The Database creation moved forward but aborted. Please see the below details.

-----------------------------------------
Enter the system id to confirm that this is the correct system
(ctrl-c goes back, type ? for help)? kognitio


-----------------------------------------
About to create a new database on system 'kognitio'.
WX2 system has: 1 node in 1 group.
Disk resources: 50.0G in 1 disk.
System has 1 unique type of node.
System has 1 unique type of disk.
System RAM 5.72G, 5.72G for data processing.
2 CPUs available for data processing.

Detected node classes:
full: 1 node

Detected Operating platforms:
Linux-2.6.32-279.el6.x86_64: 1 node

THIS WILL ERASE ALL EXISTING DATABASE DATA
About to erase database data and reset to defaults.
Enter to continue or ctrl-c to abort.
:
Logging startup to /var/log/wx2/logs-kognitio/startup.T_2014-01-07_18:42:11_IST.
--> Cleaning up unwanted files/processes.
--> Clean up failed: falling back to pname method.
--> Examining system components.
--> Configuring WX2 software.
Generation results:
WARNING: Unable to recover images as no memory images detected.
WARNING: Memory image set not valid. Rebuilding images instead.
WARNING: New boota is the same as new bootb.
--> Initialising disks.
--> Initialising Database.
--> Creating newsys.sql new system script
--> Building new database
--> Logging build to /var/log/wx2/logs-kognitio/newsys.T_2014-01-07_18:49:59_IST
--> Creating base system tables
Stopping daemons as server crash reported during SMD operation.
08S02: [Kognitio][WX2 Driver] Session has been aborted
08S01: [Kognitio][WX2 Driver] Communications link failure
08S01: [Kognitio][WX2 Driver] Communications link failure
Error running newsys.sql, rc 256.
--> Replicating newsys logs to all nodes.
Nothing to do or -n option given to wxsync.
Failed, rc 0x40000001.
Complete. Hit enter to continue.
:

Kindly advice what needs to be done for this error.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Tue Jan 07, 2014 3:02 pm

Glad to see that got further with the libraries etc. in place.

Can you check http://www.kognitio.com/forums/viewtopic.php?f=4&t=139 to see that e.g. the prerequisites in that case have been addressed.

Assuming they have, can you tar up the contents of /var/log/wx2 on the node, and also /var/log/messages (you will need to be root for that one), and attach them to this case so we can review the log files.

Regards,
Mark.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 22
Joined: Tue Dec 10, 2013 6:59 am

Re: CANNOT BOOT: wxbootcheck failed with these errors

by vishwasjione » Wed Jan 08, 2014 5:34 am

Thanks Mark,

The issue that we faced with library files and issue we are facing right now do you think if we have fresh RHL machine we will not face these issues ?

We have to think now from other perspective also as we don't have much time left in coming up with Kognitio POC results which can help us decide our final production infrastructure.


Thanks
Vishwas
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Wed Jan 08, 2014 9:00 am

Vishwas,

I would expect if you follow the instructions linked to at http://www.kognitio.com/forums/viewtopic.php?f=4&t=139 that things should work. These include things like ensuring there is no impediment to communication between processes (AppArmor, firewalls), ensuring the right libraries and other software are installed, etc.

If you run into any problems, attaching the tarred up contents of /var/log/wx2 and the /var/log/messages file from each node should allow us to assist.

Regards,
Mark.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 4
Joined: Thu Jan 02, 2014 12:30 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by RKMISRA » Wed Jan 08, 2014 1:13 pm

Hi Mark,

Attached are the logs that you requested.
Attachments
messages.tar.gz
messages
(59.93 KiB) Downloaded 750 times
kognitio_wx2.tar.gz
contents of /var/log/wx2
(181.67 KiB) Downloaded 798 times
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Wed Jan 08, 2014 1:56 pm

We can see the Linux OOM killer is getting in, from looking at the /var/log/messages file (output below).

As Kognitio is an in-memory product it will try to use all the RAM on the node having left a bit over for the Linux kernel, etc. So if you have other software running on this node, you will need to either stop that software, or configure the Kognitio software to leave more space.

1) Ideally you'd stop any other non-Kognitio software. From the messages file I can see you are running the impalad daemon, which has significant memory requirements (I believe Impala uses in-memory HDFS caching and probably other in-memory techniques). I can also see the following using RAM, none of which are part of the Kognitio software:
  • large Java processes.
  • many httpd daemons.
  • console-kit-dae - I do not know what this is!
  • various Gnome processes - normally we'd use a text-only install on server nodes, rather than having a GUI running on them.
  • firefox
  • many other user applications.
2) If that is not possible you can edit the Kognitio config file using the wxviconf tool to restrict Kognitio to using a smaller percentage of RAM. First of all you need to remove any existing /dev/shm/wx2* files by doing the following as the root user on each node, if there are any files in /dev/shm whose filename starts wx2:

rm -f /dev/shm/wx2-*

Then you need to add a setting in the [boot options] section of that file (create that section if it is not already there), of this format:

fixed_pool_size=50

This will reserve the specified percentage of RAM for non-Kognitio use. To get things started, you could try that 50 figure, which means Kognitio will only try to use half the RAM on the system.


Below is evidence of the OOM killer getting in:

...
Jan 7 18:15:29 BHUKRK318396D NetworkManager[2146]: <info> domain name 'ad.xxxxxx.com'
Jan 7 18:50:14 BHUKRK318396D kernel: wxpmatool invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Jan 7 18:50:15 BHUKRK318396D kernel: wxpmatool cpuset=/ mems_allowed=0
Jan 7 18:50:15 BHUKRK318396D kernel: Pid: 27970, comm: wxpmatool Not tainted 2.6.32-279.el6.x86_64 #1
Jan 7 18:50:15 BHUKRK318396D kernel: Call Trace:
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff810c4971>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff811170e0>] ? dump_header+0x90/0x1b0
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff812146fc>] ? security_real_capable_noaudit+0x3c/0x70
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff81117562>] ? oom_kill_process+0x82/0x2a0
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff811174a1>] ? select_bad_process+0xe1/0x120
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff811179a0>] ? out_of_memory+0x220/0x3c0
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff811276be>] ? __alloc_pages_nodemask+0x89e/0x940
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8115c2da>] ? alloc_pages_vma+0x9a/0x150
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff81131175>] ? shmem_alloc_page+0x55/0x60
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8117032d>] ? mem_cgroup_cache_charge+0xed/0x140
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff81136a2e>] ? __inc_zone_page_state+0x2e/0x30
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff81113ece>] ? find_get_page+0x1e/0xa0
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8111516a>] ? find_lock_page+0x2a/0x80
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8113434e>] ? shmem_getpage+0x27e/0x9f0
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff81134b0d>] ? shmem_fault+0x4d/0x70
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8113ec14>] ? __do_fault+0x54/0x510
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8113f1c7>] ? handle_pte_fault+0xf7/0xb50
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8115c1da>] ? alloc_pages_current+0xaa/0x110
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff81048ac7>] ? pte_alloc_one+0x37/0x50
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff810137f3>] ? native_sched_clock+0x13/0x80
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8113fe04>] ? handle_mm_fault+0x1e4/0x2b0
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff81044479>] ? __do_page_fault+0x139/0x480
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff810f31d3>] ? trace_nowake_buffer_unlock_commit+0x43/0x60
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff814fd830>] ? thread_return+0x4e/0x76e
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8100ba4e>] ? common_interrupt+0xe/0x13
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff8150326e>] ? do_page_fault+0x3e/0xa0
Jan 7 18:50:15 BHUKRK318396D kernel: [<ffffffff81500625>] ? page_fault+0x25/0x30
Jan 7 18:50:15 BHUKRK318396D kernel: Mem-Info:
...
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 3
Joined: Sat Nov 01, 2014 1:52 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by sravan » Thu Nov 13, 2014 6:25 am

Hi,

when i am trying to start the kognio server after reboot the nodes, getting the following error,
"Unable to boot: No boot information found on system". But when i checked in the log file output is
--> Cleaning up unwanted files/processes.
Cleaning old processes.
--> Clean up failed: falling back to pname method.
Erasing server working area.
--> Examining system components.
Using target {skey BOOTING-2071}
Detected:
WX2: Blocks 1, Nodes 3, Disks 3, Status Up.
Boot disks:
Boota: No disk!
Bootb: No disk!
WARNING: System is running without a hardware map file.
Unable to boot: No boot information found on system.

can you please suggest the work around to resolve this.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by markc » Thu Nov 13, 2014 8:58 am

I believe this is for a system where you originally had 2 nodes working normally, then changed to add a 3rd node from http://www.kognitio.com/forums/viewtopi ... p=634#p634.

If you tar up the files under /var/log/wx2 and post them as an attachment, we can have a look at what went wrong.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 3
Joined: Sat Nov 01, 2014 1:52 pm

Re: CANNOT BOOT: wxbootcheck failed with these errors

by sravan » Thu Nov 13, 2014 1:00 pm

Mark,

I have solved the issue which is due to disks which are not mounted after reboot. After added the entry in /etc/fstab. Now its working fine.


Thank you.
Reply with quote Top

Who is online

Users browsing this forum: No registered users and 1 guest

cron