Forum

Discussions specific to version 8.1
Multiple Poster
Offline
User avatar
Posts: 6
Joined: Thu Nov 19, 2015 6:35 pm

init.d service to auto start kognitio service

by rvasu » Wed Jan 27, 2016 11:36 pm

2 node cluster (kognitio81a and kognitio81b).
System ID is kognitio81.
DB version - 8.1.0
Server - CentOS 6 VM

Questions:
1. Does kognitio provide the /etc/init.d/service file to start up the cluster?

I wrote a script, that does a
sudo su - wxadmin -c "wxserver start"

But, I need the script and chkconfig entry on both VM, in case of a event when one VM goes down and not the other.

2. But, in normal scenarios, both VM will be rebooted during maintenance. Is it OK to run the wxadmin start from both nodes? how does kognitio handle it when the second node issues a start while the first node is in the middle of starting service?
Reply with quote Top
Contributor
Offline
User avatar
Posts: 384
Joined: Thu May 23, 2013 4:48 pm

Re: init.d service to auto start kognitio service

by markc » Thu Jan 28, 2016 8:08 am

In answer to your questions, you would not normally auto-start the DB processes - the danger here is that your nodes (typically a lot more than 2, by the way) went down because of some environmental problem, such as an a/c failure. So typically you want to ensure that problem has been completely rectified before you restart the DB. Also, you want to make sure that e.g. all the nodes are back up, so you don't end up unnecessarily running off software RAID and then later have to recreate the disk resources on any nodes which are missing at the start and are then reintroduced.

Similarly, you do not normally want to run "wxserver start" from both nodes - this will cause the DB to be restarted twice if both nodes are restarted, which is an unnecessary delay. It will also cause the DB to be restarted immediately if one node is started up after a fault, whereas most environment would prefer to check out that node, then look at restarting the DB including the node at a convenient time (as the DB can be up and running with one or more nodes missing by using software RAID).

Chapter 7 of the Kognitio Configuration and Maintenance Manual explains how software RAID is implemented, and the concepts of virtual disk stores and standby disks, plys options for automatically recreating bad disks on startup. Chapter 9 of the same manual covers fault tolerance settings for the system management daemons, which allow you to control behaviour when e.g. a node fails.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 6
Joined: Thu Nov 19, 2015 6:35 pm

Re: init.d service to auto start kognitio service

by rvasu » Thu Jan 28, 2016 6:33 pm

Thanks Mark.

Makes sense. But, in scenarios where there is an a/c failure, hardware, network or any issue, we will comment out(disable) the auto restart of the DB until the issue is resolved.

I apologize if i was not clear.. The scenario I am looking at is the scheduled bi-weekly maintenance I do where I shutdown the VM, take snapshots, start it back up. All other VMs are programmed to restart the databases running in them (oracle, mysql, postgres for ex). I would like to setup a similar auto restart of kognitio as well.

Otherwise, it will be a manual step to start up the service once the VM is up!

Assuming that the only caveat is just a time delay since both nodes start up service, is this a valid scenario to have chkconfig script to startup kognitio otherwise?
Reply with quote Top
Contributor
Offline
User avatar
Posts: 384
Joined: Thu May 23, 2013 4:48 pm

Re: init.d service to auto start kognitio service

by markc » Thu Jan 28, 2016 7:58 pm

As Kognitio is an MPP in-memory product, normally people are looking for maximum performance - and hence they run it on physical servers rather than VMs. Also, a typical installation would be in the 10-60 node range, rather than 2 nodes, so in that environment you would not consider having every node trigger a DB restart when it came up, as you could end up waiting for a long time for e.g. 60 database restarts to occur.

In your environment with 2 VMs I'm sure you could put together some scripted solution to check the status of the system (e.g. use tools like wxprobe to verify how many nodes are present), and restart the DB iff it is not currently running and all nodes are present.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 6
Joined: Thu Nov 19, 2015 6:35 pm

Re: init.d service to auto start kognitio service

by rvasu » Thu Jan 28, 2016 10:04 pm

Thanks Mark. I understand the implications of scripting service startup from all nodes if I were to run a huge environment like you mentioned.

For the 2 node VM test environment I have now, just for testing purpose, I have /etc/init.d/kognitio81 script on node a.


#!/bin/bash
# chkconfig: - 89 36
# description: This script is to auto start kognitio service upon VM reboot

# Source function library.
INITD=/etc/rc.d/init.d
. $INITD/functions

start() {
su - wxadmin -c "wxserver start"
}

stop() {
su - wxadmin -c "wxserver stop"
}

case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
status)
su - wxadmin -c "wxserver info state"
;;
*)
echo "Usage: $0 {start|stop|status|restart}"
esac

exit 0


kognitio service does not startup under both scenarios mentioned below. Please see attached (kognitio-auto-start.png) for error message.

1) When I have the script on node 'a'alone, leave node 'b' up, reboot node 'a' and try to start the service using chkconfig on node 'a'
2) When I have the script on both 'a' and 'b' nodes, reboot both nodes and have the script auto start service from both VMs (one after the other)

I see how scenario #2 may fail if the other node is still coming up when either of them try to start service. But, should't scenario #1 succeed?

What is missing in the environment/script?

I understand that you do not recommend auto start of service, but I guess I am trying to understand why the startup won't work now.
Reply with quote Top
Contributor
Offline
User avatar
Posts: 384
Joined: Thu May 23, 2013 4:48 pm

Re: init.d service to auto start kognitio service

by markc » Fri Jan 29, 2016 3:18 pm

I can't see the attachment with the error information - can you try posting that again, possibly just as a normal textual reply if the attachment isn't working?
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 6
Joined: Thu Nov 19, 2015 6:35 pm

Re: init.d service to auto start kognitio service

by rvasu » Fri Jan 29, 2016 5:21 pm

WXSERVER: Retrying connection attempt.
WXSERVER: Retrying connection attempt.
WXSERVER: Retrying connection attempt.
WXSERVER: Retrying connection attempt.
WXSERVER: Retrying connection attempt.
WXSERVER: Retrying connection attempt.
WXSERVER: Retrying connection attempt.
WXSERVER: Perhaps the SMD is not running?
WXSERVER: Retrying connection attempt.
WXSERVER: Perhaps the SMD is not running?
WXSERVER: Retrying connection attempt.
WXSERVER: Perhaps the SMD is not running?
WXSERVER: Retrying connection attempt.
WXSERVER: Perhaps the SMD is not running?
WXSERVER: Retrying connection attempt.

I can execute the same command (su - wxadmin -c "wxserver start") from command line as root user from node 'a' and startup services on both nodes. But, the output when run through chkconfig is as shown above
Reply with quote Top
Contributor
Offline
User avatar
Posts: 384
Joined: Thu May 23, 2013 4:48 pm

Re: init.d service to auto start kognitio service

by markc » Fri Jan 29, 2016 6:21 pm

The output suggests the smd was not running. Can you make your code do something like "ps -ef | grep wxsmd" to see if the smd is running? You need the node restart to have done "wxsvc start" on the nodes before you can do "wxserver start", as wxserver is just a client which speaks to the smd. "wxsvc start" will start the smd on the node, and IS something that happens by default (but you need it to have happened BEFORE running "wxserver start").
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 6
Joined: Thu Nov 19, 2015 6:35 pm

Re: init.d service to auto start kognitio service

by rvasu » Mon Feb 01, 2016 6:22 pm

Thanks Mark.

I will test your suggestion this Thursday and post the result.
Reply with quote Top
Multiple Poster
Offline
User avatar
Posts: 6
Joined: Thu Nov 19, 2015 6:35 pm

Re: init.d service to auto start kognitio service

by rvasu » Thu Mar 17, 2016 9:43 pm

Hi Mark
Thanks for the input. I had to take a break from this task for a while now. But, I wrote a init script with a while loop (there were some timing issue when 2 VMs in the cluster were restarted) to start wxsmd on both nodes first and then start the kognitio service. It seems to work without an issue.
Reply with quote Top

Who is online

Users browsing this forum: No registered users and 1 guest

cron