Standby disks

Standby disks are one way of protecting your data in the event that a disk fails or becomes unusable. Standby disks are not used by Kognitio in the normal course of events, but are instead held in reserve to replace any disks in the system which become unavailable (e.g. because the node controlling them fails).

When a system is restarted with missing disk resources, standby disks will activate to take their place, and the standby disks will have their data recreated from parity information. After the standby disks has been activated, the old disks can be recovered/repaired/replaced and added back to the system, where they will act as new standby disks to help deal with any future failures.

Adding standby disks

There are two ways to add standby disks; prior to commissioning a system, and after a system has been commissioned.

Prior to Commissioning

A number of disks can be reserved as standby disks by adding the num_standby_disks=n parameter under the [wxsmd] section of the configuration file, where n is the number of disks that will be reserved as standby disks. The remaining number of disks must be a multiple of the RAID cluster size. Also, the parameter reliability_features=yes needs to be set under the [wxsmd] section.

After these parameters are set, running a newsys will initialise the disks, and the standby disks will be ready from then on.

After Commissioning

If you already have a running system and wish to add standby disks to it, follow the steps below.

  1. ensure that the node with the standby disk has the same version of Kognitio software installed as the existing nodes, and (if identical hardware) the same drivers and firmware version. For example, run ethtool -i eth2 to check driver/firmware version for eth2, and similarly for other interfaces.

  2. edit the config file and add num_standby_disks=n to the [wxsmd] section, where n is the number standby disks required. Add reliability_features=yes to the same section if it is not already present.

  3. copy /opt/kognitio/wx2/etc/config from an existing node onto the node with the standby disk

  4. start the SMD on the standby node with wxsvc start and check the new disk can be seen with wxprobe -wD. At this point the uid should be something like <hostname:partition>

  5. run wxtool -Z to zero the standby disk

Note that Kognitio can become unresponsive when zeroing disks on nodes that are running DB processes, and so it is recommended that disks be zeroed before RAM and CPU resources from those nodes have been introduced into the system.

You can check progress of the disk format by reviewing the output file in the current SMD directory on the standby node. Once complete you will see a message indicating the uid has been replaced. For example:

30-05_14:34:38_BST: Stdout: Replaced DISK UID wxsupp02-wx2-rack4-enc1-5:/dev/cciss/c0d0p2 with WXD:4FC62185:00000001, uid_replaced 0.
30-05_14:35:11_BST: Stdout: Format uid wxsupp02-wx2-rack4-enc1-5:/dev/cciss/c0d0p2 resource /dev/cciss/c0d0p2 complete, rc 0.
  1. run wxserver start sysimage without recovery to introduce the disk, RAM and CPU resource from the new node. Review the clustermap in the current startup directory to check the new disk uid is present as a standby disk:

    > cat clustermap
           Uid             |     Raid    | state | mpid | cost | local | node
                           |(cluster,pos)| state |      |      |       |
    WXD:4FC622B1:00000001  |(     -1,  0)|     5 |      |<standby disk>| wxsupp02-wx2-rack4-enc1-5
    WXD:4FC61E93:00000001  |(      0,  0)|     1 |    9 |    0 |     1 | wxsupp01-wx2-rack4-enc1-1
    WXD:4FC61E93:00000002  |(      0,  1)|     1 |   19 |    0 |     1 | wxsupp01-wx2-rack4-enc1-2
    WXD:4FC61E93:00000003  |(      1,  0)|     1 |   29 |    0 |     1 | wxsupp01-wx2-rack4-enc1-3
    WXD:4FC61E93:00000004  |(      1,  1)|     1 |   39 |    0 |     1 | wxsupp01-wx2-rack4-enc1-4
    

The database is now running with a standby disk.

  1. Consider running a shape test on the new system to ensure no problems have been introduced by adding the node.

Replacing a failed disk with a standby disk

The following steps are not part of the integration procedure but just demonstrate how the standby disk can be introduced so DO NOT do this unless you are performing a test or actually want to replace a failed disk with a standby.

  1. ensure reliability features are on

  2. stop the database with wxserver stop

  3. stop the SMD on a non-standby node with wxsvc stop (Only required if testing otherwise skip this step)

  4. start the database with wxserver start and assuming the default setting for the reliability_features are enabled expect to see three failed attempts to restart due to the missing disk and then a fourth succesful attempt as the missing disk is permanently replaced by the standby.

The first three boot attempts will fail as by default reliability features will try to reboot three times looking for all hardware to be present before trying the standby disks:

> wxserver start
Kognitio WX2 System Controller v7.02.01-rel120509 on wxsupp01
(c)Copyright Kognitio Ltd 2001-2012.

   -->  Cleaning up unwanted files/processes.
   -->  Examining system components.
   -->  Configuring WX2 software.
Generation results:
ERROR: 1 disks missing.
Logging startup to /var/log/wx2/logs-wxsupp01/startup.T_2012-05-30_14:51:38_BST.
   -->  Cleaning up unwanted files/processes.
   -->  Examining system components.
   -->  Configuring WX2 software.
Generation results:
ERROR: 1 disks missing.
Logging startup to /var/log/wx2/logs-wxsupp01/startup.T_2012-05-30_14:51:44_BST.
   -->  Cleaning up unwanted files/processes.
   -->  Examining system components.
   -->  Configuring WX2 software.
Generation results:
ERROR: 1 disks missing.
Logging startup to /var/log/wx2/logs-wxsupp01/startup.T_2012-05-30_14:51:49_BST.
   -->  Cleaning up unwanted files/processes.
   -->  Examining system components.
   -->  Configuring WX2 software.
Generation results:
WARNING: Attempting to replace disk uid WXD:4FC61E93:00000004 with WXD:4FC62185:00000001.
WARNING: Will permanently replace disk WXD:4FC61E93:00000004 with WXD:4FC62185:00000001.
   -->  Initializing Database.
   -->  Loading system tables, user tables and view images
   -->  Starting automatic disk recreate from standbys
Completed crimage in 00:00:44.
Startup complete.  SERVER IS NOW OPERATIONAL.

The default behavior for the reliability_features is to automatically recreate the standby disk so check ipe_xor_element to ensure the recreate is underway:

>select mpid, status from ipe_xor_element order by 1;
       MPID|     STATUS
         18|          1
         37|          1
         55|          1
         73|          4
Query           1               4 rows     ----   0:00.2   0:00.2   0:00.2

If the recreate didn’t start automatically, you can start it manually with the command recreate disk <mpid>. A disk status of 0 indicates a bad disk, 1 is a good disk, 4 is recreating, 5 is awaiting reconfigure, 7 is recreating parity.