Logging

Logs in Kognitio are kept in two places. Some logs are kept inside the Kognitio system, in tables on slab 2 (the logging slab). These tables include:

  • IPE_COMMAND: a history of past queries

  • IPE_ALLTRANSACTION: a history of transactions, including start and end times of transactions, and transaction states(committed / rolled back)

  • IPE_ERRORLOG: a history of errors

  • IPE_ALLLOGIN: a history of past logins, including the source IP of the login, login/logout times and other info

Other logs are kept outside of Kognitio, as files in the underlying Linux environment. these files are described below.

Linux log files

Log files in Linux are written to the system log directory. The default system log directory on a Kognitio Standalone system is /var/log/wx2/logs-<system_id>. On Kognitio on Hadoop, the default system log directory is under the install directory, in installdir/logs/logs-<system_id>/. All log files for a given cluster will be written to the system log directory or one of it’s subdirectories. The system log directory can be changed by setting the parameter log_dir=/path/to/log/dir in the [paths] section of the config file.

The system log directory may contain the following files and sub-directories, among others:

  • actions.<date>: These files log all SMD actions performed on the given date, along with the user, a timestamp, and the action performed. Relevant actions include executables which invokes the SMD, for example wxprobe, wxserver and others

  • startup.<datetime>: These directories contain log files related to a particular startup of a Kognitio system/cluster.

  • newsys.<datetime>: These directories contain logs related to a particular newsys

  • smd.<datetime>: These directories contain the majority of messages logged during normal Kognitio operations

  • wxtester.<datetime>: Contains log files related to a Shape Test run. Not normally used

  • reconfigures: This directory contains the wxsubmit output for all reconfigure operations that have been performed

  • smd-angel: This is a flat file that is maintained by the SMD angel. Entries get written to this file whenever the SMD is started. Entries also get written when the SMD crashes/is restarted/etc

Within each smd.<datetime> directory, the following files may be present:

  • serverdbg.<timestamp>: Contains general log messages from the server.

  • session.<timestamp>: Contains a history of session connections and disconnections

  • locking.<timestamp>: Contains messages about locking problems, for exampe deadlocks and lock timeouts

  • output.<timestamp>: Contains messages related to the local SMD. This file is not synced across nodes

Each time a new log subdirectory is created by the logging-module (e.g. an smd.<datetime> directory), a hint file is written out to point to its location. The hint files is placed in the system logging directory and are called .log-<type>, where <type> is either ‘newsys’, ‘smd’, ‘startup’ or ‘wxtester’. The files contain the full path of the directory that has just been created. For example, .log-smd will contain the path of the most recent smd.<datetime> directory. If a hint file of a given type existed before the log directory was created, the old contents are overwritten. Thus, the name of the most recent SMD log directory can be found with the command cat <system log directory>/.log-smd.

The wxlogd <type> command is provided with the software as a shortcut to display the path of the most recent directory of the given type. Thus, the following example commands are possible on a Standalone system:

  • cd `wxlogd smd`. Move to the most recent smd directory

  • cd `wxlogd startup`. Move to the most recent startup directory

  • cd `wxlogd newsys`. Move to the most recent newsys directory

  • ls `wxlogd smd`/serverdbg*. List all serverdbg files created since the last SMD restart

On a Kognitio on Hadoop system, the above commands will only work from a special sub-shell. First run kodoop mgr <system_id> shell to invoke the shell, then run one of the above commands.

Log file attributes

The log files created by the logging module have several attributes associated with them that can be set in the [logs] section of the configuration files. The log attributes control the way the logs are written out and the data that is placed in them. The attributes that can be set for a log are:

  • log_level: This controls how much information is put into the log. It is described later. Default value is 50.

  • add_time: This controls whether or not a date/time string will be put at the beginning of each line in the log. It is given as a Boolean value. Default value is yes.

  • max_time and max_size: These control log rotation, which is described later. Default values are 0 and 0x100000

  • full_mode: This controls the behavior of the log when the log volume is low on space and is described later. Default value is Discard.

  • min_days_history, max_days_history and logs_max_mb: These control log expiry, which is described later. Default values are 7, 30 and 50.

To set an attribute value for all logs in the system, set the [logs] AttributeName configuration value. The attributes can also be set for all logs in a given class by setting the [logs] Class_Attribute configuration value (e.g. [logs] smd_log_level=100 to enable debug output in all SMD logs). Finally, each attribute can be set for a given log file within a class by setting the [logs] Class_LogName_Attribute value. When multiple settings apply to a particular log, the most specific setting for an attribute is always used in preference to more general settings. This means that, for example, [logs] startup_output_log_level would be used for the startup.date/output log file in preference to any [logs] startup_log_level setting.

Log rotation

The logging-module can perform automatic rotation on the various logs it creates. Automatic rotation is useful to prevent a long-running process from creating a single huge file that fills up the log volume. Automatic rotation can be performed according to how long the log file has been open or according to how big the log file has grown. When a log file with rotation enabled is created, the log file’s name has a date string appended to it. When no rotation is enabled, the log’s name does not have a date string at the end of it. The date indicates when the file was created. When the log is rotated, the file is closed and a new one opened with a new date string. The old log file is still there, but it can now be safely expired by the SMD without crashing the process that created it. When rotating logs, the process writes a string to the end of the old log containing the name of the new log file, and it also writes a string to the beginning of the new log file containing the name of the old log file. Log rotation is controlled by the max_time and max_size attributes for the log. The max_time attribute specifies the maximum number of minutes the log can be open for. 0 disables time open based rotation. Similarly, the max_size attribute specifies the maximum size in bytes of the file. This size is approximate and the file may get a little bigger than this size. A value of 0 for a log disables size based log rotation. By default, no rotation at all is performed on the startup logs since these are typically created all in one go and do not grow over time. Size based rotation is performed on the SMD logs with a max_size value of 1Mb.

Log expiry

The logging-module can perform automatic expiry on the various logs it creates. Automatic expiry is useful to prevent old log files filling up the log volume. Automatic expiry can be performed according to how old the log file is or according to how full the logging directories are.

The expiry algorithm has sufficiently intelligence to prevent it expiring a current or useful log file, so, for example the log file from the last new system will always be preserved, regardless of when it was created or how full the logging directories become. Log expiry is controlled by the min_days_history, max_days_history and logs_max_mb attributes for the log. The min_days_history attribute specifies the age the file must reach before it is ever considered for expiry. The max_days_history attribute specifies the number of days after which a file should be expired. The logs_max_mb attribute specifies the maximum amount of logging information that should be maintained. Once this limit is reached, the oldest log files that are eligible for expiry will be removed until the amount of logging information is once again under the logs_max_mb threshold. Whatever the value of logs_max_mb, all eligible files will be expired once their age exceeds max_days_history and no file that is under the min_days_history threshold will ever be expired.

Log full behavior

When writing log files, the logging module checks to see how much space is available on the logging volume before writing each line. If the number of Mb of free space is lower than the [logs] min_free_mb configuration value (default 50), the logging volume is deemed to be full. If the logging volume is full, log writing behavior is controlled by the full_mode attribute for each log. The full_mode attribute can take the following values:

  • Ignore: Ignore the fact that the log is full and try to write anyway. Ignore any write errors.

  • Crash: Cause whatever process is writing the log to crash. An error message is written to stdout and a core dump is produced.

  • Block: Block the thread writing to the log until the log volume is no longer full, and then continue to write normally.

  • Discard: Write a message to the log file saying that log entries are being discarded and throw away all subsequent writes to the log. When the volume is no longer full, write another message saying logging is resumed and continue to write normally.

Logging levels

Information written to logs through the logging module is written with an accompanying log level. Each log also has a log_level attribute associated with it which controls how much information is actually written to the log. Log information is written if the logging level is <= the log_level attribute, otherwise the information is discarded. The log levels typically used for writes are:

  • 1: Essential log contents. Without entries at this level the log would be pointless.

  • 5: Error information.

  • 10: Warning information.

  • 50: Normal output. This should not be too big.

  • 100: Debug output. This is used for diagnosing various problems.

  • >100: Very verbose debug output. Turning these levels on can generate a LOT of output!

Setting the log_level to 0 for a log will turn it off completely because nothing ever outputs at level 0. The file will not even get created because log files are created on demand (i.e. when something is written to them). Note that for many of the startup logs ALL information is written at level 1 so it is only possible to turn the log on or off. For the SMD logs, occasionally setting the level to 100 can be very useful for helping Kognitio diagnose a problem.