Monitoring RAM usage

Kognitio provides various tools for keeping track of RAM usage, from both Windows (via Kognitio Console) and a Linux shell. Before reading this article, it’s important to understand <link>(how RAM works) in Kognitio, in particular:

  • how RAM space is divided up into ramstores

  • how <link> hashing can cause different ramstores to contain different amounts of data

  • the different types of <link> image distribution

  • the 85% PMA limit for user images

Monitoring via Console

Console is Kognitio’s UI for Windows application. The following tools are available for monitoring RAM space:

Dashboard

Console’s dashboard provides a visual view of the state of disk and RAM resources. Click on view > reports > dashboard to access the dashboard view. The view shows overall disk and RAM usage as a percentage of the total available. It also has a graph which updates every 10 seconds. The graph has the following information:

color

text

meaning

red

max usage

this shows how full the fullest ramstore is

green

avg

the average

blue

min

this shows how full the least full ramstore is

magenta

total table images

the total RAM space taken up by table images

orange

total view images

the total RAM space taken up by view images

purple

total temp space

Total RAM usage overview

In a query pane, pressing ctrl+f2 will bring up a query which, when run, returns the overall RAM usage figures in the system.

RAM usage for each table and view

In a query pane, pressing ctrl+f7 will bring up a query which, when run, shows the amount of RAM used by each object in the system. It also shows other details such as whether the object is a table or view, and how the image is <link> distributed. Note that the amount of RAM used will depend on the distribution type. For replicated images, the RAM used by the object will be much higher because there is a copy of the object on each ramstore.

RAM usage for each ramstore

The system table ipe_process contains information on free and used RAM for each ramstore. One way to see how much ram is free on each RS is to run the following query:

select processor_nr, mpid, ram_size, ram_free, ram_free*100/ram_size as percent_free from ipe_process where type='rs'

Monitoring via Command Line

The queries mentioned above for ctrl+f2 and ctrl+f7 are available from the command line via wxsubmit. The only difference is that they need to be run with $f2 and $f7 respectively. First connect to a system with wxsubmit:

wxsubmit -s mysystem myuser;

Then type, for example, $f2 and press return

>$f2;
RAM GB   |Used GB  |Free GB  |Avail GB |% Used|% UnFr|RS GB    |RS No
115.9    |      0.4|    115.5|    115.5|  0.3%|  0.0%|      3.6|   32
Query           1                1 row     ----   0:00.1   0:00.1   0:00.1

System Tables

All of the above tools rely on information stored in various RAM-related system tables in the SYS schema. These tables contain more detailed information on data residency in RAM:

ipe_allram_images

This table has records for each table for each ramstore. Each record in ipe_ram_images contains information such as the number of records the given table has on the given ramstore, the transaction which created those records, distribution, and other information.

For example, to find out how much RAM is being used by view images in the system, run:

SELECT SUM(ram_used)
FROM ipe_allram_images a,
     ipe_allview_img b
WHERE a.table_id=b.image_id;

ipe_allram_partitions

This table has records for each table for each ramstore for each <link> partition. It is similar to ipe_ram_images other than the inclusion of partition information.

ipe_allview_img

This table contains a record for every imaged view in the system, along with it’s image_id, distribution type and other information

Freeing up RAM Space

RAM should be freed up in different ways depending on whether the object in question is a system table image or a user table or view image. System tables and their images are in the SYS schema, and are named with the ‘IPE’ prefix. User objects are usually in other schemas, and will not be named with the ‘IPE’ prefix.

user table images and user view images

Firstly, RAM can be freed up by simply dropping table images and view images:

DROP VIEW IMAGE myview;
DROP TABLE IMAGE mytable;

This should be done with care, as dropping an image will cause queries on that object to be much slower, as all data must be read from disk. Also, recreating the image later could be a lengthy process.

However, if a table has deleted rows, these rows will still be present both on disk and in the table’s RAM image, because deleted rows are only marked with a delete marker - they are not fully removed until a special operation is performed. In the case of disk, the deleted rows can be fully removed with a <link> repack or a <link> reclaim.

For table images, deleted rows can be removed in two ways:

  • drop and recreate the table image

  • run defrag table image

The syntax for defrag table image is:

DEFRAG TABLE IMAGE table [FORCE]

By default the command will only perform the de-fragmentation if more than 10% of the rows will be discarded from the image; however the FORCE option ensures the de-fragmentation is performed regardless of how much RAM will be freed.

To remove deleted or outdated rows from view images, the view image must be dropped and recreated. Defrag cannot be run on view images.

system table images

Sometimes you may see significant RAM being used by system tables such as IPE_ALLCOLUMN. To recover RAM space from system tables, see recovering space for SYS tables. Note that system tables and their images should never be dropped.