Forum

General discussion on using the Kognitio Analytical Platform.
Contributor
Offline
User avatar
Posts: 48
Joined: Tue May 28, 2013 1:44 pm

Query runs out of memory but lots of memory free

by anonymous2 » Tue Jun 04, 2013 5:22 pm

I am running some queries and keep getting out of memory errors (e.g. PM0007 with version 8, RS0001 with earlier versions), but I can see there is lots of memory free:

>$f2
RAM GB |Used GB |Free GB |Avail GB |% Used|% UnFr|RS GB |RS No
670.3| 83.2| 587.1| 200.7| 12.4%| 65.8%| 3.6| 186

What is going wrong?
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: Query runs out of memory but lots of memory free

by markc » Tue Jun 04, 2013 5:28 pm

The problem here is probably that you have a very skewed image - as you can see from the $f2 output, although you have 587.1GB free, only 200.7GB is available. Also, with version 8 the default amount of memory available for table/view images is 70% of RAM, so that 200.7GB is the remaining 30% which is available for use during queries, not for storing table/view image data.

To resolve this, review SYS.IPE_RAM_IMAGES to see which table is most heavily skewed, and drop its image. The documentation contains information on partial hashing which you should be able to use to remove skew like this.

The following query will show you the images exhibiting the worst skew at the end of the output:

select table_id, min(numrecs), max(numrecs), max(numrecs) - min(numrecs) from sys.ipe_ram_images group by 1 order by 4;
Reply with quote Top
Contributor
Offline
User avatar
Posts: 386
Joined: Thu May 23, 2013 4:48 pm

Re: Query runs out of memory but lots of memory free

by markc » Wed Sep 24, 2014 3:25 pm

The previous comment related to existing images being skewed.

If you get an out-of-memory error like RS0001 or PM0007 when trying to create an image and there is no memory skew shown by the $f2 submit query before trying to create the image, there are two possibilities:
  1. The image you are trying to put in RAM is just too large - this is likely if you are not hashing it, as without hashing you are unlikely to skew the image. If the image is too large you can try putting a subset of it in RAM (e.g. using a horizontally fragmented image if it is a table, or a predicated view if it is a view - see section 2 of the Kognitio Guide for more details).
  2. The image you are trying to put in RAM is skewed, such that a lot of the data goes to a small subset of the ram stores (typically one of them), and is likely to fill the memory of that ram store resulting in an out-of-memory error. In this case, run "select hash_mpid(hash_value(<comma-separate list of attributes being hashed on>)) from <object being imaged>" to see if there is skew. If there is you will need to use partial hashing (see the Kognitio Guide and search for "partial hash") to make the skewed values get treated differently. The documentation explains how to do this.
Reply with quote Top

Who is online

Users browsing this forum: No registered users and 1 guest

cron