lua - How to get past 1gb memory limit of 64 bit LuaJIT on Linux? -

the overview prototyping code understand problem space, , running 'panic: unprotected error in call lua api (not enough memory)' errors. looking ways around limit.

the environment bottom line torch, scientific computing framework runs on luajit, , luajit runs on lua. need torch because want hammer on problem neural nets on gpu, there need representation of problem feed nets. (stuck) on centos linux, , suspect trying rebuild pieces source in 32bit mode (this reported extend luajit memory limit 4gb) nightmare if works @ all of libraries.

the problem space not particularly relevant, in overview have datafiles of points calculate distances between , bin (i.e. make histograms of) these distances try , work out useful ranges. conveniently can create complicated lua tables various sets of bins , torch.save() mess of counts out, pick later , inspect different normalisations etc. -- after 1 month of playing finding easy , powerful.

i can make work looking @ 3 distances 15 bins each (15x15x15 plus overhead), adding explicit garbagecollection() calls , using fork()/wait() each datafile outer loop keep running if 1 datafile (of several thousand) still blows memory limit , crashes child. gets painful each successful child process has read, modify , write current set of bin counts -- , largest files 36mb. go larger (more bins), , prefer hold counts in 15 gigs of ram can't seem access.

so, here paths have thought of; please comment if can confirm/deny of them will/won't me outside of 1gb boundary, or improve efficiency within it. please comment if can suggest approach have not thought of.

• am missing way fire off lua process can read arbitrary table in from? no doubt can break problem smaller pieces, parsing return table stdio (as system call lua script) seems error prone, , writing/reading small intermediate files lot of disk i/o.

• am missing stash-and-access-table-in-high-memory module ? seems want, not found yet

• can ffi c data structures put outside 1gb? doesn't seem case lack full understanding of causing limit in first place. suspect me efficiency improvement on generic lua tables few pieces have moved beyond prototyping? (unless bunch of coding each change)

• surely can out writing extension in c (torch appears support nets should go outside of limit), brief investigation there turns references 'lightuserdata' pointers -- mean more normal extension won't outside 1gb either? seems has heavy development cost should prototyping exercise.

i know c going ffi or extension route doesn't bother me - know experience encapsulating algorithms in way can both elegant , painful 2 places hide bugs. working through data structures containing tables within tables on stack doesn't seem great either. before make effort end result solve problem.

thanks reading long post.

only object allocated luajit limited first 2gb of memory. means tables, strings, full userdata (i.e. not lightuserdata), , ffi objects allocated ffi.new count towards limit, objects allocated malloc, mmap, etc. not subjected limit (regardless if called c module or ffi).

an example allocating structure malloc:

ffi.cdef[[     typedef struct { int bar; } foo;     void* malloc(size_t);     void free(void*); ]]  local foo_t = ffi.typeof("foo") local foo_p = ffi.typeof("foo*")  function alloc_foo()     local obj = ffi.c.malloc(ffi.sizeof(foo_t))     return ffi.cast(foo_p, obj) end  function free_foo(obj)     ffi.c.free(obj) end 

the new gc implemented in luajit 3.0 iirc not have limit, haven't heard news on it's development recently.

source: http://lua-users.org/lists/lua-l/2012-04/msg00729.html