CACHE = 4,1,1 allows memory of 24MB.
-------------------
#15MB of memory
echo 15728640  > /var/cgroups/cache-test/memory.limit_in_bytes

#5MB of memory
echo 5242880  > /var/cgroups/cache-test/memory.limit_in_bytes

#6MB of memory
echo 6291456 > /var/cgroups/cache-test/memory.limit_in_bytes

#600MB of memory
echo 629145600 > /var/cgroups/cache-test/memory.limit_in_bytes


cmake . && make && cgexec -g memory:cache-test ./cache

----------------------------------------------
These notes contain my findings when setting up stxxl and cgroups to work together. I am making these notes as a future reference for a time when
I may not remember the things which I am learning now.

Firstly, if one creates a STXXL vector as follows and runs with cgroups, limiting to 5MB of memory:

const int CACHE = 1;
const stxxl::uint64 n = 16000;
typedef stxxl::VECTOR_GENERATOR<TYPE,1,CACHE>::result vector_type;
vector_type array;
for (stxxl::uint64 i = 0; i < n*n; i++)
{
  array.push_back(i);
}

It took me about 22 seconds to create this vector. The total allocated memory was 978MB.
My disk operated at about a bandwidth of 40MB/s while setting up this vector. Only disk writes are performed.
This vector contains 256 million integers. Therefore the size of each integer is approximately 4 bytes.
A total of 488 blocks were written to disk (each block is 2MB).

------------------------------------------------------
Let's consider what happens if we changed the following lines and stop using cgroups:

const int CACHE = 128;
const stxxl::uint64 n = 16000;
typedef stxxl::VECTOR_GENERATOR<TYPE,4,CACHE>::result vector_type;
typedef stxxl::vector<TYPE, 4, stxxl::lru_pager<CACHE> >::iterator itr;

This took about 18 seconds to create. Although 978MB was allocated in total, nothing was written to disk. This is because the memory available
is enough to store the whole vector.
---------------------------------------
For the case below (without cgroup):
const int CACHE = 50;
const stxxl::uint64 n = 16000;
typedef stxxl::VECTOR_GENERATOR<TYPE,1,CACHE>::result vector_type;
typedef stxxl::vector<TYPE, 1, stxxl::lru_pager<CACHE> >::iterator itr;

The total size needed to store the vector is 978MB. However, the vector is designed to hold only 200MB.
So 778MB is written to disk (389 blocks). It took 21 seconds to run.
-----------------------------------------
Same as above, but with cgroup (5MB):
All statistics were the same as above, but this one took 61 seconds to run.
--------------------------------------
Cgroup (6MB), cache size = 200MB, size of data = 978M, data dimensions =  (16000x16000),
but actual time needed was 58s
-----------------
Cgroup (6MB), cache size = 6MB, pages = 1, size of data = 978M, data dimensions =  (16000x16000),
but actual time needed was 52s
----------------
It takes about 14 seconds to read 550MB of memory. 39MB/s is the average disk bandwidth.
-----------------
non cache adaptive matrix multiply:
Suppose we are multiplying 2N*2N matrices. We make 8 recursive calls to N*N matrix multiply.
Then just before adding the two N*N matrices, we provide enough memory for 3*(2N*2N).
Then after the addition, we take away this memory.
-----------------
Three ways to track I/O usage (other than stxxl):
1) pidstat -d -e
2) cat /var/proc/PID/io
3) iotop -qqq -b -Paok -n 2 -d 30 | sort -n -k6 -r | head -n 20

Just be careful to only count I/Os done during the matrix multiplication.
-----------------------------------
Interaction between PAGES_PER_CACHE,BLOCKS_PER_PAGE and BLOCK_SIZE_IN_BYTES
------------------------------------
sudo cgdelete memory:cache-test && sudo cgcreate -g "memory:cache-test" -t abiyaz:abiyaz
-----
This error gets thrown for very low cgroup memory settings:
[STXXL-ERRMSG] io_error thrown in ~vector(): Error in virtual void stxxl::syscall_file::serve(void*, stxxl::file::offset_type, stxxl::file::size_type, stxxl::request_interface::request_type) :  this=0x56307a31f8f0 call=::write(fd,buffer,bytes) path=/var/tmp/stxxl fd=5 offset=0 buffer=0x7f27b52b3000 bytes=4194304 type=WRITE rc=-1 : Cannot allocate memory: iostream error: iostream error
----
https://stackoverflow.com/questions/33759054/using-cgexec-vs-cgroup-procs-for-memory-accounting-using-cgroups
------------------
1) why is it that with cgroup (3rd experiment in spreadsheet), the writes are not counted? or are they even done at all?
2) memory profile for non-cache cache_adaptive
3) taking the memory profile frin the above and feeding it to the cache cache_adaptive <-- shell script
------------------------
sudo cmake ./build && sudo make --directory=./build && sudo ./cache.sh 0 100 cache-test
---------------
1024x1024: Data: 1163132928
2048x2048: Data: 3578789888
4096x4096: Data: 1430257664 (non-adaptive)
8192*8192: Data: 1426063360 (adaptive)
16384x16384: Data: 1408237568 (non-adaptive)
--------------------
stxxk disk configuration direct = off/try/on
--------------------
data: Wq)b
---------------------------------
Create file of size 2GB.
Write file
--------------
sudo bash -c "echo 15000000000 > /var/cgroups/cache-test/memory.memsw.limit_in_bytes"
sudo bash -c "echo 64000000 > /var/cgroups/cache-test/memory.limit_in_bytes"
cat /var/cgroups/cache-test/memory.memsw.limit_in_bytes
sudo cgexec -g memory:cache-test ./a.out nullbytes
-------------------
https://serverfault.com/questions/255661/linux-oom-disk-i-o-also-swap-what-is-it-good-for
------
sudo dd if=/dev/urandom of=null count=256 bs=1048576
sar -A -o sar_256 1 >/dev/null 2>&1 &

#03:01:11 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util