INSTALL

Process-in-Process (PiP) Installation

* Overview and GLIBC issue
  PiP is a library to create processes in a process's virtual address
  space implemented at user-level. Unfortunately the current GLIBC,
  specifically libpthreads.so, has a problem and PiP is unable to
  create sub-processes. To fix this issue, you have to install our
  patched GLIBC before building the PiP library.

* Patched GLIBC
  You can download the patched GLIBC from the following URL.

git clone -b pip-centos7 git@git.sys.aics.riken.jp:software/PIP-glibc

  With the GLIB which Linux distribution provides, the number of PiP
  tasks (see below) is limited to 15 or so. This limitation comes from
  the fact that the number of name spaces is hard-coded as 16. In
  addition to this, the current PiP library cannot run with some GLIBC
  libraries. To avoid these issues, we recommend you to have our
  patched GLIBC.

  Here is the recommended configure and other options for the patched
  GLIBC based on CentOS 7.2,

  +++ on x86_64

mkdir PIP-glibc.build
cd PIP-glibc.build
../PIP-glibc/configure --prefix=<GLIBC_INSTALL_DIR> CC=gcc CXX=g++ CFLAGS="-mtune=generic -fasynchronous-unwind-tables -DNDEBUG -g -O3 -fno-asynchronous-unwind-tables" --enable-add-ons --with-headers=/usr/include --enable-kernel=2.6.32 --enable-bind-now --build=x86_64-redhat-linux --enable-multi-arch --enable-obsolete-rpc --disable-profile --enable-nss-crypt --enable-systemtap
make -j3 && make install

  +++ on Aarch64 (ARM64)

mkdir PIP-glibc.build
cd PIP-glibc.build
../PIP-glibc/configure --prefix=<GLIBC_INSTALL_DIR> CC=gcc CXX=g++ CFLAGS="-fasynchronous-unwind-tables -DNDEBUG -g -O3 -fno-asynchronous-unwind-tables" --enable-add-ons --with-headers=/usr/include --enable-kernel=2.6.32 --enable-bind-now --build=aarch64-redhat-linux --enable-obsolete-rpc --enable-systemtap --disable-profile --enable-nss-crypt
make -j3 -r CFLAGS=" -fasynchronous-unwind-tables -DNDEBUG -g -O3" PARALLELMFLAGS=
make -j1 install PARALLELMFLAGS=

* Building PiP
  At the PiP configuration, you must specify the installed GLIBC.

git clone git@git.sys.aics.riken.jp:software/PIP
cd PIP
./configure --prefix=<PIP_INSTALL_DIR> --with-glibc-libdir=<GLIBC_INSTALL_DIR>/lib
make install

  After the successful PiP installation, you must do the following,

<PIP_INSTALL_DIR>/bin/piplnlibs

  This command creates a number of symbolic links to the SOLIBs which
  are not installed by the patched GLIBC installation.

* Building PiP-GDB
  The following procedure installs the pip-gdb(1) command
  as <PIP_GDB_INSTALL_DIR>/bin/pip-gdb :

git clone git@git.sys.aics.riken.jp:software/PIP-gdb
cd PIP-gdb
./build.sh --prefix=<PIP_GDB_INSTALL_DIR> --with-glibc-libdir=<GLIBC_INSTALL_DIR>/lib --with-pip=<PIP_INSTALL_DIR>

* Test programs
  A number of test programs can be found in the PIP/test directory.

* To compile, link, and run your PiP programs
  - PiP root process (spawning PiP tasks)
    must be linked with the PiP library and must specify the link
    option as follows if you have the patched GLIBC,

	--dynamic-linker=<GLIBC_INSTALL_DIR>/lib/ld-2.17.so

    Once you specify this option, PiP root process uses the patched
    GLIBC libraries, no matter how LD_LIBRARY_PATH is
    specified. Remember that the other SOLIBs are already copied
    (indeed, symbolic links) into that directory by the "piplnlibs"
    command (see above). The PiP root process is not required to be
    PIE.

  - PiP task (spawned by PiP root process)
    must be compiled with "-pie -pthread", must be linked with "-fpie
    -rdynamic -pthread" options. PiP task programs are not required to be
    linked with the PiP library. Thus programs to be ran as PiP tasks
    are not required to modify their source code. Since PiP root and
    PiP task(s) share the same (virtual) address space and ld-linux.so
    is already loaded by PiP root, PiP tasks use the patched GLIBC.

  - Running PiP programs
    Consult EXECMODE file located at the same directory with this
    file for details.

  - How to check if PiP programs run under PiP environment
    check if they are shared the same address space by the following
    command,

	cat /proc/<PID>/maps

    Here, <PID> is the pid of the root process or one of the PiP
    tasks.

* To debug your PiP programs

    The following procedure attaches all PiP tasks, which are created
    by same PiP root task, as GDB inferiors.
	$ pip-gdb
	(gdb) set pip-auto-attach on
	(gdb) attach <PID-of-your-PiP-program>

    The attached inferiors can be seen by the following GDB command:
	(gdb) info inferiors 
	  Num  Description              Executable        
	  4    process 6453 (pip 2)     /somewhere/pip-task-2 
	  3    process 6452 (pip 1)     /somewhere/pip-task-1 
	  2    process 6451 (pip 0)     /somewhere/pip-task-0 
	* 1    process 6450 (pip root)  /somewhere/pip-root

    You can select and debug an inferior by the following GDB command:
	(gdb) inferior 2
	[Switching to inferior 2 [process 6451 (pip 0)] (/somewhere/pip-task-0)]

    When an already-attached program calls pip_spawn() and becomes
    a PiP root task, the newly created PiP child tasks aren't attached
    automatically, but you can add empty inferiors and then attach
    the PiP child tasks to the inferiors.
    e.g.
	.... type Control-Z to stop the root task.
	^Z
	Program received signal SIGTSTP, Stopped (user).

	(gdb) add-inferior 
	Added inferior 2
	(gdb) inferior 2
	(gdb) attach 1902

	(gdb) add-inferior 
	Added inferior 3
	(gdb) inferior 3
	(gdb) attach 1903

	(gdb) add-inferior 
	Added inferior 4
	(gdb) inferior 4
	(gdb) attach 1904

	(gdb) info inferiors 
	  Num  Description              Executable        
	* 4    process 1904 (pip 2)     /somewhere/pip-task-2 
	  3    process 1903 (pip 1)     /somewhere/pip-task-1 
	  2    process 1902 (pip 0)     /somewhere/pip-task-0 
	  1    process 1897 (pip root)  /somewhere/pip-root

    You can write the "set pip-auto-attach on" setting in your $HOME/.gdbinit
    file.  In that case, you can attach all relevant PiP tasks by:
	$ pip-gdb -p <PID-of-your-PiP-program>

* FAQ on installation

  - After installation, any commands aborted with SIGSEGV

    This can happen when you specify LD_PRELOAD to include the installed
    PiP library. The LD_PRELOAD environment must be specified only when
    running PiP program with the "process:preload" running mode. Please
    consult the EXECMODE file at the same directory with this file.

* Documents
  PiP documents are generated by using doxygen.

  - Man pages
    Man pages will be installed at <PIP_INSTALL_DIR>/share/man.

        man -M <PIP_INSTALL_DIR>/share/man libpip

  - HTML
    HTML documents will be installed at <PIP_INSTALL_DIR>/doc/pip.

* MPI with PiP
  Currently, we are working with ANL to develop MPICH using PiP. This
  repository, located at ANL, is not yet open to public at the time of
  writing.


  Enjoy !

  Atsushi Hori <ahori@riken.jp>
  2018 May 25