go-make

A parallel version of the make tool in Go which uses RPC. Works with basic Makefiles only (see below).

To compare with a sequential execution, we developped a sequential version, in the folder sequential.

The parallel implementation is based on a master-slave architecture. The master parses the Makefile and waits for slaves to ask for tasks. If a rule is ready (whose dependencies are all computed), the master returns it.

The communication relies on the RPC protocol. The master starts a RPC server and exposes two methods:

• giveTask: a slave asks for work, the master returns a ready rule. If there are none, it appends the slave in a waiting list.
• receiveResult: a slave returns the result of a rule (the generated file). If other rules become ready, the master wakes up the waiting slaves and waits for them to ask for work.

Our program does not manage failures. If a slave crashes, the master may not end.

Install

Dependencies:

• Go

To compile the sequential version:

make sequential

For the parallel one:

make

Run locally

Sequential

make run_seq makefile-path-relative-to-tmp

For instance:

make run_seq ../makefiles/8

Master

make run_master makefile-path-relative-to-tmp

For instance:

make run_master ../makefiles/8

Slave

make run_slave1
# Or: make run_slave2

Run on Grid5000

For each script, run it without parameter to show the help.

# Connect to grid5000
ssh <login>@access.grid5000.fr

# Connect to a site
ssh <site>

# Get nodes
oarsub -I -l nodes=TODO,walltime=1 -t deploy

# Clone the project in the NFS (Network File System)
cd ~
git clone https://github.com/Vayel/go-make

# Deploy
cd go-make/scripts
./deploy.sh

Sequential

Example:

# cd ~/go-make/scripts
./sequential.sh ~/go-make/makefiles/1 all ~/go-make/logs/seq.json ~/go-make/logs/nodes/seq.txt

• ~/go-make/logs/seq.json contains the execution time
• ~/go-make/logs/nodes/seq.txt contains the name of the node used for computations.

Parallel

Example:

# cd ~/go-make/scripts
./master.sh ~/go-make/makefiles/1 all ~/go-make/logs/master.json ~/go-make/logs/nodes/master.txt &
./slave.sh 3 ~/go-make/logs ~/go-make/logs/nodes/slaves.txt

• ~/go-make/logs/master.json contains the execution time of the master
• ~/go-make/logs/slave_X.json contains the execution times (total, working and waiting) of the slave X
• ~/go-make/logs/nodes/master.txt contains the name of the master node
• ~/go-make/logs/nodes/slaves.txt contains the name of the slave nodes

Run tests

Example:

# cd ~/go-make/scripts
# use Python 3.4
python3 test.py 1 2 1 2 ~/go-make/makefiles/1

• ~/measures.json contains the measures
• ~/go-make/logs contains the same elements as in Parallel section

Plot measures

Gnuplot is required.

Example:

# cd ~/go-make/scripts
./plot.sh ~/measures.json .

Supported Makefiles

This tool handles only basic Makefiles. All rules must look like:

target: dep1 dep2 dep3 ...
	instruction1
	instruction2
	...

All rules must create a file of the same name in the working directory. If a rule needs an external file (which is not generated by a dependency), for instance some code to be compiled, we consider it has been transfered beforehand. The same applies for the instructions: all commands called by the rules must be available on the machines.

A rule may not have any dependency (it is mandatory for at least one in fact). The instructions must be directly interpretable by the program sh. In particular, our program does not manage variables in Makefiles.