Skip to content

Running the Example Pipelne

Jump to bottom
ccblazer44 edited this page Apr 29, 2017 · 8 revisions

  1. Clone this repository 'git clone https://github.com/acgt-tax-consultants/orchard.git'.

  2. Move into the newly created orchard folder.

$ cd orchard
  1. Install all the required dependancies.
$ pip install -e .
  1. Move into the folder with the example pipeline,
$ cd example
  1. Our example pipeline already has a link file for you to use, so while in the example folder run the orchard template command to create a config file based of this link file.
$ orchard template link.yaml
  1. Now that orchard generated a config file for us, we need to fill it in. Open config.yaml in your text editor of choice and fill in all the required fields, we will ignore all the optional fields.

Our example modules just write text to a file, we will pass the output from each module on as the input for the next module. We will use simple integers for the digit argument. Here is what it should look like:

Save config.yaml.

  1. Now we will run orchard build to validate our config file and get ready to run.
orchard build link.yaml config.yaml

This will generate our first branch, there should be a '1' folder inside example now. It will also generate our Luigi file, there should be a file named 'test.py' inside example now.

  1. In order to run orchard, the Luigi Daemon must be running. Open a new terminal window and run the Luigi Daemon.
luigid

Keep luigid running for the rest of this tutorial.

  1. Go back to the terminal window in the example folder, and run orchard.
orchard launch test.py ModuleThree

We just successfully ran our example orchard pipeline. You can see the outputs from each module are now in the 1 folder.

  1. Now we will branch the pipeline. Open up config.yaml in your text editor again. We will branch on ModuleTwo. Change the 'digit' argument in ModuleTwo from 2 to 20 and save config.yaml. Now we need to build orchard again.
orchard build link.yaml config.yaml

Since this is a new branch, the next branch folder is created, '2'. Next, run orchard again.

orchard launch test.py ModuleThree

You can see the outputs from our new successful branch run in the 2 folder. You can see the output from ModuleOne, 'b.txt', is symlinked to the output from the previous branch and was not regenerated. You can also see in the Luigi summary that ModuleOne was not executed.

  1. We can also take a look at what luigid is doing for us. Open your browser and go to 'http://localhost:8082/'. Here we see a visualization of our pipeline.

Thank you for trying orchard!

Clone this wiki locally