Support custom build script

[asavoy]

Adds a build_script variable to allow specifying a custom build script.

This is intended as an escape hatch for use cases where the default build script is not suitable. An example is provided in examples/build.sh.

I've opened a PR to get a sense of whether there is interest in this. If so, I'm open to changes and suggestions (perhaps more documentation and a simpler build example).

[raymondbutcher]

Hi @asavoy. I like this, thanks for submitting it.

Could you please use the example to create a new test under the tests directory? That will serve as a full example and show usage, as well as allow me to try it out.

I think the references to Terraform External Data Sources and Terraform External Program Protocol in your example script are wrong because 1) it's base 64 encoded and 2) it's being passed in as an argument, neither of which are part of Terraform's specs. It's just how hash.py happens to be passing data into build.py.

[asavoy]

@raymondbutcher

Thanks for the corrections, I've changed accordingly.
I've turned the example into a test. Let me know if you have any further feedback.

[raymondbutcher]

I've made a branch https://github.com/claranet/terraform-aws-lambda/compare/ray/custom-build-script with a couple of tweaks and fixes. Please take a look and let me know what you think. I would have tried to push it to your branch (not sure if I have permissions or not) but it required rebasing to fix the merge conflicts and I didn't want to do a git push -f on someone else's branch :)

I like this feature. I am also thinking that separately we should add this Docker behaviour to build.py and have a flag like var.build_with_docker so that this fairly common use case can be handled without a custom build script (but still keep the option for using a custom build script).

[raymondbutcher]

One thing we need to solve is that #31 changes hash.py from:

build_command = "{build_script} {build_data}".format(...)

to:

build_command = "python {build_script} {build_data}".format(...)

So it will use whichever version of Python is running, in a way that works in both Linux and Windows.

We don't want to run all custom build scripts with Python, so python could be moved into the build script argument. However, the build script argument is used when generating the hash. Do we need 2 arguments, build_script and build_command or build_interpreter? That doesn't seem elegant.

[asavoy]

@raymondbutcher Cool, thanks for looking into it so quickly! Appreciate the time you've taken to get it working under Linux.

My notes so far:

• I've tested the new branch and it works correctly (I'm running off macOS)

• I've reset my branch to match yours

• base64 -D must be macOS-specific, but base64 --decode is fine

• Regarding removing default args from the build script, I've updated the Usage section in the script accordingly. Also fixed a typo near there.

• I think first-class support for Docker builds is a great idea. I think it is difficult to use this module without it, since so many useful Python packages need native extensions. Perhaps it might make a nice successor to this feature.

• Regarding the change to build_command for Windows support. I agree that adding a parameter like build_interpreter isn't elegant. Are we sure that changing to python build.py is necessary? If Python is correctly installed, build.py will execute properly in Windows. At least that's the case when using the command prompt on my Windows machine.

[raymondbutcher]

I'm leaning towards:

module "build_with_single_shell_script" {
  ...
  build_command = "${path.module}/custom-build.sh '$filename' '$source' '$runtime'"
  build_paths   = ["${path.module}/custom-build.sh"]
  ...
}

module "build_with_python" {
  ...
  build_command = "python ${path.module}/custom-build/build.py --filename='$filename' --source='$source' --runtime='$runtime'"
  build_paths   = ["${path.module}/custom-build", "${path.module}/other-dependencies"]
  ...
}

module "build_with_make" {
  ...
  build_command = "make --directory=${path.module}/custom-build filename='$filename' source='$source' runtime='$runtime'"
  build_paths   = ["${path.module}/custom-build"]
  ...
}

Notes:

• hash.py replaces $filename with the calculated filename, plus $source and $runtime. Your example script would no longer require base64 and jq as it could just use positional arguments.
• it requires specifying the build source files/directories separate to the command (so the files can be hashed and lambda package rebuilt whenever the build scripts change) but I think this might be necessary if it's going to be flexible
• it supports hashing files or directories if the build script uses multiple files

[raymondbutcher]

Hi @asavoy I've submitted #34 which rebased your branch and fixed the conflicts, then implemented the build_command/build_paths changes I suggested here on top of that. Would you mind taking a look before I merge it?

[asavoy]

Hi @raymondbutcher, unfortunately I don't expect to have time to review anything for at least a few weeks due to work commitments and vacation. Don't let me hold you back, but just wanted to clarify when I'll next be able to return to it. Thanks for your efforts in integrating the feature.

[raymondbutcher]

No worries. I've merged the other one just now. Thanks again!