Protobuf support

This uses tree-sitter-proto to parse the Procol Buffer schema file
for Protobuf extractor.

In addition, since `proto_library` typically requires accompanying
rules, I've added `secondary_rules` map, which will generate
rules within the same `BUILD.bazel` (as opposed to relying on macros).
The dependency relationship between the main target and the secondaries
can be specified using `extra_key_to_list` as
`"deps": [":${name}"]`.

I've also refactored the BUILD generation function so we can actually
write a unit test, AND I've also added an example Bazel repository.

.github/ci_scripts/integration_test.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+set -o errexit  # abort on nonzero exitstatus
+set -o nounset  # abort on unbound variable
+set -o pipefail # don't hide errors within pipes
+
+cd example
+TOOLING_WORKING_DIRECTORY=/tmp/bzl-gen-build source build_tools/lang_support/create_lang_build_files/regenerate_protos_build_files.sh
+bazel test ...

.github/workflows/test.yml

@@ -12,8 +12,19 @@ jobs:
         include:
           - os: macos-latest
             platform: macos
+            jobtype: 1
+          - os: macos-latest
+            platform: macos
+            jobtype: 3
+          - os: ubuntu-latest
+            platform: linux
+            jobtype: 1
           - os: ubuntu-latest
             platform: linux
+            jobtype: 2
+          - os: ubuntu-latest
+            platform: linux
+            jobtype: 3
     steps:
       - uses: actions/checkout@v2
       - uses: actions-rs/toolchain@v1
@@ -33,7 +44,18 @@ jobs:
           distribution: temurin
           java-version: 8
           cache: sbt
-      - run: cd crates; cargo test --all-features
-      - run: cd crates; cargo test
-      - run: cd language_generators/scala-defref-extractor; ./sbt "test; scalafmtCheckAll"
-      - run: ./prepare_all_apps.sh
+      - name: Bazel cache
+        uses: actions/cache@v3
+        with:
+          path: "/home/runner/.cache/bazel"
+          key: bazel-${{ hashFiles('**/.bazelrc', '**/.bazeliskrc', '**/WORKSPACE', '**/WORKSPACE.bazel', '**/MODULE.bazel') }}
+      - if: ${{ matrix.jobtype == 1 }}
+        run: cd crates; cargo test --all-features
+      - if: ${{ matrix.jobtype == 1 }}
+        run: cd crates; cargo test
+      - if: ${{ matrix.jobtype == 2 }}
+        run: cd language_generators/scala-defref-extractor; ./sbt "test; scalafmtCheckAll"
+      - if: ${{ matrix.jobtype == 3 }}
+        run: |
+          ./prepare_all_apps.sh
+          .github/ci_scripts/integration_test.sh

.gitignore

@@ -7,3 +7,5 @@
 
 .vscode
 .metals
+
+bazel-*

DEVELOPING.md

@@ -0,0 +1,100 @@
+# Building
+
+Run this first to build everything.
+
+```bash
+./prepare_all_apps.sh
+```
+
+# Testing
+
+Export `REPO_PATH` to be the path of the bazel repo your working on here...
+
+```bash
+# Pick jvm or python
+
+ECOSYSTEM=jvm
+ECOSYSTEM=python
+ECOSYSTEM=protos
+
+cargo run --bin bzl_gen_build_driver --release -- \
+  --input-path build_tools/lang_support/create_lang_build_files/bazel_${ECOSYSTEM}_modules.json \
+  --working-directory $REPO_PATH \
+  --cache-path ~/.cache/bazel_codegen \
+  extract \
+  --extractor protos:/tmp/bzl-gen-build/protos-entity-extractor \
+  --extractor java:/tmp/bzl-gen-build/java-entity-extractor \
+  --extractor scala:/tmp/bzl-gen-build/scala-entity-extractor \
+  --extractor python:/tmp/bzl-gen-build/python-entity-extractor \
+  --extracted-mappings /tmp/extracted_mappings.${ECOSYSTEM}.json
+
+cargo run --bin bzl_gen_build_driver  --release -- \
+  --input-path build_tools/lang_support/create_lang_build_files/bazel_${ECOSYSTEM}_modules.json \
+  --working-directory $REPO_PATH \
+  --cache-path ~/.cache/bazel_codegen \
+  extract-defs \
+  --extracted-mappings /tmp/extracted_mappings.${ECOSYSTEM}.json \
+  --extracted-defs /tmp/extracted_defs.${ECOSYSTEM}.json
+
+cargo run --bin bzl_gen_build_driver --release -- \
+  --input-path build_tools/lang_support/create_lang_build_files/bazel_${ECOSYSTEM}_modules.json \
+  --working-directory $REPO_PATH \
+  --cache-path ~/.cache/bazel_codegen \
+  build-graph \
+  --extracted-mappings /tmp/extracted_mappings.${ECOSYSTEM}.json \
+  --extracted-defs /tmp/extracted_defs.${ECOSYSTEM}.json \
+  --graph-out /tmp/graph_data.${ECOSYSTEM}.json
+
+cargo run --bin bzl_gen_build_driver --release -- \
+  --input-path build_tools/lang_support/create_lang_build_files/bazel_${ECOSYSTEM}_modules.json \
+  --working-directory $REPO_PATH \
+  --cache-path ~/.cache/bazel_codegen \
+  print-build \
+  --graph-data /tmp/graph_data.${ECOSYSTEM}.json
+```
+
+Shared API type from languages:
+
+```javascript
+    {
+        # A remote label @pip_... or local repo relative path src/a/b/c
+        "label_or_repo_path": ""
+
+        # List of objects defined in this file/target
+        "defs": [],
+
+        # List of references to other things defined in this target
+        "refs": [],
+
+        # List of commands that can be supplied to bzl_build_gen
+        # These are for special cases
+        #
+        # `ref` Add a manual reference to the specified def even though it didn't exist in this file
+        # `unref` Remove a reference seen in this file
+        # `def` Add a manual definition in this file on something that isn't visible to the parsers of the file
+        # `undef` Remove a definition as seen in this file
+        # `runtime_ref` Add a runtime dependency in this file. This is a colored edge that can be upgraded to a ref if another file or something in the file uses it.
+        #
+        #    value(SrcDirective::Ref, tag("ref")),
+        #    value(SrcDirective::Unref, tag("unref")),
+        #    value(SrcDirective::Def, tag("def")),
+        #    value(SrcDirective::Undef, tag("undef")),
+        #    value(SrcDirective::RuntimeRef, tag("runtime_ref")),
+        #    value(SrcDirective::RuntimeUnref, tag("runtime_unref")),
+        bzl_gen_build_commands: []
+    }
+```
+
+
+# Implications on the rules/macros used in the repo..
+
+We expect to be able to supply the following keys generically:
+- `srcs` , list of per language source files
+- `deps` , list of dependencies from the graph
+- `runtime_deps`, list of dependencies used at runtime, but not compile time. (For some languages like python, a macro wrapping py_library can help merge this into deps)
+
+# Testing on the repo side
+
+```bash
+TOOLING_WORKING_DIRECTORY=/tmp/bzl-gen-build ./bazel run build_tools/lang_support/create_lang_build_files:regenerate_jvm_build_files
+```

crates/Cargo.toml

@@ -2,6 +2,7 @@
 
 members = [
     "driver",
+    "protobuf_extractor",
     "python_extractor",
     "python_utilities",
     "shared_types",