Feature: Add Verb-Prelude property to load config files for requests

* Add ability to set, load and parse external config files into verb-var's
* Update tests to validate properties and requests
* Update README.md for user documentation on the feature
* Minor updates to test harness
* Rebase of previous commit with sync-up of origin/main

Makefile

@@ -21,7 +21,7 @@ setup-tests:  ## Install everything required for testing (Python dependencies).
 	python3 --version
 	test -d $(ENV) || python3 -m venv $(ENV)
 	$(ACTIVATE) && \
-	pip install -U pip wheel && \
+	pip install -U pip wheel setuptools && \
 	pip install -r test/requirements-dev.txt
 
 test: ## Run all ERT tests (set SELECTOR to specify only one).

README.md

@@ -446,6 +446,85 @@ To quickly copy the value of a variable into the clipboard, use the keyboard pre
 
 **Note**: Values set with `verb-var` and `verb-set-var` will be lost if the buffer is killed.
 
+### Verb Variables from External Files
+
+To further keep sensitive information safe and separate from Verb `.org` files, Verb variables can also be loaded from either JSON or Emacs Lisp external configuration files. Use the `Verb-Prelude` Org mode property followed by the name of the external configuration file to load. Any file is loaded and applied as a prelude before the request being generated and sent.
+
+**Note**: Files that are GPG or EasyPG encrypted can opened and decrypted automatically by Emacs if configured appropriately. See: [Emacs Auth-source manual](https://www.gnu.org/software/emacs/manual/auth.html) for more information.
+
+The `Verb-Prelude` property may be set at the buffer level (very top of the Org document) or various header levels, and every heading level in the hierarchy with the `Verb-Prelude` property will also be loaded. If the same setting within an environment configuration file is specified at a parent and a child level, then the child will override the parent. This allows more specific or different settings to be done for a lower-level request than at a higher-level. All other file unique variables would be additive to the collection of Verb variables.
+
+Below is an example of configuring external configuration files at top buffer level, and then a different file for *Foobar Development* at lower level 2.
+
+``` org 
+#+Verb-Prelude: /path/to/prod-foobar-env.el.gpg
+
+* Foobar Blog API                    :verb:
+
+template https://foobar-blog-api.org/api/v1
+Accept: application/json
+
+** Foobar Production
+
+get /status
+
+** Foobar Development
+:properties:
+:Verb-Prelude: /path/to/dev-foobar-env.el.gpg
+:end:
+
+template https://foobar-blog-api.org/api/v2
+Accept: application/json
+
+get /status
+```
+
+In the scenario, when sending the `get /status` request for Foobar Development, `prod-foobar-env.el.gpg` is loaded first from the `#+Verb-Prelude:` buffer property. And then `dev-foobar-env.el.gpg` is loaded from properties under *`** Foobar Development`* header. The dev file would override any variables having the same key name between the two environment files. Again, all other variables would be additive. A common example would be that an *api_token* variable would likely be different between the two environments, but *user_name* would be the same for both environments. It would be logical to then have *api_token* variable specified in both, depending on environments of requests. And *user_name* specified in highest level of the Org structure to be shared among Production and Development environments.
+
+For an Emacs Lisp Prelude file, it can run any code found in the `.el` file. (Even to dynamically call and get a temporary Bearer token!) For this reason, a yes-no warning prompt is presented when loading those files unless `verb-suppress-load-unsecure-prelude-warning` is set to non-nil value. This, of course, is another reason to GPG encrypt the configuration files to help prevent configuration poisoning. Here's an example of loading `verb-var`'s using an Emacs Lisp file:
+
+``` elisp
+; prod-foobar-env
+(message "Setting up verb variables!")
+; "email" variable is set from emacs user-mail-address
+(verb-set-var "email" user-mail-address)
+(verb-set-var "global_api_key" "hc33Vzco7TAkMKLNzIVgps7KiKLnUYIWJ7y9T")
+(verb-set-var "account_id" "Kb9SVZXtTVFYzt7rvgYv5WXJS31lh7OT")
+(verb-set-var "zone-ids" '(:example.com "IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL"
+                           :myotherdomain.com "jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3"))
+(verb-set-var "example.com" "IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL")
+(verb-set-var "myotherdomain.com" "jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3")
+```
+
+In above, a couple of advantages to using Elisp Prelude files is shown, such as having comments; dynamically getting email address from Emacs `user-mail-address` variable; and outputting a message, which can be useful for logging and debugging.
+
+For JSON environment file, the values (and even sub-values) are simply set using `verb-set-var` for each key value pair.  Example of JSON configuration (with not real keys or id's):
+
+``` json
+{
+  "email": "[email protected]",
+  "global_api_key": "hc33Vzco7TAkMKLNzIVgps7KiKLnUYIWJ7y9T",
+  "account_id": "Kb9SVZXtTVFYzt7rvgYv5WXJS31lh7OT",
+  "zone-ids":
+    {"example.com": "IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL",
+     "myotherdomain.com": "jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3"
+    }
+}
+```
+
+Results in this for `verb-show-vars` command:
+
+``` env
+email: [email protected]
+global_api_key: hc33Vzco7TAkMKLNzIVgps7KiKLnUYIWJ7y9T
+account_id: Kb9SVZXtTVFYzt7rvgYv5WXJS31lh7OT
+zone-ids: (:example.com IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL :myotherdomain.com jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3)
+example.com: IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL
+myotherdomain.com: jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3
+```
+
+This will result in the same `verb-show-vars` just above for the Elisp example (assuming `user-mail-address` is *"[email protected]"*). Notice, in the example, that `zone-ids` plist key value pairs are also flattened into their own key-value pairs for easy referencing.
+
 ### Last Response
 
 If you wish to access the last response's attributes, use the `verb-last` variable (type: `verb-response`). The following example does this; add it to the ending of your `guide.org` file:

test/env.el

@@ -0,0 +1,2 @@
+(verb-set-var "foo" "hello")
+(verb-set-var "bar" "world")

test/env.json

@@ -0,0 +1,4 @@
+{
+  "foo": "hello",
+  "bar": "world"
+}

test/init.el

@@ -25,6 +25,7 @@
 
 ;; Set up Verb
 (setq verb-auto-kill-response-buffers 3)
+(setq verb-suppress-load-unsecure-prelude-warning t)
 
 ;; Keybindings
 (with-eval-after-load 'org (define-key org-mode-map (kbd "C-c C-r") verb-command-map))

test/test.org

@@ -150,6 +150,16 @@ Content-Type: application/xml
 
 bar2
 {{(verb-part)}}
+** prelude-elisp
+:properties:
+:Verb-Prelude: env.el
+:end:
+get /echo-args?{{(verb-var foo)}}={{(verb-var bar)}}
+** prelude-json
+:properties:
+:Verb-Prelude: env.json
+:end:
+get /echo-args?{{(verb-var foo)}}={{(verb-var bar)}}
 * connection-fail-port
 # Valid host but invalid port
 get http://localhost:1234/test

test/verb-test.el

@@ -2226,6 +2226,15 @@
   (server-test "multipart"
                (should (string= (buffer-string) "OK"))))
 
+(ert-deftest test-server-request-prelude-elisp ()
+  (let ((verb-suppress-load-unsecure-prelude-warning t))
+    (server-test "prelude-elisp"
+               (should (string= (buffer-string) "hello=world")))))
+
+(ert-deftest test-server-request-prelude-json ()
+  (server-test "prelude-json"
+               (should (string= (buffer-string) "hello=world"))))
+
 (ert-deftest test-server-response-big5 ()
   (server-test "response-big5"
                (should (coding-system-equal buffer-file-coding-system 'chinese-big5-unix))

verb.el

@@ -32,6 +32,7 @@
 
 ;;; Code:
 (require 'org)
+(require 'org-element)
 (require 'ob)
 (require 'eieio)
 (require 'subr-x)
@@ -278,6 +279,14 @@ a call to `verb-var'."
   :group :verb
   :type 'boolean)
 
+(defcustom verb-suppress-load-unsecure-prelude-warning nil
+  "When set to a non-nil, suppress warning about loading Elisp Preludes.
+Loading Emacs Lisp (.el) configuration files as a Prelude is potentially
+unsafe, if this setting is nil a warning prompt is shown asking user to allow
+it to be loaded and evaluated.  If non-nil, no warning is shown when loading
+Elisp Prelude external files."
+  :type 'boolean)
+
 (defface verb-http-keyword '((t :inherit font-lock-constant-face
                                 :weight bold))
   "Face for highlighting HTTP methods.")
@@ -889,6 +898,7 @@ Note that the entire buffer is considered when generating the request
 spec, not only the section contained by the source block.
 
 This function is called from ob-verb.el (`org-babel-execute:verb')."
+  (verb-load-prelude-files-from-hierarchy)
   (save-excursion
     (goto-char pos)
     (let* ((verb--vars (append vars verb--vars))
@@ -963,6 +973,10 @@ Once all the request specs have been collected, override them in
 inverse order according to the rules described in
 `verb-request-spec-override'.  After that, override that result with
 all the request specs in SPECS, in the order they were passed in."
+  ;; Load all prelude verb-var's before rest of the spec to be complete, unless
+  ;; specs already exists which means called from ob-verb block and loaded.
+  (unless specs
+    (verb-load-prelude-files-from-hierarchy))
   (let (done final-spec)
     (save-restriction
       (widen)
@@ -991,6 +1005,39 @@ all the request specs in SPECS, in the order they were passed in."
                           "Remember to tag your headlines with :%s:")
                   verb-tag))))
 
+(defun verb-load-prelude-files-from-hierarchy ()
+  "Load all Verb-Prelude's of current heading and up, including buffer level.
+Children with same named verb-vars as parents, will override the parent
+settings."
+  (save-restriction
+    (widen)
+    (save-excursion
+      (let (preludes)
+        (while
+            (progn
+              (let* ((spec (verb-request-spec
+                            :metadata (verb--heading-properties
+                                       verb--metadata-prefix)))
+                     (prelude (verb--request-spec-metadata-get spec
+                                                               "prelude")))
+                (when prelude
+                  (push prelude preludes)))
+              (verb--up-heading)))
+        (let* ((prelude (car (org-element-map (org-element-parse-buffer)
+                                 'keyword
+                               (lambda (keyword)
+                                 (when (string= (upcase (concat
+                                                         verb--metadata-prefix
+                                                         "prelude"))
+                                                (org-element-property
+                                                 :key keyword))
+                                   (org-element-property :value keyword)))))))
+          (when prelude
+            (push prelude preludes)))
+        ;; Lower-level prelude files override same settings in hierarchy
+        (dolist (file preludes)
+          (verb-load-prelude-file file))))))
+
 (defun verb-kill-response-buffer-and-window (&optional keep-window)
   "Delete response window and kill its buffer.
 If KEEP-WINDOW is non-nil, kill the buffer but do not delete the
@@ -1112,6 +1159,38 @@ This affects only the current buffer."
             (yes-or-no-p "Unset all Verb variables for current buffer? "))
     (setq verb--vars nil)))
 
+(defun verb-load-prelude-file (filename)
+  "Load a elisp or json configuration file, FILENAME, into verb variables."
+  (interactive)
+  (save-excursion
+    (let ((file-extension (file-name-extension filename)))
+      (when (member file-extension '("gpg" "gz" "z" "7z"))
+        (setq file-extension (file-name-extension (file-name-base filename))))
+      (cond
+       ((string= "el" (downcase file-extension)) ;; file is elisp
+        (if (or (bound-and-true-p verb-suppress-load-unsecure-prelude-warning)
+                (yes-or-no-p
+                 (concat (format "The file: %s may contain values " filename)
+                         "that may not be safe.\n\nDo you wish to load?")))
+            (load filename)))
+       ((string-match-p "^json.*" (downcase file-extension)) ;; file is json(c)
+        (let* ((file-contents (with-temp-buffer
+                                (insert-file-contents filename)
+                                (set-auto-mode)
+                                (goto-char (point-min))
+                                (comment-kill (count-lines
+                                               (point-min) (point-max)))
+                                (verb--buffer-string-no-properties)))
+               (json-object-type 'plist)
+               (data (json-read-from-string file-contents)))
+          (cl-loop for (k v) on data by #'cddr
+                   do (verb-set-var (substring (symbol-name k) 1) v)
+                   if (and (listp v) (cl-evenp (length v)))
+                   do (cl-loop for (subk subv) on v by #'cddr
+                               do (verb-set-var
+                                   (substring (symbol-name subk) 1) subv)))))
+       (t (user-error "Unable to determine file type for %s" filename))))))
+
 (defun verb-show-vars ()
   "Show values of variables set with `verb-var' or `verb-set-var'.
 Values correspond to variables set in the current buffer.  Return the