diff --git a/Makefile b/Makefile index cefb734..97b0bf4 100644 --- a/Makefile +++ b/Makefile @@ -21,7 +21,7 @@ setup-tests: ## Install everything required for testing (Python dependencies). python3 --version test -d $(ENV) || python3 -m venv $(ENV) $(ACTIVATE) && \ - pip install -U pip wheel && \ + pip install -U pip wheel setuptools && \ pip install -r test/requirements-dev.txt test: ## Run all ERT tests (set SELECTOR to specify only one). diff --git a/README.md b/README.md index ecbb500..6ae6863 100644 --- a/README.md +++ b/README.md @@ -446,6 +446,80 @@ To quickly copy the value of a variable into the clipboard, use the keyboard pre **Note**: Values set with `verb-var` and `verb-set-var` will be lost if the buffer is killed. +### Verb Variables from External Files + +To further keep sensitive information safe and separate from Verb `.org` files, Verb variables can also be loaded from either JSON or Emacs Lisp external configuration files. Use the `Verb-Prelude` Org mode property followed by the name of the external configuration file to load. Any file is loaded and applied as a prelude before the request being generated and sent. + +**Note**: Files that are GPG or EasyPG encrypted can opened and decrypted automatically by Emacs if configured appropriately. See: [Emacs Auth-source manual](https://www.gnu.org/software/emacs/manual/auth.html) for more information. + +The `Verb-Prelude` property may be set at the buffer level (very top of the Org document) as an Org mode file `keyword` or various heading levels, and every heading level in the hierarchy with the `Verb-Prelude` property will also be loaded. If the same setting within an environment configuration file is specified at a parent and a child level, then the child will override the parent. This allows more specific or different settings to be done for a lower-level request than at a higher-level. All other file unique variables would be additive to the collection of Verb variables. Because of the specific application usage of `Verb-Prelude`, the Org mode variable `org-use-property-inheritance` is not used for `Verb-Prelude` properties. They are always inherited. + +Below is an example of configuring external configuration files at top buffer level, and then a different file for *Foobar Development* at lower level 2. + +``` org +#+Verb-Prelude: /path/to/prod-foobar-env.el.gpg + +* Foobar Blog API :verb: +template https://foobar-blog-api.org/api/v1 +Accept: application/json + +** Foobar Production +get /status + +** Foobar Development +:properties: +:Verb-Prelude: /path/to/dev-foobar-env.el.gpg +:end: + +get /status +``` + +In the scenario, when sending the `get /status` request for Foobar Development, `prod-foobar-env.el.gpg` is loaded first from the `#+Verb-Prelude:` buffer property. And then `dev-foobar-env.el.gpg` is loaded from properties under *`** Foobar Development`* header. The dev file would override any variables having the same key name between the two environment files. Again, all other variables would be additive. A common example would be that an *api_token* variable would likely be different between the two environments, but *user_name* would be the same for both environments. It would be logical to then have *api_token* variable specified in both, depending on environments of requests. And *user_name* specified in highest level of the Org structure to be shared among Production and Development environments. + +For an Emacs Lisp Prelude file, it can run any code found in the `.el` file, (even to dynamically call and get a temporary `Bearer` token!). For this reason, a yes-no warning prompt is presented when loading those files unless `verb-suppress-load-unsecure-prelude-warning` is set to non-nil value. This, of course, is another reason to GPG encrypt the configuration files to help prevent configuration poisoning. Here's an example of loading `verb-var`'s using an Emacs Lisp file: + +``` elisp +; prod-foobar-env +(message "Setting up verb variables!") +; "email" variable is set from emacs user-mail-address +(verb-set-var "email" user-mail-address) +(verb-set-var "global_api_key" "hc33Vzco7TAkMKLNzIVgps7KiKLnUYIWJ7y9T") +(verb-set-var "account_id" "Kb9SVZXtTVFYzt7rvgYv5WXJS31lh7OT") +(verb-set-var "zone-ids" '(:example.com "IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL" + :myotherdomain.com "jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3")) +(verb-set-var "example.com" "IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL") +(verb-set-var "myotherdomain.com" "jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3") +``` + +In above, a couple of advantages to using Emacs Lisp Prelude files is shown, such as having comments; dynamically getting email address from Emacs `user-mail-address` variable; and outputting a message, which can be useful for logging and debugging. + +For JSON environment file, the values (and even sub-values) are simply set using `verb-set-var` for each key value pair. Example of JSON configuration: + +``` json +{ + "email": "email@example.com", + "global_api_key": "hc33Vzco7TAkMKLNzIVgps7KiKLnUYIWJ7y9T", + "account_id": "Kb9SVZXtTVFYzt7rvgYv5WXJS31lh7OT", + "zone-ids": { + "example.com": "IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL", + "myotherdomain.com": "jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3" + } +} +``` + +Results in this for `verb-show-vars` command: + +``` env +email: email@example.com +global_api_key: hc33Vzco7TAkMKLNzIVgps7KiKLnUYIWJ7y9T +account_id: Kb9SVZXtTVFYzt7rvgYv5WXJS31lh7OT +zone-ids: (:example.com IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL :myotherdomain.com jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3) +example.com: IRMTk8T0RKgFxL3bL8KWv5FDDVtoe1VL +myotherdomain.com: jL0OeCs9XOSsLUfCUKCRRSHKsQpM5WB3 +``` + +This will result in the same `verb-show-vars` just above for the Emacs Lisp example (assuming `user-mail-address` is *"email@example.com"*). Notice, in the example, that `zone-ids` plist key value pairs are also flattened into their own key-value pairs for easy referencing. + ### Last Response If you wish to access the last response's attributes, use the `verb-last` variable (type: `verb-response`). The following example does this; add it to the ending of your `guide.org` file: diff --git a/test/env.el b/test/env.el new file mode 100644 index 0000000..08416f8 --- /dev/null +++ b/test/env.el @@ -0,0 +1,2 @@ +(verb-set-var "foo" "hello") +(verb-set-var "bar" "world") diff --git a/test/env.json b/test/env.json new file mode 100644 index 0000000..93868db --- /dev/null +++ b/test/env.json @@ -0,0 +1,4 @@ +{ + "foo": "hello", + "bar": "world" +} diff --git a/test/init.el b/test/init.el index 34fca58..f7d25ff 100644 --- a/test/init.el +++ b/test/init.el @@ -25,6 +25,7 @@ ;; Set up Verb (setq verb-auto-kill-response-buffers 3) +(setq verb-suppress-load-unsecure-prelude-warning t) ;; Keybindings (with-eval-after-load 'org (define-key org-mode-map (kbd "C-c C-r") verb-command-map)) diff --git a/test/test.org b/test/test.org index a597188..c219ede 100644 --- a/test/test.org +++ b/test/test.org @@ -150,6 +150,16 @@ Content-Type: application/xml bar2 {{(verb-part)}} +** prelude-elisp +:properties: +:Verb-Prelude: env.el +:end: +get /echo-args?{{(verb-var foo)}}={{(verb-var bar)}} +** prelude-json +:properties: +:Verb-Prelude: env.json +:end: +get /echo-args?{{(verb-var foo)}}={{(verb-var bar)}} * connection-fail-port # Valid host but invalid port get http://localhost:1234/test diff --git a/test/verb-test.el b/test/verb-test.el index a4527e3..9933e14 100644 --- a/test/verb-test.el +++ b/test/verb-test.el @@ -2226,6 +2226,15 @@ (server-test "multipart" (should (string= (buffer-string) "OK")))) +(ert-deftest test-server-request-prelude-elisp () + (let ((verb-suppress-load-unsecure-prelude-warning t)) + (server-test "prelude-elisp" + (should (string= (buffer-string) "hello=world"))))) + +(ert-deftest test-server-request-prelude-json () + (server-test "prelude-json" + (should (string= (buffer-string) "hello=world")))) + (ert-deftest test-server-response-big5 () (server-test "response-big5" (should (coding-system-equal buffer-file-coding-system 'chinese-big5-unix)) diff --git a/verb.el b/verb.el index bba3c6e..b7e59c8 100644 --- a/verb.el +++ b/verb.el @@ -32,6 +32,7 @@ ;;; Code: (require 'org) +(require 'org-element) (require 'ob) (require 'eieio) (require 'subr-x) @@ -278,6 +279,14 @@ a call to `verb-var'." :group :verb :type 'boolean) +(defcustom verb-suppress-load-unsecure-prelude-warning nil + "When set to a non-nil, suppress warning about loading Elisp Preludes. +Loading Emacs Lisp (.el) configuration files as a Prelude is potentially +unsafe, if this setting is nil a warning prompt is shown asking user to allow +it to be loaded and evaluated. If non-nil, no warning is shown when loading +Elisp Prelude external files." + :type 'boolean) + (defface verb-http-keyword '((t :inherit font-lock-constant-face :weight bold)) "Face for highlighting HTTP methods.") @@ -889,6 +898,7 @@ Note that the entire buffer is considered when generating the request spec, not only the section contained by the source block. This function is called from ob-verb.el (`org-babel-execute:verb')." + (verb-load-prelude-files-from-hierarchy) (save-excursion (goto-char pos) (let* ((verb--vars (append vars verb--vars)) @@ -963,6 +973,10 @@ Once all the request specs have been collected, override them in inverse order according to the rules described in `verb-request-spec-override'. After that, override that result with all the request specs in SPECS, in the order they were passed in." + ;; Load all prelude verb-var's before rest of the spec to be complete, unless + ;; specs already exists which means called from ob-verb block and loaded. + (unless specs + (verb-load-prelude-files-from-hierarchy)) (let (done final-spec) (save-restriction (widen) @@ -991,6 +1005,39 @@ all the request specs in SPECS, in the order they were passed in." "Remember to tag your headlines with :%s:") verb-tag)))) +(defun verb-load-prelude-files-from-hierarchy () + "Load all Verb-Prelude's of current heading and up, including buffer level. +Children with same named verb-vars as parents, will override the parent +settings." + (save-restriction + (widen) + (save-excursion + (let (preludes) + (while + (progn + (let* ((spec (verb-request-spec + :metadata (verb--heading-properties + verb--metadata-prefix))) + (prelude (verb--request-spec-metadata-get spec + "prelude"))) + (when prelude + (push prelude preludes))) + (verb--up-heading))) + (let* ((prelude (car (org-element-map (org-element-parse-buffer) + 'keyword + (lambda (keyword) + (when (string= (upcase (concat + verb--metadata-prefix + "prelude")) + (org-element-property + :key keyword)) + (org-element-property :value keyword))))))) + (when prelude + (push prelude preludes))) + ;; Lower-level prelude files override same settings in hierarchy + (dolist (file preludes) + (verb-load-prelude-file file)))))) + (defun verb-kill-response-buffer-and-window (&optional keep-window) "Delete response window and kill its buffer. If KEEP-WINDOW is non-nil, kill the buffer but do not delete the @@ -1112,6 +1159,42 @@ This affects only the current buffer." (yes-or-no-p "Unset all Verb variables for current buffer? ")) (setq verb--vars nil))) +(defun verb-load-prelude-file (filename) + "Load a elisp or json configuration file, FILENAME, into verb variables." + (interactive) + (save-excursion + (let ((file-extension (file-name-extension filename))) + (when (member file-extension '("gpg" "gz" "z" "7z")) + (setq file-extension (file-name-extension (file-name-base filename)))) + (cond + ((string= "el" (downcase file-extension)) ;; file is elisp + (if (or (bound-and-true-p verb-suppress-load-unsecure-prelude-warning) + (yes-or-no-p + (concat (format "The file: %s may contain values " filename) + "that may not be safe.\n\nDo you wish to load?"))) + (load-file filename))) + ((string-match-p "^json.*" (downcase file-extension)) ;; file is json(c) + (let* ((file-contents + (with-temp-buffer + (insert-file-contents filename) + (set-auto-mode) + (goto-char (point-min)) + ;; If a modern json / js package not installed, then comments + ;; cannot be removed or supported. Also, not likely to have + ;; json comments if this is the case. + (when comment-start + (comment-kill (count-lines (point-min) (point-max)))) + (verb--buffer-string-no-properties))) + (json-object-type 'plist) + (data (json-read-from-string file-contents))) + (cl-loop for (k v) on data by #'cddr + do (verb-set-var (substring (symbol-name k) 1) v) + if (and (listp v) (cl-evenp (length v))) + do (cl-loop for (subk subv) on v by #'cddr + do (verb-set-var + (substring (symbol-name subk) 1) subv))))) + (t (user-error "Unable to determine file type for %s" filename)))))) + (defun verb-show-vars () "Show values of variables set with `verb-var' or `verb-set-var'. Values correspond to variables set in the current buffer. Return the