Major refactoring?

[alphapapa]

@thblt @yuhan0

I wonder if we should do a survey of Outshine's features and consider which ones are still necessary to implement in this package.

For example, outline cycling in non-Org, non-outline buffers is already handled by outline-minor-mode, so I'm not sure that we need to implement it again in Outshine. ISTM that, at most, we might need to implement some convenience functions on top of outline-minor-mode.

For issuing Org commands in non-Org buffers, maybe we could move to something like the not-org prototype library, which would relieve us of having to reimplement Outshine versions of Org commands (if not all, then at least some of them).

The bottom line, of course, is that Emacs and Org have changed a lot since Outshine was first written. I don't have a good grasp of where its functionality fits and overlaps now, so I'm not sure where we need to go from here.

What do you think? Thanks.

[yuhan0]

As far as I know, Outline-minor-mode does not do cycling, only the various showing and hiding commands that are much less friendly to remember than simply hitting TAB multiple times.
"Convenience functions on top of outline-minor-mode" is the whole point of the outline-magic bits in Outshine, so I don't quite understand that part.

I agree about removing unnecessary features - for example there are quite a few orphan functions that aren't referenced by anything else, and "hidden line cookies" seems to be quite buggy and a much better case for overlays.

About the use-outorg family of commands, perhaps it's better for someone who regularly uses those features to comment on them - my personal use is limited to outshine-todo.

[yuhan0]

Also, I think the current method of piecing together and mutating a dozen global variables in outshine-calc- functions and then referencing them all over the place is rather messy and could stand to be refactored for cleanliness/modularity.

For reference here's some of the inline notes I took of the minor mode activation :

  ;; Compute basic outline regular expressions.
  ;; The following function calls initialize buffer-local variables which are
  ;; used in subsequent outshine-calc-* functions.
  (outshine-set-outline-regexp-base)
  ;; Initializes the variables:
  ;; - outshine--enforce-no-comment-padding-p
  ;; - outshine--regexp-base
  (outshine-normalize-regexps)
  ;; Initializes the variables:
  ;; - outshine--normalized-comment-start
  ;; - outshine--normalized-comment-end
  ;; - outshine--normalized-outline-regexp-base
  ;; - outshine--normalized-regexp-prefix

  ;; outshine-calc-outline-regexp is dependent on:
  ;; - outshine-regexp-outcommented-p - customizable defvar, should depend on prog-mode (Emacs 24)
  ;; - outshine--normalized-comment-start (thru -calc-comment-region-starter)
  ;; - outshine--enforce-no-comment-padding-p
  ;; - outshine--normalized-outline-regexp-base
  (let ((out-regexp (outshine-calc-outline-regexp)))
 ...

[alphapapa]

As far as I know, Outline-minor-mode does not do cycling, only the various showing and hiding commands that are much less friendly to remember than simply hitting TAB multiple times. "Convenience functions on top of outline-minor-mode" is the whole point of the outline-magic bits in Outshine, so I don't quite understand that part.

You may be right. What I'm trying to get at is, Outshine seems to do many things, and I don't even understand them all, and some of it seems to depend on outorg. So I think it would be helpful for us if we, say, refactored its functionality into independent parts (and maybe separate files, as well). If we better define what it's supposed to do, and group it into relevant parts, we can better understand it and improve it.

[yuhan0]

See the most recent PR #55, which should help a little in the way of grouping independent functionality.
Basically most of the work of separating the outorg commands into a separate file had already been done back in bd43f2b, except it seems to have been only partially completed, leaving duplicated functions and commented regions which I just cleaned up. (reducing the size of the main outshine.el file down to a more manageable sounding 2500 LOC)

[alphapapa]

@thblt Looking at alphapapa/outorg#9, and considering how closely these packages are interrelated, I'm wondering if we should make a GitHub "organization" to hold these repos. What do you think?

[thblt]

I've been postponing asking this exact question for some time now. I only use Outshine to manage Org-like headings in code, but it does a lot more, especially when combined with outorg and navi. From what I understand:

• outshine does the heading stuff, and may (I don't know) duplicate some Outline functionality. Features that are, for what I understand, provided only by Outshine are, at least: fontification, cycling and contextual keybindings.
• outorg allows to edit a source code buffer comments as an org-mode file.
• navi creates an indirect buffer as a remote control.

A minor potential issue with Outshine, IMO, is that we still hijack Outshine somehow. Programming major modes configure it to consider classes, functions, etc, as structure; Outshine modifies this config to use its heading system instead.

I've been wondering about the possibility of making Outshine an autonomous, generic, structural editor for various text formats. It could then:

• provide pluggable structure detection formats, of which the current "headings" system could be just an example; this would allow to have the same package manage various types of structural elements (headings, classes, functions)
• on this basis, enable fontification/fringe marking of those elements;
• add folding, cycling, etc.

This could be done by taking inspiration from Origami, which already has pluggable parsers (but no concept of (1) having more than one parser per buffer or (2) parsers being of different "types). (Or even by actually extending Origami, which seems unmaintained)

From this, navi functionality could be trivially replicated.

(In this perspective, Outshine becomes a complete replacement of Outline and whichever code folding system people use)

[Fuco1]

I just set out to make something on top of outline which will fold/show/unfold just by hitting tab like in org-mode. The default outline UX is horrendous... and one of the reasons org exists in the first place.

If you want to implement that I can maybe volunteer to try to hack on it if we can make it fit nicely with the rest of the package.

[tj64]

Here some historical remarks (as far as I remember):

all those outline extensions that went into outshine did things outline-(minor-mode) could not do, but maybe it improved in the maintime and some of these functions are obsolete duplicates now?

hidden line cookies
there is acually a repository (tj64/org-hlc), and I think that library was in a working state.
So all hidden line cookie functonality could be removed from outshine, anybody interested could use
org-hlc instead. I wanted to do that (remove it) but in the end did not make it.
There is an overlay based implementation on github too, forgot the name unfortunately, from the author of poporg.el. Its very nice. But overlays have their limits with bigger files, things might get very slow.

use outorg
when I wrote outshine, quite a few "org-mode" feature requests came in, and I wanted some features from org-mode too. Reusing org code if often difficult to impossible, rewriting not an option, so using outorg from outshine seemed the way to go, and it works quite well, except when permanent org files are expected (e.g. clocking).

By script I created templates for almost all org command, implemented a few, and left the others for users interested in that functionality. If all org commands that make sense in programming mode would
be implemented with "outshine-use-outorg", outshine headers would be just as 'smart' as org headers.

Its all prepared, but somebody would need to do the work.
And for clocking etc one would need new concepts, like a permanenet associated org file or so.

WRT dependency, oushines works without outorg (when outshine-use-outorg is omitted), but outorg needs outshine. Both libraries are a bit complex, so merging is maybe not such a good idea?
I wanted outshine-use-outorg and could not find a way to avoid mutual dependency.

imenu
I never used, but implemented due to feature request, and remember it worked

other ideas
I remember I had some ideas started, but did not really implement, but I don't remember which unfortunately

@thblt
"making Outshine an autonomous, generic, structural editor for various text formats."
Wow, that sounds really ambitious, but interesting too

Cheers
Thorsten

[yuhan0]

I've been giving this a bit of thought too - what do you all think of refactoring the current system of generating the outline-regexp into a single outshine-spec local variable, such that it allows for custom overrides based on major mode?

Here's the proposed API:

(defcustom outshine-default-spec-alist
  '((base-char . ?*)
    (commented . t)
    (padding . 1))
  "Variables used to calculate the outline-regexp. To override
these settings for particular major modes, see `outshine-custom-spec-alist'.")

(defcustom outshine-custom-spec-alist
  '((markdown-mode (commented . nil)
                   (base-char . ?#))
    (emacs-lisp-mode (base-char . ?\;)
                     (padding . nil))
    (haskell-mode (padding . nil))
    (text-mode (commented . nil)))
  "The first matching key in the list that the current major mode derives from
will be used to override default settings")

This would address issues like #57 in general, the user simply adds their custom mode settings to the list

(add-to-list 'outshine-custom-spec-alist
  '(lean-mode (base-char . ?-)
              (padding . nil)))

And it would construct a outline regexp with the repeated base-char portion in a capture group, simplifying the logic of functions like outshine-calc-outline-level and outshine-fontify-headlines which currently use quite a hackish series of substring manipulations to get that information.

e.g. in python-mode:

;;   commented               max-level
;;   |                       |
;;   |   padding             | base-char
;;   |   |                   | |
(rx "#" " " (group (repeat 1 8 ?*)) " ") ;; => "# \\(\\*\\{1,8\\}\\) "

;; in a Python buffer:
# *** heading

(length (match-string 1)) ;; => 3

[yuhan0]

@thblt that sounds like an interesting idea, but for a completely different package as it would probably require a complete ground-up rewrite of the existing Outshine code.

I think outshine's method of "hijacking" outline-regexp isn't an issue, it's one of those variables like beginning-of-defun-function that's meant to be overridden.
It also means that we can reuse all the folding and structure manipulation functions from outline-minor-mode, and automatically interface with other modes which expect semantic grouping of outline structures for free. E.g. with lispy-mode, calling lispy-eval on a outline heading sends the entire subtree to the repl, and this works automatically with Outshine.

[simonmichael]

Thanks for thinking about this. 3 years later, count me in favour of any such cleanup. Straightforward tab-cycling of document sections with easily customizable headings is still a valuable feature and nothing seems to do it well. (I have had it working most with outshine but it's rather difficult, eg #77.)