modules and blocks

[cherob67]

Good morning

I want to share some thoughts regarding blocks and presets. I started from a "life-size" test of retouching on a personal photo that I have already retouched in DT.
Here is the photo (jpeg associated with the raw):

Here is the principle of treatment by zones (masks) with modification (according to the zones) of the luminosity, the color, the contrast and the local contrast

Here is the photo obtained with VKDT

And here is the the pipe (with already a large number of nodes - but there is even more complex ! )

which I propose the following structure:

From this structuring, the idea that i propose is to establish a level above the elementary modules which would be the elements (name to be found - perhaps the blocks or presets) corresponding of course to an aggregation of elementary modules (or smaller elements) and ensuring more complex and complete functions. These blocks/presets could be established by the developers with the possibility for each user to create (and share) their own.

Exemple:

The module editor would provide access to this intermediate level allowing easier development and reading of the pipe. The settings corresponding to each module composing an element would be accessible on the right column with an indentation to distinguish the different modules.

[butcherg]

Interesting. In the manner of functions and procedures in programming, I'd propose the ability to construct modules as collections of modules, using "first-class" modules provided by vkdt and/or other modules defined in, say, .mod files. Like a "named network".

The "blend" module and it's ilk would need a new connection type, multi-input... ??

There's published research on such networks; maybe there's already proposed formal definitions?

[hanatos]

this looks great! i think some high level primitives like this are necessary, yes.

so far, blocks are subgraphs that contain one input and one output connector, along with an instance name shared between the modules in the block.

i suppose that could be extended to an arbitary number of i/o connectors, so the subgraph templates could be created from a number of selected nodes, and also inserted as such. ideally the gui would provide the high level overview as in your last screenshot here, and be able to expand it if required.

so
(a) extent block definitions to multiple connectors i/o
(b) write ui to generate such blocks from selections
(c) write ui to apply such blocks (put nodes with stored positions in the graph for manual connection)
(d) write ui to expand/collapse blocks for better overview

i think (a)-(c) are straight forward, (d) would need some (gui only) data model that keeps information about which modules should remain grouped as blocks.

in addition to that, i see a need for
(e) implement high level elements such as reduction (your blend multi block) or iteration/accumulation over frames

i think #70 needs to be resolved before this: we want to have a common place where to store such gui extra information along with a graph.

i'd really like to read about formal graph interaction methods, if you find any links that'd certainly be interesting.

[cherob67]

I think VKDT is more and more interesting with already great capabilities and has a bright future ahead of it. My regret is that I am not able to help you much more. I'm just an photographer...

[hanatos]

..not saying anyone should do anything here, just putting together a roadmap how features could be implemented. will likely do that myself, time permitting :)

thinking about it maybe such large graphs will likely need a more feature rich backend: https://github.com/thedmd/imgui-node-editor

[butcherg]

Powerful constructs need careful consideration, avoid dooming our descendants with inflexible tools. E.g., params are a good start, need to try to not lock in to a particular UI implementation... I'm at limited Internet this week, but working intermittently on compiling vkdt with MSYS2.

…

On Mon, Mar 27, 2023, 7:55 AM hanatos ***@***.***> wrote: ..not saying anyone should do anything here, just putting together a roadmap how features could be implemented. will likely do that myself, time permitting :) thinking about it maybe such large graphs will likely need a more feature rich backend: https://github.com/thedmd/imgui-node-editor — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC4RLEDP6UEQH3LCU635BTW6GL67ANCNFSM6AAAAAAWHMWBRM> . You are receiving this because you commented.Message ID: ***@***.***>

[hanatos]

conceptualising a bit, with request for comments:

gui

create block

mark a couple of modules in the graph editor and push button/shortcut. this can be used as cleanup/convenience to group modules. does not change the underlying graph (in particular instance names or connectivity of modules).

collapse block

some shortcut/button on block. hides all internal modules and shows the block as a dummy module
with dummy connectors. cleans up the view, does not alter the graph.

load block from file

adds a new block to the graph editor panel. this includes creation of the internal modules and their connections.
module positions will be relative to the block dummy node and only visible if the block is not collapsed. this adds
new modules to the graph. their instance names will be composed of the module instance in the block file and
a block instance name given as argument when loading the block file (means both can effectively be 4 chars).

create block file

mark a block in the gui and push button/shortcut. this will create a block definition file to be loaded later. means all modules involved in this process should not have instance names > 4 chars.

backend

files

• the .cfg files shall have a [gui] section ignored by all current code storing module positions and block data.
• block files need a new format replacing the current blocks.

block file

• list of modules with name, instance, relative position
• list of internal connections between these modules
• list of parameters of internal modules
• list of virtual connections from internal modules to virtual block connectors (which will be visible if the block is collapsed)
• block instance name + individual annotation? help/tooltip text? colour? pin names?

cfg file block annotations

this is a bit different to block files since these do not have to be generic. it is enough to store the block id (instance name) per module for grouping functionality. also want metadata (block tooltip/colour/pin names etc)

history

do block operations go into history? they are just a gui thing and irrelevant for functionality/render result. obviously low level graph manipulation which happens when working with blocks does go into history (add module, connect modules, etc).

connecting blocks

the gui will initiate connections between modules or blocks alike and not necessarily even know which of the two it is. code needs to take care the right thing happens (block input connector may delegate to multiple internal module inputs).

deleting blocks

can be initiated from the gui explicitly. but also happens

• if internal modules are regrouped into new blocks,
• if internal modules are connected cross-block boundary (as it renders the block state invalid, may create a new virtual pin for instance).

high-level primitives on blocks

working efficiently with a large number of modules:

• insert arrays of blocks with counting instance names
• perform reduction of outputs (for example cascaded blend modules inside blocks)

[butcherg]

Blocks aren't just a GUI thing, they're like a third-class network member, after nodes then modules. We should be able to use blocks in .cfg files as just another network element.

Do you envision adding really useful blocks to the vkdt source tree?

Oh, got me thinking about fundamental ops, the following modules might be of use in defining blocks:

• math (add, subtract, multiply, divide)
• lut (1d tone curve)
• clut (3d color)
• matrix (3x3, with this, lut, and an i-icc module I'd define a icc block)

[butcherg]

Consider this:

• separate block file format, .blk, not constrained by dt_token
• blocks/ directory, search cwd then blocks/ for defined blocks
• separate blocks section in .cfg files, parsed into component modules for graph loading

My academic background is compilers/programming languages, this looks like a programming language... 😆

[cherob67]

I try an answer :

• What is a block?
  A block is a set of connected modules and/or blocks (-> to be seen) performing a global complex function.
  In the pipe, the block is identified by a unique label.
  In the pipe, blocks and modules can coexist.

• Creation of a block
  The block can be created in the pipe by a group of modules connected to each other.
  Ability to create a block within a block (→ to see?)
  It can also already exist in a pre-established database or in a modifiable local database of blocks (files).

• Incoming and outgoing connection points of the block
  They are defined when the block is created. An incoming connection point is linked to one or more module inputs internal to the block and/or to an outgoing connection point. An outgoing connection point is linked to a single output of one of the modules internal to the block or to a single incoming point of the block.

• In-block bindings:
  These are the links between modules contained in the block, the links between incoming and outgoing points and modules.
  When the block is created, the links to modules external to the block are automatically split with creation of an incoming or outgoing point.
  Direct links between a module internal to the block and a module external to the block are prohibited.

• Block without module:
  Possible. Allows to create kind of n → m bus to simplify the graph (for example i-lut spectra or abney, etc.).

• Graphically:
  The block is like a module, ie a rectangular object with the label on top and the incoming and outgoing connection points below. A different color makes it easy to distinguish.
  The position of the modules of the incoming and outgoing connection points of the block are chosen during the creation of the block from one of the vertices of the block
  The block can be displayed:
  → zoomed in (open): the modules and internal connections are then displayed (-> to see: in a popup type window)
  → zoomed out (reduced): the block appears as a basic element of the pipe. The block can be opened with the mouse (by a double click for example, or with a specific button) or from the keyboard by a key combination .

exemple 👍
-> closed

->opened

• Modification of a block:
  the block can be "opened" and you can then add/remove/move modules, the connection points of the block, the links between modules/blocks internal to the block, the links between the modules and the incoming and outgoing points of the block.
  The block thus modified can be saved in the block database with a different label (initial database cannot be modified).

• Block files (.blk):
  they contain the list of modules/constituent blocks, the incoming and outgoing connection points, the connections between modules/blocks, the connections between the modules and the incoming and outgoing points, the positions in the block of the incoming and outgoing modules and points, the presets.

• Block settings:
  when you select a block, the corresponding settings are accessible in the right-hand column ( -> to see? settings accessible in a popup type window):

• block label

  • label of module 1
    • setting 1
    • setting 2
      -....
  • label of module 2
    • setting 1
    • setting 2
      -....
      -.....
  • label of block 1 (to see ???)
    • label of module 1
      • setting 1
      • setting 2
        -....
    • label of module 2
      • setting 1
      • setting 2
        -....
    • ...
      -...

• The .cfg files:
  They are located in the directory of the photo file. The name of the .cfg file uses the name of the photo file.
  The .cfg file contains
  the references of the processed image
  the complete list
  modules
  blocks with their content (modules, etc.)
  the positions of the modules and blocks
  links between modules and/or blocks
  The settings

• Removed a block in the pipe:
  Deleting removes the block, all its content (adjustment modules and internal links) and all incoming and outgoing links of the block.
  No automatic deletion – rather inconsistency message and blocking the current action causing the inconsistency.

• Treatment history:
  It concerns the work in progress. It makes it possible to go back over the different stages of the processing carried out, to make comparisons and to resume the processing at an intermediate stage. It necessarily contains the information of the modules and blocks used and their links, including those deleted during processing.

[hanatos]

good discussion! let me try to answer some things:

@butcherg :

gui thing or not

nodes are 1:1 compute kernels, modules are 1:1 gui controls/uniforms/some cpu side callbacks that make sense to process at a coarser level (buffer size/type negotiations etc). blocks? don't change anything in the pipeline.

having blocks in cfg files in the non-gui part wouldn't change much. maybe make it more convenient to write the cfg by hand (so it's a ui thing, if not gui in this case), and would make the cfg less self-contained since then it depends on block definitions (so these need versions etc).

fine grained modules

i'll certainly not have add/mul/div style nodes or even modules. these are too fine grained to yield good performance. colour treatment should go into the colour module. every node carries all data from global memory and back, that's slow.

block files

there is a blocks/ directory with the definitions, which are (as of now) just cfg with some pattern to be replaced. i'd make this pattern a bit more generic. i don't think the token restriction is too bad, we can have "excessive string annotations(tm)" in the future. actually there are two such directories: one shipped with vkdt (const and will be changed by updates), and one in the user's home directory (can be written to, untouched by updates).

compiler

yeah everything is a graph.. though i'd more like some high level language to modify the graph. say place arrays of blocks and condense lists of blocks to a single output via reduction. the syntax trees in compilers are different (way more fine grained and not cyclic to begin with)

@cherob67 :

yes that looks very similar to what i envisioned.

recursive blocks inside blocks: i don't think i want that. that'd be like recursive nodes in substance designer. i suppose you can always regroup even more modules to even larger blocks.

i'd probably allow cross block boundary connections if the block is expanded, and in the process destroy the block (if you want to keep it, connect to the block connectors while it is collapsed). in particular that'd make a block appear as a regular module with strange name and maybe some annotation if it is collapsed. if expanded, it'd just be a group of modules maybe with a background colour. the connectors would still be the module connectors as is now.

i agree with most of your notes. took note that:

• you don't want to auto delete blocks but fail the connections/operations instead or at least prompt before doing it
• want block stuff in history. probably easy to ignore for cli and other than that history is kinda gui. thinking about it maybe putting all block related info into this list may be the best way of doing it (except the prefab/shipped/saved blk templates).

[butcherg]

Okay, I get basic math ops, but what about matrix/lut/clut? I don't think you want to deal with littlecms or OCIO directly in vkdt/libvkdt, so you need ops that allow construction of interfaces (blocks?) Same with lensfun, compatible or convertible warp ops...

…

On Wed, Apr 19, 2023, 7:12 AM hanatos ***@***.***> wrote: good discussion! let me try to answer some things: @butcherg <https://github.com/butcherg> : gui thing or not nodes are 1:1 compute kernels, modules are 1:1 gui controls/uniforms/some cpu side callbacks that make sense to process at a coarser level (buffer size/type negotiations etc). blocks? don't change anything in the pipeline. having blocks in cfg files in the non-gui part wouldn't change much. maybe make it more convenient to write the cfg by hand (so it's a ui thing, if not gui in this case), and would make the cfg less self-contained since then it depends on block definitions (so these need versions etc). fine grained modules i'll certainly not have add/mul/div style nodes or even modules. these are too fine grained to yield good performance. colour treatment should go into the colour module. every node carries all data from global memory and back, that's slow. block files there is a blocks/ directory with the definitions, which are (as of now) just cfg with some pattern to be replaced. i'd make this pattern a bit more generic. i don't think the token restriction is too bad, we can have "excessive string annotations(tm)" in the future. actually there are two such directories: one shipped with vkdt (const and will be changed by updates), and one in the user's home directory (can be written to, untouched by updates). compiler yeah everything is a graph.. though i'd more like some high level language to modify the graph. say place arrays of blocks and condense lists of blocks to a single output via reduction. the syntax trees in compilers are different (way more fine grained and not cyclic to begin with) @cherob67 <https://github.com/cherob67> : yes that looks very similar to what i envisioned. recursive blocks inside blocks: i don't think i want that. that'd be like recursive nodes in substance designer. i suppose you can always regroup even more modules to even larger blocks. i'd probably allow cross block boundary connections if the block is expanded, and in the process destroy the block (if you want to keep it, connect to the block connectors while it is collapsed). in particular that'd make a block appear as a regular module with strange name and maybe some annotation if it is collapsed. if expanded, it'd just be a group of modules maybe with a background colour. the connectors would still be the module connectors as is now. i agree with most of your notes. took note that: - you don't want to auto delete blocks but fail the connections/operations instead or at least prompt before doing it - want block stuff in history. probably easy to ignore for cli and other than that history is kinda gui. thinking about it maybe putting *all* block related info into this list may be the best way of doing it (except the prefab/shipped/saved blk templates). — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC4RLFSXUNLXZORX65LSXLXB7JDDANCNFSM6AAAAAAWHMWBRM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hanatos]

agree, i don't want those libraries. if at all, i want facilities to load matrices/luts but not the code to apply it (would not fit the pipeline/language). also i'm not overly keen on 100% icc coverage, more like whatever is really needed to get stuff done.

matrix: https://github.com/hanatos/vkdt/blob/master/src/pipe/modules/colour/main.comp#L138

2.5D lut (dcamprof style): https://github.com/hanatos/vkdt/blob/master/src/pipe/modules/colour/main.comp#L99

3D lut for colour correction or crazy manipulation (RBF-based): https://github.com/hanatos/vkdt/blob/master/src/pipe/modules/colour/main.comp#L162

2D->2D lens distortion: https://github.com/hanatos/vkdt/blob/master/src/pipe/modules/lens/main.comp#L26

don't quite get the connection to blocks, but image parameters are passed along the pipeline from input to the individual connectors, for instance here the raw input sets the colour matrix: https://github.com/hanatos/vkdt/blob/master/src/pipe/modules/i-raw/main.cc#L343

that would probably also be the case to load/convert lens distortion metadata (if any) and in the lens module commit_param callback these can be parsed and passed as uniforms to the actual processing code.

[cherob67]

Thank you for your reply.

I just have a comment about modules and blocks.
I think for photographers who will use vkdt, blocks will become the basic element in organizing their pipe for photo editing, especially once they build up their library of blocks. While for developers the basic element will remain the module.

I'm a photographer and not a developer, that's why I tend to want to give the block a clearer reality that goes beyond the simple aggregation of modules.