Is this only for scripted pipelines? The documentation of parallel in declarative pipelines specifically disallows nesting.

Technically one can still achieve nested parallelism via the script block. However, the fact that it's not supported as a "first-class" capability might make it dubious to support here.

@KalleOlaviNiemitalo yes, scripted pipeline only

@solvingj good point about "first-class" capability. I wonder whether the current limitation is due to the underlying datastructure, or just the UI render. There's a change that it's the former as the blueocean plugin also renders similar graph for the nested parallel pipeline.

From https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/15978cd172a8d34e2cd6bf8a6bfeaa5514d44dad/rfc/JENKINS-41334-parallel-stage-execution.md, it looks like nested parallel stages were disallowed in declarative pipelines to let Blue Ocean display the stages more easily. Perhaps that restriction could be removed if this plugin generally replaces the Blue Ocean visualisation.

I think disallowing nested parallel stages might be the single most limiting part of Jenkins. Removing this restriction would GREATLY improve my pipelines and would make declarative pipelines unparalleled powerful (sorry about the pun).

I currently have to write awkward script blocks in my declarative pipelines to get the nesting parallel stages. This works fine enough to get what I need. I just wish that Jenkins could have nested parallel stages as a first class feature instead of having to use a hack (special nested scripted blocks).

JENKINS-62773 requests allowing nested parallel stages.

JENKINS-51808 requests allowing a dependency graph between stages that are executed in parallel. That could be useful for some jobs here but I imagine it would be tricky to visualize.

👍 This feature would be much appreciated.

Maybe it's simpler to abandon the table-like visualization altogether and use something like a plain graph. There are libraries for this out there (but they are on NodeJS). I think it doesn't matter whether it fits the agenda or the religion. There are many janky parts of Jenkins where things don't really fit. It should just work. Parallel stages are powerful yet lacking in visualization.

Any update on this?

You can use the BlueOcean plugin, that is capable of displaying even more complex workflows.

Sadly, it isn't, at least not for my case. I have nested parallel blocks and it shows only a few of them, precisely which is determined by a toin coss.

In addition to that, I cannot see all the "task nodes", and all the visible ones can turn green (and the banner turns green), as if the job is done, but some of the nodes are either waiting, working, or have failed.

@PhotoTeeborChoka if you know a workaround, please let me know, I have to use a combination of both BlueOcean and pipeline-graph-view for this, but I'm still looking at a Jenkins-Frankenstein's monster.

pipeline-graph-view expands the parallel block vertically, but everything else is horizontal so I have about 300 nodes horizontally, and every 70 or so there's a vertical block of 15 'task nodes' (I don't know the terminology), but for viewing logs of a specific one I have to click on it, and then plug into the matrix and turn both keys at the same time to find it in the bottomless pit of the dropdown logs for each stage. Please tell me there's a better way.

I don't know about the scale you describe. But generally, what did not work for my with this plugin was visualized correctly in BlueOcean plugin.

One thing I noticed recently is this pipeline here:
https://github.com/jenkinsci/docker-inbound-agent/blob/24600786d9342f27c3beac7f55a5e422dd2d360b/Jenkinsfile

which has parallel and matrix nested (using a combination of declarative and scripted) displays very broken while the build is running but fine after completion:
https://github.com/jenkinsci/docker-inbound-agent/blob/24600786d9342f27c3beac7f55a5e422dd2d360b/Jenkinsfile

would be good to see if we can fix that

I guess a potential starting point for re-implementing the visualization using a graph would be taking a look at the ECharts API plugin, which somehow integrates nodejs into itself to provide ECharts (if I am not mistaken). As long as I am using Jenkins, it is only a question of time until I delve into this. I think Blue Ocean was a waste of resources in terms of development and in terms of discussion value+complexity to distinguish the two in 'works for me' comments. There are so many other areas that could have been improved...

looking forward to see this feature implemented

@mdealer did you have time to delve into this?

I've been considering this as part of #233 - unfortunately, it's not a complete fix - I think the graph represented better, but I don't think the front end supports nested parallel blocks.

I've been trying to think through my expectations for a nested graph. I think for:

node {
    parallel([
        "A": {
            stage("Build") {
                echo("Build A")
            }
            stage("Test") {
                parallel([
                    "A1" : {
                        stage("Config") {
                            echo("Configuring test A1")
                        }
                        stage("Test") {
                            echo("Running test A1")
                        }
                    },
                    "A2" : {
                        stage("Config") {
                            echo("Configuring test A2")
                        }
                        stage("Test") {
                            echo("Running test A2")
                        }
                    }
                ])
            }
        },
        "B": {
            stage("Build") {
                echo("Build B")
            }
            parallel([
                "B1" : {
                    stage("Config") {
                        echo("Configuring test B1")
                    }
                    stage("Test") {
                        echo("Running test B1")
                    }
                },
                "B2" : {
                    stage("Config") {
                        echo("Configuring test B2")
                    }
                    stage("Test") {
                        echo("Running test B2")
                    }
                }
            ])
        }
    ])
}

I would expect something like this (albeit with the branches ordered down the page, A1, A2, B1, B2):

You can see the raw dot graph here

@timbrown5 , unfortunatelly #223 was reverted by #250 ... Altough it isn't a complete fix as you said, what is needed to make it happen? I really appreciate your time and efforts twoards a solution for this issue. I'm really looking forward to use this plugin as a replacement for old blueocean

Understood. Please, keep us posted about it and enjoy your holiday!!

hey @timbrown5, did you enjoy your holidays? I hope so :)

Just a kind reminder about this feature. Did you have time to dive into this?

Hi @timbrown5 ,

First of all thank you for the tree layout fix.

But I think PR #285 will not include a fix for this issue anymore.

Do you have an alternative idea to fix this issue? (maybe in another PR)

Thank you again for your contributions.

@timbrown5 any chance you'll give this another try?

Hi, first thanks for your effort. Is there a chance to give this fix another try? Currently there is a responsibility circle beteween Blue Ocean and Stage View Plugin where both sides state that we should look for a solution on the other one. It looks like this plugin is our only chance to have a proper visualisation of complex, dynamic pipelines.

AFAIK Blue Ocean is dead, it's on life support / bugfixes only mode. So yes, better visualization here would be good to have :)

It's really sad that blue ocean is dad. It has so much better UI from this plugin or any other plugin of such type. I wish we could extract just the pipeline view from there without any other capabilities.

Hmm, howdy @timbrown5 I been looking at your #285, PipelineGraphLayout code but honestly for me its like playing a game of sticks. Change one thing and everything else freaks out. I'd be happy if just all nested nodes are visualized at the moment for all coding methods.

One of the biggest problems i keep running into while fiddling with code is that TreeVisitor/GraphAdapter code seems to translate pipelines differently depending on how you code your pipeline. I was trying to test Pure Declarative Pipelines with simple parallel stages and Pure Scripted Pipelines with nested parallels and Mixed half declarative with scripted nested parallels. (which is my preference)

The node data seems be structured per style/method of coding is not consistent. Somtimes nodes are not registered or should i say its not paired with its parents and so the visualizer is not given the information to make it appear. If you continue the project, maybe you can solve the data stream being consistant for all possible styles?

If we could get all the methods to translate into the the same or similar node structure. Ideally in the order the pipeline is structured all into a single arrays for each major top parent. it would make it much easier to break down the nodestages and children of children of children nesting are way into infinity..etc. Right now sometimes it is...sometimes its not in order.

I'm trying to refactor the react code a bit in my fork in https://github.com/Danoloan10/pipeline-graph-view-plugin with the goal that it makes it easier to make a nested graph view.

@Danoloan10 Awesome, if this feature gets implemented, it will encourage more efficient pipelines.

Is this project still active? Anyone assigned to this specific issue? The fact that nested parallel stages are not rendered right even though things work in the backend is unfortunate for diagnosis and debugging purposes.

@agirault From my understanding, it's kinda an overhaul hobby project for willing developers. Granted the product as is seems to be functional enough for most average pipelines but complex ones are harder to handle. But as i said before since the declarative and scripted coding styles results in different instructions to the graph visualizer its hard to update the graph visualizer to work for both styles. If you a seasoned developer be my guess. lol

I wanted to get this working also but ran into the problem. Every change to the graph visualizer resulted in one side coding style becoming even more broken.

Found out about this a while ago: https://docs.cloudbees.com/docs/release-notes/latest/plugins/cloudbees-pipeline-explorer-plugin/

Apparently, it is kind of a Jenkins fork, or what am I looking at? Most of it smells like Jenkins but it looks different and feels closed. I could be mistaken, maybe someone can shed some light on this. It says it utilizes the Jenkins engine, but the rest is confusing. Does it mean we can jump to CloudBees, keep all of our existing Jenkins and utilize their commercial support and the advanced debugging plugins? Cannot wrap my head around how this is still open source.

Found out about this a while ago: https://docs.cloudbees.com/docs/release-notes/latest/plugins/cloudbees-pipeline-explorer-plugin/

Apparently, it is kind of a Jenkins fork, or what am I looking at?

This CloudBees project is a proprietary one developed from scratch, unrelated to this plugin or BlueOcean.

To use it you have to be a CloudBees customer indeed.

It says it utilizes the Jenkins engine

It uses it to retrieve the logs.

It's been a while since I've been able to get into this but I'll share some ideas I found.

@SilverKnight1 is right in that this will be a difficult feature to implement. The graph UI is designed as columns of parallel stages. This assumption can be found in all of the React code, and I have not yet found its bedrock. Because of this the redesign must be radical and change the way some things are being displayed. For example, stages within stages are not properly displayed even when they are sequential. And steps outside any stage in a scripted pipeline are not represetned at all. We will need something like what @timbrown5 designed in his dot diagram.

I've tried to model the high-level concepts that we're managing in this UML domain diagram:

click here for the plantuml

It's a bit complex, let me break it down:

• Consecutive steps that are neither parallel nor stage are grouped into StepsBlocks
• The cornerstone of the GUI is displaying StepsBlocks. All elements aggregating StepsBlocks directly or indirectly must be displayed somehow, specially those that have names.
• About labels:
  • LabeledCircle is the Stage circle we all know and love
  • LabeledBranchout is the labels that are floating on top of branchout curves (you can see them on top of arrows in @timbrown5's diagram)
  • LabeledArea would be @timbrown5's dotted subgraphs
• A ScriptedPipeline is just a collection of StepsBlocks. Thus there can be steps outside stages, as well as directly inside a stage that has other stages in it.
• StepsBlocks that are not inside any stage can be represented as a circle without name.
• Any stage with stages or a parallel branching in it will need to be represented as an Area
• ScriptedParallelBranch is the map argument passed to the parallel step. They are fundamentally different from stages at the Jenkins Core level.
• Circles would represent only pure StepsBlocks with no parallel or stage (i.e.: OtherStepsBlocks)

I hope this helps when me or anyone starts writing the new UI code, because it will need to be mostly done anew.

Just a thought: maybe instead of trying to visualize nested-parallel pipelines in all their glory and complexity (which seems to be not only technically difficult, but unclear what the preferred design would be), could this plugin show nested parallel steps with a special circle icon, that when clicked would open those parallel stages elsewhere (the current view/new window/popup window)?

I think this is doable, the issue seems to be that

1. the frontend loops through all children to build the graph.
2. if a stage has children, it recurses into them,

• except when it sees a parallel branch it doesn't recurse, but handles these later.

3. for parallel branches it does a separate loop which doesn't recurse, so it ignores nested parallel branches.

If this was updated to loop through all stages and add parallel branches as rows and not new columns (like it does for stages) it would be a start (adding the paths might need to be updated to support this).

Has the issue really been closed? #285's OP says otherwise

Over eager closing fix words, the description had doesn’t fix issue ID which meant it got closed as it has fix issue ID

@Danoloan10 , are you currently working on the new UI code in order to address this improvement? Does #285 helps you to move forward?

BTW, ty very much for your efforts.
Also kudos to @timbrown5 and @timja for merge #285

I looks at this a little and it seems straightforward to do what I said above and get the nested stages included (but not in the correct place/with the correct connections).
Before:

After:

(note that the 'After' stage is placed incorrectly and none of the nested stages are placed correctly or have the correct connections to the rest of the graph).

I took a loot and updating the placement and connection code, but it already seems convoluted and this might make it worse. It feels like there should be classes in this code to encapsulate all the data required to calculate connections/positions together (and split the large functions) - e.g. calculating a map of connections are required while looping through the stages.

Pipeline I used for testing:

node {
    parallel([
        "A": {
            stage("Build") {
                echo("Build A")
            }
            stage("Test") {
                parallel([
                    "A1" : {
                       echo("Test A1")
                    },
                    "A2" : {
                       echo("Test A2")
                    }
                ])
            }
        },
        "B": {
            stage("Build") {
                echo("Build B")
            }
            stage("Test") {
                parallel([
                    "B1" : {
                       echo("Test B1")
                    },
                    "B2" : {
                       echo("Test B2")
                    }
                ])
            }
        }
    ])
}

Added #360

Hello @timbrown5 and everyone else who is contributing to this,

What is the current status of this issue? Is it still in the progress?

I am pretty sure that you would make life easier to a lot of us here, so I am really happy if you continued to push forward towards this.

As always, thanks in advance for your efforts! :)

This would be super useful!

Nested stages parallel works (almost) perfectly in Pipeline Console and has for a while now. Does anybody know what is needed to move forward for this to work in Pipeline Overview as well?