[Proposal] Process Management Refactor

[avamingli]

Proposers

@my-ship-it @Ray-Eldath

No response

Proposal Status

Under Discussion

Abstract

Process Management Refactor(PMR for short, off hand).

The final goal of PMR is to manage processes of CBDB in a cloud native way.
Current CBDB allocate\deallocate Gangs by QD, and each QD have to keep connection with all QEs.
For the plan to execute on QEs, QD build and dispatch them after gang created.
However, we could make QD only send a request to allocate/deallocate gangs on segment postmaster process.
And later plan info could be dispatched to segment postmaster who will take response to sync that to QEs.
We could save a lot when there are more segments or parallel processes.

This(PMR) is a long-term project, for the first Most-Value-Product-0(MVP0),
out target is: QD dispatch plan info(to be executed for QEs) to only one QE(Writer Gang), and Writer Gang sync that plan info to other Reader Gangs.
That is, QD only dispatch plan once, not to all Gangs.
The sync way of plan info is like snapshot sync between Writer Gang and Reader Gangs.

Motivation

The final goal of PMR is to manage processes of CBDB in a cloud native way.

Implementation

To be discussed.

Rollout/Adoption Plan

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[avamingli]

All of this proposal could be discussed, feel free to comment.

[my-ship-it]

The MVP LGTM.
In longer term, postmaster could fork multiple QEs, and there will be a leader QE, which is responsible to start other QEs, receive & share plan, and check result.

[Ray-Eldath]

the first draft of design has been put forward. notably, cursor (extended query protocol) and InitPlan complicates things considerably.

---

There're three actions of SharedQueryPlan:

• READ
• WRITE

SharedQueryPlan works as follows:

• READ:
  • happens before calling exec_mpp_query
  • checks if a SharedQueryPlanHash (a shmem HTAB) have the entry of the key gp_session_id (current session id).
    • if so, checks whether SharedQueryPlan->ready && SharedQueryPlan->commandCount == gp_command_count. when success, attach the dsm handle and read serializedPlantree and serializedQueryDispatchDesc. then decrease reference of the entry by 1. when reference reaches 0, reclaim the DSM allocated by writer QE.
  • if either such entry doesn't exist or gp_command_count doesn't match, use polling to wait until SharedQueryPlan is ready. if
• WRITE:
  • in exec_mpp_query, after plantree and ddesc are deserialized.
  • if Gp_is_writer && !plan->utilityStmt, create a new DSM segment, store serializedPlantree and serializedQueryDispatchDesc to it, then set the corresponding dsm handle into SharedQueryPlanEntry struct. because we don't know who will exit at last, the destroy of dsm segment must be handled in a symmetric way. during WRITE, the writer will go over sliceTable and calculate the count of slices that consisted of current segindex, set it in SharedQueryPlanEntry->reference so that DSM can be reclaimed later. after a SharedQueryPlanEntry has been initialized, set it to SharedQueryPlanHash with current session id (gp_session_id) as key.

note that extended query protocol are not supported. SQP only kicks in if !QEDtxContextInfo.cursorContext && GpIdentity.segindex >= 0.

FAQ

#### Why serializedQueryDispatchDesc cannot be shared as well? and why is writer still need to try to read from SharedQueryPlan? should writer just set it?

these two problems are all caused by InitPlan. it invalidates a basic assumption of this feature: there'll be only one gang have Gp_is_writer == true and that gang doesn't change during any query's execution lifetime. this is unfortunately not true. for the query plan:

the exactly same plan will be dispatched twice, with different ddesc (but same plantree). first dispatch to the InitPlan part (slice1 and slice2) during preprocess_initplans, and dispatch again to slice0. the slicetree isn't a tree, it has multiple roots (hence a forest, or a DAG), and multiple roots needs to be dispatched multiple times. because of this quirk, we can only guarantee that for the same slicetable root, ddesc and writer gang will be the same (though plantree is same across all roots). but the referenced is calculated across all roots, that means it isn't right. to fix this, we can either introduce tree-traversal during reference calculation, or we can make the whole thing symmetric: no matter how many different writers are there, the first one gets to write SQP. and even there's another writer during later phases of execution, it still reads the plan written by first writer (i.e., first-writer-wins). I choose the later scheme, but I have to give up synchronization of ddesc due to the two dispatches have different ddesc. as I've said, ddesc and Gp_is_writer are not the same across sliceTable roots.

I'm open to any other possible design.

[Ray-Eldath]

update: turns out ExecSlice has a rootIndex field to show which root is the slice part of. use that directly and ditch the symmetric scheme.