[Feature] Upgrade Python code and dependencies (gpMgmt module)

[SaelKimberly]

Description

Hello everyone.

I am a python developer, and I would like to discuss the situation with gpMgmt. There are several problems that I think should be solved:

1. Reorganization of the module. I would like to bring all the code into compatibility mode with Python 3.9, the minimum supported version of the Python community, and get away from using hacks that are outdated.
2. Formalize the module's CLI, and switch to using ArgumentParser everywhere (the old CLI system can be moved to a submodule). In essence, the entire CLI call system can be reduced to a single cli module, and management utilities can be called as follows:

# old style:
gpstop
gpreload -d ...
# new style:
gp stop
gp reload -d

And all available commands will be listed with:

gp --help

3. Fix all kinds of errors and warnings of the Ruff utility (the most advanced linter for Python), and clarify input data types and fields in all functions and classes, in accordance with best practices.
4. Completely avoid using star imports (from module import *) and absolute imports, excluding Python system modules and dependency packages.
5. Optionally, switch to using "pytest" instead of "unittest'. It is indeed a simpler and more useful testing platform, and it can also automatically calculate test coverage using the "pytest-cov" module.

Use case/motivation

I see the need for such changes for several reasons:

1. Python support and code safety. There is a part in the gpMgmt module written using Python 2.x, which is absolutely unacceptable from the point of view of support and security. Newer versions of Python are fundamentally unable to run this code, and no vulnerabilities can be fixed. In fact, as of today, the minimum supported version of Python is 3.9, and Python 3.8 receives security updates only in the form of code. This automatically introduces a restriction on some key dependencies, so for example, PyGreSQL cannot be lower than version 6.x.
2. Due to insufficient code typing, as well as numerous violations of good practices, the module code is full of non-obvious issues that can lead to incorrect program behavior. If python-related development had been carried out using modern IDEs with support for code auto-completion and code linting, it would have been possible to avoid calling non-existent functions and incorrect imports. Also, a strict attitude to typing and conventions when calling functions can eliminate the likely use of uninitialized or even non-existent variables passed as arguments.
3. I would like to use cloudberry, but as a Python developer, I understand that this system needs to be carefully refined before I can be sure that it will work correctly in production. I am especially concerned that the code of this quality manages the entire system.

I can probably start developing an alternative control module. Basically, it will come down to a neat code migration, and therefore will not take much time. However, some parts will probably require the implementation of large changes in the code base. Also, I would like to switch from using pygresql to psycopg3, which is adapted to use on most systems.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[edespino]

Thank you for the comprehensive proposal to upgrade the Python code and dependencies in the gpMgmt module. I appreciate the detailed suggestions for improving the current codebase, including:

1. Reorganizing the module for compatibility with Python 3.9+
2. Formalizing the module's CLI using ArgumentParser
3. Addressing errors and warnings identified by the Ruff linter
4. Improving input data type clarity in functions and classes
5. Eliminating star imports and moving towards relative imports
6. Considering a switch from unittest to pytest

I also note the suggestion to transition from pygresql to psycopg3. It's worth mentioning that this transition is already being considered as part of the upcoming Greenplum merge work, which aligns well with the proposal's goals.

These proposed changes would indeed address several important issues in the current codebase. However, while I acknowledge the value of these improvements, I believe we should consider a more fundamental shift in our approach to cluster management. Instead of investing time in refactoring the current Python-based system, I propose we explore an alternative cluster management mechanism that addresses the core limitations of our current approach, particularly the reliance on passwordless SSH.

Proposed Alternative: Rust-based gRPC Framework

I recommend developing a new cluster management framework using Rust and gRPC. This approach offers several significant advantages:

1. Enhanced Security: Eliminates the need for passwordless SSH, replacing it with a more secure gRPC-based communication system.
2. Improved Performance: Leverages Rust's high performance and gRPC's efficient protocol for faster cluster operations.
3. Robust Type Safety: Utilizes Rust's strong type system to catch potential errors at compile-time, enhancing overall system reliability.
4. Future-Proof Architecture: Provides a solid foundation for future enhancements and better scalability for large clusters.
5. Unified Codebase: Replaces the mix of Python versions and shell scripts with a cohesive, modern codebase.

Note: While Rust and gRPC are suggested here due to their strong alignment with our needs, they are not the only options. There are several modern technologies and approaches we could consider that would provide significant improvements over the current passwordless SSH mechanism. The key is to move towards a more secure, efficient, and maintainable solution for cluster management.

Key Components of the Proposed Framework

1. Node Service: A Rust binary running on each cluster node, exposing gRPC endpoints for node management.
2. Control Plane: A central service for cluster-wide operations and state management.
3. CLI Tool: A Rust-based command-line interface for cluster administration.

Benefits Over Python Refactoring
While upgrading the Python code would offer some improvements, shifting to a Rust-based gRPC framework provides more substantial long-term benefits:

• Comprehensive Security Improvement: Addresses fundamental security concerns rather than updating an inherently insecure model.
• Performance at Scale: Better equipped to handle the demands of large, complex clusters.
• Reduced Runtime Errors: Rust's ownership model and type system prevent many common programming errors.
• Easier Maintenance: A single, modern language simplifies ongoing development and maintenance.
• Attractive to New Contributors: This approach has the potential to attract developers interested in implementing Cloudberry management utilities without requiring deep knowledge of the Cloudberry kernel codebase. The clear separation between the management layer and the core database functionality could lower the barrier to entry for new contributors, potentially expanding our developer community.

[SaelKimberly]

I agree, indeed, using Rust would be preferable! It is more reliable, has no need for GC, is fast, memory efficient and avoids undefined behavior.

Meanwhile, a complete rewrite of the codebase is not an easy task. The gRPC protocol is good because a prototype of the future system can also be written in Python, using part of the old code, and then, having established the required conventions for the cluster nodes, create the final system in Rust.

I know Rust a bit myself, but I'm just not sure if I can design and recreate a system from scratch on Rust.

In addition, Python code will probably remain a de facto working method of management, along with a new generation system. Refining and stabilizing this code base will probably help both developers of the new system and users of older versions of cloudberry.

[edespino]

Thank you for your thoughtful response. I appreciate your agreement on the benefits of Rust and your insights on the challenges of a complete rewrite. I'd like to clarify a few points about the implementation approach I'm proposing:

• Incremental Implementation: While my initial response outlined an overview of a comprehensive rewrite, I want to emphasize that the actual implementation would be done incrementally. I'm not suggesting an immediate, full-scale rewrite of all utilities.

• Starting with a Single Utility: As a concrete first step, I propose we focus on implementing just one utility using the new framework. I suggest we start with one (e.g. 'gpstart', 'gpstop', 'gpstate'). This allows us to:

  • Prove the concept with a manageable scope
  • Establish patterns and best practices for future development
  • Gain practical experience with the new technology stack in our specific context

• Coexistence with Existing Utilities: The existing Python-based utilities would remain in place and continue to receive minor updates as needed to support Cloudberry. I suggest there is no intention to perform a major overhaul on these existing tools in the short term.

• Gradual Transition: As we develop and refine the new 'gpstate' implementation, we can evaluate its performance, maintainability, and developer experience. This will inform decisions about potentially migrating other utilities in the future.

• Learning Opportunity: This approach provides an opportunity for the team to gradually build expertise in Rust and gRPC (or whatever technology stack we ultimately choose) in the context of our specific use case.

• Prototype in Python: Your suggestion of potentially prototyping in Python using gRPC before moving to Rust is interesting. This could be a valuable intermediate step if the team feels it would ease the transition.

The goal is to move towards a more robust, secure, and maintainable system while being mindful of the challenges involved in transitioning a complex, production-grade system. By starting small and proving the concept with a single utility, we can make informed decisions about broader adoption based on real-world experience.

I welcome further discussion on this approach and am open to adjusting the plan based on team input and project needs.

[jianlirong]

Compared to Rust + gRPC, I personally lean towards Go + gRPC for several reasons:

1. Unlike database kernel modules, performance is not a primary concern for management tools, and Go is sufficient.
2. Go has a much wider developer base compared to Rust's steeper learning curve.
3. There are many open-source distributed system scheduling and management frameworks (e.g., Kubernetes) based on Go + gRPC, along with numerous best practices for microservices. We can leverage a lot from the existing projects, rather than starting from scratch.

[jianlirong]

Despite ongoing discussions about the finer points of implementation and future plans, we can all agree that: (1) There is a clear need to enhance the Python code within gpMgmt; (2) CBDB's current reliance on passwordless SSH and remote script execution for cluster management is outdated and poses significant security risks; a more modern approach is necessary; (3) The importance of maintaining the usability and stability of gpMgmt tools cannot be overstated, as they are used daily by CBDB users.

[jianlirong]

I suggest we can start by addressing the first issue: upgrading the Python version and dependencies, refactoring the code, and adding more tests to the gpMgmt module.

@SaelKimberly would you like to submit some PRs?

[SaelKimberly]

Yes, of course, I would like to start PR, but I have a couple of questions:
What is the minimum version of Python that we will support? I suggest 3.9 or 3.11

[jianlirong]

Several factors may impact the Python version, including:

• gpMgmt tools;
• PL/Python;
• the operating system's default Python installation.

Because I haven't written code in a while, this list is likely incomplete. Others are welcome to add to it. Thanks.

For example, @edespino maybe you can tell us which version of Python is usually associated with Rocky Linux 8 & 9 (or other operating systems where users deploy CBDB).