Skip to content

Home

Jump to bottom
blythed edited this page Jul 4, 2023 · 26 revisions

How to develop in SuperDuperDB

This development workflow is known to work on some recent MacOS and *nix computers, [specific platform info TBD].

You will need to globally install the following programs outside your Python environment using pyenv, Brew, apt or your favorite package manager.

(MongoDB Community edition], our primary database, is now installed through Docker and doesn’t need to be installed locally.)

Virtual environments

We do all our development in [Python virtual environments](https://realpython.com/python-virtual-environments-a-primer/) or Docker containers so as not to pollute our system Python installations. Poetry can [create and manage](https://python-poetry.org/docs/managing-environments/) a virtual environment if you don’t have one. This involves installing and then activating the project Poetry environment (see [below](https://www.notion.so/How-to-develop-in-SuperDuperDB-1ebc198084b94e79a24769e8d3ad6746?pvs=21) for more details).

Suggestion: making things break if there’s no virtual environment

pip3 might point to any executable so should always be avoided.

python -m pip or poetry will use whatever your current Python is, which might accidentally be the wrong one.

If you develop from the command line, a useful idea to avoid problems is to make sure that python does not resolve to anything if no virtual environment is loaded, so your commands just break rather than silently installing things in the wrong place.

Don’t worry, you can and should have python3.8, python3.10 and any other Python versions you have installed in your PATH.

To test, deactivate or otherwise leave your virtual environment, and then type -a python should say bash: type: python: not found

To fix this, change your PATH or if there’s something in there that can’t go, let me know!

If you use an IDE, this probably won’t be necessary.

Getting the code

The code is located at https://github.com/SuperDuperDB/superduperdb-stealth.

We each work on personal forks of this repository, so create your own fork on your personal GitHub account [here](https://github.com/SuperDuperDB/superduperdb-stealth/fork).

From that landing page, click on the green Code button, then Clone, SSH, and then Copy, to get a URL like this: [email protected]:<your-name-here>/superduperdb-stealth.git

Go to the parent of the directory you want to work in, type git clone and paste that URL, to get something like:

git clone [email protected]:<your-name-here>/superduperdb-stealth.git

Now type cd superduperdb-stealth. All operations are done in this directory.

Either create a new Python 3.8 virtual environment and make sure it’s active, or make sure that python is not in your PATH, so poetry will create a new virtual environment for you in a cache.

Then type poetry install and wait about 15(!) minutes.

This step downloads different software depending on your hardware and software configuration and might cause issues on previously unseen configurations, let us know!

Make sure that Docker is running.

Finally, type make test and if all has gone well you should see a couple of dozen lines of success messages, and about fifty lines of warnings from other packages, see https://github.com/SuperDuperDB/superduperdb-stealth/issues/219.

make test will be quite slow on the first run, as it loads a Docker and compiles all the Python, but subsequent runs will be quite fast. If progress stops for more than a few seconds, then it is likely a problem communicating with a Docker container.

Development workflow

Please take a look at [Git workflow](https://www.notion.so/Git-workflow-7599f2de2a7f4b3caab63292b2f07a38?pvs=21) and[How to use poetry](https://www.notion.so/How-to-use-poetry-3536c96ceba7420d825c02a7a767ffd4?pvs=21) for the basics and some notes and tips.

Starting out:

For each new feature or bug fix, create a new branch on your own fork

The work loop:

  1. Make changes to the Python code
  2. make fix-and-test which fixes style errors and runs style tests and unit tests.
  3. Commit often
  4. Update from mainoften
    1. git pull --rebase upstream/main and git push -f
    2. or git update if you have it
    3. Finally poetry install --sync will install any missing dependencies, and remove unknown ones.

When ready to submit:

  1. Distill the work into one or more minimal, self-consistent commits
  2. Run make test which runs style tests and unit tests but fixes nothing
  3. Open a pull request

During reviews:

  1. Fix code in response to comments in one of two ways:
    1. Use [--fixup commits](https://blog.sebastian-daschner.com/entries/git-commit-fixup-autosquash)
    2. Rebase changes into existing commits and git push -f
  2. Update from main
  3. Run make test

Finally:

  1. Rebase any --fixup commits then git push -f
  2. Update from main
  3. Run make test
  4. Go to the pull request
  5. Wait until it has passed the Continuous Integration tests
  6. Pull it if you have pull rights on the main repo, or ping an administrator.

Documentation rebuild (not needed at this instant):

  1. cd docs
  2. make clean
  3. make html

Note:

In the early phases where no one sees the repo, we can even rewrite history to erase egregious mistakes, so don’t get too stressed.

Clone this wiki locally