Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize repo and add simple memory tests #1

Merged
merged 53 commits into from
Feb 22, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
0e20c14
add initial scaffold
d33bs Jan 5, 2024
1669ed0
add pre-commit checks workflow
d33bs Jan 5, 2024
1123ddb
update linting, add pipeline and runner modules
d33bs Jan 5, 2024
0c7271a
linting
d33bs Jan 5, 2024
66db5b7
move dockerfile, add a baseline test, pipeline cfg
d33bs Jan 5, 2024
0026fba
rename / typo
d33bs Jan 5, 2024
49c6218
readd dockerfile
d33bs Jan 5, 2024
3bb470a
Update pyproject.toml
d33bs Jan 5, 2024
79c6fe3
update pipeline
d33bs Jan 5, 2024
97319d9
move dockerfile again
d33bs Jan 5, 2024
94e8ecc
adjust pipeline for multiple input
d33bs Jan 5, 2024
67a4cbb
linting
d33bs Jan 5, 2024
48c3ca2
move test; add fire; adjust for multiple input
d33bs Jan 5, 2024
359f181
add pandas; add debug_leak test; linting cfg
d33bs Jan 5, 2024
13cec46
remove pandas; rename basic tests
d33bs Jan 5, 2024
e421a1b
add callbacks to gc test
d33bs Jan 5, 2024
6f1f94f
update docs
d33bs Jan 5, 2024
74eaab6
updates for linting
d33bs Jan 5, 2024
33aeb5f
update to show the stdout from container
d33bs Jan 6, 2024
d96c844
change dockerfile path
d33bs Jan 6, 2024
6252a98
move to solely list-based execution
d33bs Jan 7, 2024
d13775a
avoid the use of runner env group with tests
d33bs Jan 7, 2024
b39cd38
add poethepoet workflow for tests
d33bs Jan 7, 2024
eb03730
remove no longer used union
d33bs Jan 7, 2024
58438f0
remove print, adjust language
d33bs Jan 8, 2024
bc89a9e
simplify the prints for async ops
d33bs Jan 8, 2024
8cd67fc
add github actions workflow for running tests
d33bs Jan 8, 2024
df1585e
move to main for running tests
d33bs Jan 8, 2024
1559c6e
add guidance and overview in readme
d33bs Jan 8, 2024
bd43651
add python fire mentions
d33bs Jan 8, 2024
3fb9f37
set the platform of the container
d33bs Jan 8, 2024
43eef00
update naming and docs
d33bs Jan 8, 2024
e0236cc
typo
d33bs Jan 8, 2024
540694b
add poethepoet as a runner dependency
d33bs Jan 10, 2024
84feff0
bandit skip tests for assert linting
d33bs Jan 10, 2024
7f9bfee
update tests with asserts for garbage
d33bs Jan 10, 2024
df22eb9
add debug config; add lines for visual clarity
d33bs Jan 10, 2024
63a35fe
expand readme with some definitions + new sections
d33bs Jan 10, 2024
13c962c
add memory allocation definitions
d33bs Jan 13, 2024
9e9a1b9
assumption about python memory
d33bs Jan 13, 2024
9d4f689
add more detail to python garbage collection
d33bs Jan 13, 2024
ddcef59
reorganizing and adding test module documentation
d33bs Jan 14, 2024
49e6f03
add extra cli arg capabilities
d33bs Jan 14, 2024
f9f57b7
reorganize for better flow; specify allocators
d33bs Jan 15, 2024
96a90f4
further details on allocators
d33bs Jan 17, 2024
8b73e4c
add diagrams
d33bs Jan 17, 2024
b1c20da
additional diagrams, specificity
d33bs Jan 18, 2024
76ce4d1
add blocks content
d33bs Jan 19, 2024
33487e6
Update README.md
d33bs Jan 19, 2024
0b5c92e
modifications; adding reference counting guide
d33bs Jan 19, 2024
b44e161
add analogy; memray; begin scalene content
d33bs Jan 21, 2024
7689ba6
add scalene content
d33bs Jan 21, 2024
4aad156
remove content and images ported to blog post
d33bs Jan 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/run-memory-tests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
# used for running memory checks through GitHub Actions for open observability
name: run memory tests

on:
push:
branches: [main]
pull_request:
branches: [main]

jobs:
run_memory_tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Setup poetry and poethepoet
run: |
python -m pip install poetry poethepoet
- name: Install poetry runner env
run: |
poetry install --with runner -v --no-interaction
- name: Run memory tests through Dagger pipeline
run: |
poetry poe run_all_tests
61 changes: 60 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,62 @@
# demo-python-memory-reference-accounting
# Demonstrating Python Memory Allocator Reference Counting (and Related)

Demonstrating Python memory allocator reference counting and debug.
The goal of this repository is to help demonstrate Python memory allocation using source controlled code.
Work for this project was originally inspired by [cytomining/CytoTable#75](https://github.com/cytomining/CytoTable/issues/75) which explores related topics.

## Project outline

```mermaid
---
title: Project Overview
---
flowchart LR

pyproject["pyproject.toml"]
poeworkflow["poethepoet\nworkflows"]
dockerfile["Dockerfile"]
subgraph test_container["test container"]
subgraph tests
test_env["Python environment\n(for tests)"]
test_modules["Test module(s)"]
end
end
subgraph runner
runner_env["Python environment\n(for runner)"]
runner_cli["Runner CLI\n(Python Fire)"]
pipeline["Container pipeline\n(through Dagger)"]
end


pyproject --> |defines| test_env
pyproject --> |defines| runner_env

test_env --> |runs| test_modules
test_modules -.-> |display\nresults through| pipeline
runner_env --> |enables| runner_cli
runner_cli --> |runs| pipeline

dockerfile -.-> |defines\ncontainer for| pipeline

pipeline --> |executes| test_container

pyproject --> poeworkflow
poeworkflow -.-> |declarative\nworkflows for| runner
```

See above for a quick overview of project components and their relationship.
[Poetry](https://python-poetry.org/docs/) is used to define Python environment dependencies within [dependency groups](https://python-poetry.org/docs/master/managing-dependencies/#dependency-groups) in a `pyproject.toml` file.
Declarative [Poe the Poet tasks](https://poethepoet.natn.io/index.html) may also be found in the same `pyproject.toml` file to help define reproducible workflows.
A "runner" command-line interface (CLI) is provided through [Python Fire](https://github.com/google/python-fire) to help enable the use of the container-based pipelines.
Container-based pipelines are provided through [Dagger's Python SDK](https://docs.dagger.io/sdk/python/) to help isolate potential OS-specific distinctions for memory allocation work in Python.
Testing workflows are designed to run "locally" within a developer's environment (for example, leveraging [pyenv](https://github.com/pyenv/pyenv), [poetry](https://python-poetry.org/docs/), and [Docker Desktop](https://www.docker.com/products/docker-desktop/)) or within [GitHub Actions images](https://github.com/actions/runner-images) (`dagger-io` installs the necessary dependencies).

## Development

The following are suggested steps to get started with development for this project.

1. Install Python from [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation)
1. [Install Poetry](https://python-poetry.org/docs/#installation)
1. [Install Poe the Poet](https://poethepoet.natn.io/installation.html)
1. [Install Docker Desktop](https://www.docker.com/products/docker-desktop/)
1. Run Poe the Poet workflow(s): e.g. `poetry poe run_all_tests`
151 changes: 1 addition & 150 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 13 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,22 @@ packages = [{include = "pymaccounter", from="src"}]

[tool.poetry.dependencies]
python = ">=3.10,<4.0"

[tool.poetry.group.runner.dependencies]
dagger-io = "^0.9.5"
anyio = "^4.2.0"
fire = "^0.5.0"
pandas = "^2.1.4"

# define poethepoet convenience access
# to test runner workflow definition
[tool.poe.tasks.run_all_tests]
cmd = """
python src/pymaccounter/runner.py \
'["test_baseline.py", \
"test_multiply_gc.collect.py", \
"test_multiply_gc.debug_leak.py", \
"test_multiply_gc.callbacks.py"]'
"""

[build-system]
requires = ["poetry-core"]
Expand Down
2 changes: 1 addition & 1 deletion build/docker/Dockerfile → src/pymaccounter/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ RUN pip install --no-cache-dir poetry
COPY . .

# install env from poetry
RUN poetry install -v --no-interaction
RUN poetry install --without runner -v --no-interaction

# set an alias for running python through the poetry env
# hadolint ignore=DL3059
Expand Down
2 changes: 1 addition & 1 deletion src/pymaccounter/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
__init__.py for pymaccounter
"""

from .runner import run_test
from .runner import pipeline_run_tests
Loading