Add a CLI, add some light unit testing, and change config from Python to Yaml/Json #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Also deletes a duplicate method and refreshes pyproject.toml
CLI: - New `chart-review` script gets installed along with Python module. - One sub-command right now: `accuracy` which calculates accuracy matrixes across labels for two reviewers and a base third Config: - Switch away from Python config files and towards yaml/json files. - I've added yaml versions of the two studies in the repo, as examples.
This will make it easier for someone just using the python to call it, if they want to.
mikix
commented
Oct 12, 2023
| @@ -0,0 +1,41 @@ | |||
| """Methods for high-level accuracy calculations.""" | |||
There was a problem hiding this comment.
This file is basically a generic version of the calculation in paper.py
comorbidity
reviewed
Oct 13, 2023
| Chart Review operates on a project folder that holds your config & data. | ||
| 1. Make a new folder. | ||
| 2. Export your Label Studio annotations and put that in the folder as `labelstudio-export.json`. | ||
| 3. Add a `config.yaml` file (or `config.json`) that looks something like this (read more on this format below): |
There was a problem hiding this comment.
I like your idea and I strongly prefer config.json over yaml
There was a problem hiding this comment.
Ah fair - but note:
- Json is technically a subset of yaml. (That is, a yaml parser can also read json)
- So what I've done here is use a yaml parser and look for both
config.yamlandconfig.json-- it will read either one - The reason I personally prefer yaml for config files is that you can have comments, which are often very useful for explaining why a config is the way it is (and also json can be annoyingly fussy about stuff like trailing commas, but that's less important than the comments thing)
So the way I made this PR, either yaml or json works - whichever the researcher in question is more comfy with.
How do you feel about that? (Or do you feel like standardizing on a specific syntax is worth disallowing yaml?)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
OK on my first pass at interacting with the chart-review code, I wrote down some possible improvements here: #1
This is the PR to solve them. Specifically:
accuracywhich does the 3-way accuracy calculation frompaper.pyin the covid folder. 🤷 it seemed like a reasonable place to start, but we should talk about what top-level operations make sense.config.jsonwill work too.I've split this PR up into different commits, which should hopefully make it easier to read. But there's still a fair bit.