Merge pull request #10 from broadinstitute/ccds

shntnu · web-flow · commit 05c0ae4a268d · 2025-08-04T16:23:27.000-04:00
Create new workflow standard + resources
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1 @@
+handbook.md
diff --git a/.markdownlint.yaml b/.markdownlint.yaml
@@ -0,0 +1,8 @@
+MD007:
+  indent: 4          # List indent
+MD013: false         # Line length
+MD024: false         # Multiple headers with same content
+MD029:
+  style: ordered     # Ordered list style
+MD033: false         # Inline HTML
+MD046: false         # Code block style
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -3,13 +3,23 @@ repos:
   rev: v5.0.0
   hooks:
   - id: trailing-whitespace
+    exclude: \.dvc$
   - id: check-added-large-files
-    args: ['--maxkb=10240']
+    args: [--maxkb=10240]      # 10MB limit
+  - id: check-yaml
+  - id: end-of-file-fixer
+    exclude: \.dvc$
+
 - repo: https://github.com/astral-sh/ruff-pre-commit
   rev: v0.12.7
   hooks:
-    - id: ruff
-      types_or: [ python, pyi ]
-      args: [ --fix ]
-    - id: ruff-format
-      types_or: [ python, pyi ]
+  - id: ruff
+    args: [--fix]
+  - id: ruff-format
+
+- repo: https://github.com/igorshubovych/markdownlint-cli
+  rev: v0.45.0
+  hooks:
+  - id: markdownlint
+    args: [--fix]
+    files: \.(md|markdown)$
diff --git a/.ruff.toml b/.ruff.toml
diff --git a/01-lab-notebook.md b/01-lab-notebook.md
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,95 @@
+# CLAUDE.md
+
+Guidance for Claude Code when working in this repository.
+
+## Repository Context
+
+Documentation repository for Carpenter-Singh Lab at the Broad Institute. Maintains lab standards for data science workflows using Cookiecutter Data Science (CCDS) principles.
+
+## Key Tasks When Working Here
+
+### 1. Documentation Improvements
+
+- Check for consistency in command examples across documents
+- Ensure all code blocks are properly formatted and executable
+- Verify that file paths and project names use consistent placeholder syntax (e.g., `{PROJECT_NAME}`)
+- Look for outdated dependencies or version numbers that may need updating
+
+### 2. Common Updates
+
+When asked to update documentation:
+
+- Follow the Documentation Guidelines below
+- Focus on executable commands over explanations
+
+### 3. Adding New Sections
+
+When adding new documentation:
+
+- Follow the existing markdown structure and heading hierarchy
+- Place new content in the appropriate file (workflows.md for process-related content)
+- Use real-world examples from the referenced repositories when possible
+
+### 4. Quality Checks
+
+Before finalizing any changes:
+
+- Ensure all bash commands use proper escaping and line continuations
+- Verify that Python/YAML/TOML code blocks have correct syntax
+- Check that any new commands follow the established patterns in the document
+
+## Repository Structure
+
+```text
+carpenter-singh-lab-standards/
+├── README.md          # Main entry point, links to other docs
+├── workflows.md       # Comprehensive CCDS workflow guide
+└── CLAUDE.md         # This file
+```
+
+## When Making Changes
+
+- workflows.md is the authoritative source for lab procedures
+- New documentation must complement, not duplicate, existing content
+- Make processes clear and executable, not flexible
+
+## Documentation Guidelines
+
+When updating documentation in this repository, follow these principles:
+
+### Target Audience
+
+- **Expert data scientists** who are new to the lab
+- They will be instructed to read documentation thoroughly
+- They want to understand THE way this lab operates, not explore options
+- Assume technical competence - skip basic explanations
+
+### Writing Style
+
+- **Prescriptive over descriptive**: "Do X" not "You might consider X"
+- **Dense over verbose**: One clear statement beats three explanatory paragraphs
+- **Executable over abstract**: Every process needs exact, working commands
+- **Real over generic**: Use actual filenames from real projects, not placeholders
+
+### Content Organization
+
+- **Role-based sections**: Clearly separate what analysts vs maintainers need
+- **Most-used first**: Daily workflow before one-time setup
+- **Progressive disclosure**: Core workflow in main text, edge cases in appendix
+- **Decision trees**: "If X → do Y" for any branching logic
+
+### Critical Elements to Include
+
+- **Coordination requirements**: Bold warnings when team sync is needed
+- **Order dependencies**: Explicit when sequence matters (e.g., hook installation)
+- **Known gotchas**: Document issues with specific workarounds
+- **The "why" sparingly**: Only when it affects implementation choices
+
+### What to Avoid
+
+- Beginner explanations ("Git is a version control system...")
+- Multiple options without clear guidance
+- Philosophical discussions about methodology
+- Untested command sequences
+
+Remember: Documentation should be the authoritative reference that turns an expert data scientist into an expert lab member.
diff --git a/README.md b/README.md
@@ -1,2 +1,17 @@
-# Carpenter-Singh Lab Standards
+# Carpenter-Singh Lab Guide
 
+Resources and standards for the Carpenter-Singh Lab at the Broad Institute.
+
+## Documentation
+
+### Project Organization
+
+- **[Project Workflows](workflows.md)** - How we organize and run data science projects using Cookiecutter Data Science principles
+
+### Learning Resources
+
+- **[Resources](resources.md)** - Scientific literature, tools, and educational materials for image-based profiling
+
+## Contributing
+
+This is a living document. Please contribute additional guides, patterns, and standards as we develop them.
diff --git a/resources.md b/resources.md
@@ -0,0 +1,70 @@
+# Resources
+
+## Educational Resources
+
+### Computing Skills
+
+- [The Missing Semester of Your CS Education](https://missing.csail.mit.edu/)
+- [The Unix Workbench](https://www.coursera.org/learn/unix)
+- [Data Science Cheatsheet](https://github.com/aaronwangy/Data-Science-Cheatsheet)
+
+### Textbooks
+
+- [Data Analysis for the Life Sciences](https://leanpub.com/dataanalysisforthelifesciences)
+
+## Scientific Literature
+
+### Image-based Profiling Essentials
+
+- [Image-based profiling: due for a machine-learning upgrade?](https://www.nature.com/articles/s41573-020-00117-w) - 2020 review of applications in image-based profiling
+- [Data-analysis strategies for image-based cell profiling](https://www.nature.com/articles/nmeth.4397) - Introduces key analysis steps
+- [Applications in image-based profiling of perturbations](https://www.sciencedirect.com/science/article/pii/S0958166916301112) - Describes common applications
+- [Cell Painting: a decade of discovery and innovation in cellular imaging](https://www.nature.com/articles/s41592-024-02528-8) - Systematic review of Cell Painting
+- [Cell Painting protocol](https://www.nature.com/articles/s41596-023-00840-9) - Detailed protocol
+
+## Analysis Tools & Libraries
+
+### Profiling Libraries
+
+- **pycytominer**: Core library for image-based profiling ([GitHub](https://github.com/cytomining/pycytominer))
+- **copairs**: Evaluation metrics for profiling ([GitHub](https://github.com/cytomining/copairs))
+- More tools at [https://github.com/cytomining/](https://github.com/cytomining/)
+
+### Deep Learning
+
+- **DeepProfiler**: Deep learning for image-based profiling ([GitHub](https://github.com/cytomining/DeepProfiler))
+
+### Learning Resources
+
+- [Profiling Handbook](https://cytomining.github.io/profiling-handbook/) - Step-by-step instructions for producing image-based profiles from raw images
+- [LINCS Workflow](https://github.com/broadinstitute/lincs-cell-painting/tree/e9737c3e4e4443eb03c2c278a145f12efe255756/profiles#workflow) - Workflow diagram of image-based profiling
+
+## Development Environment Setup
+
+### Package Management
+
+- Install applications using [brew](https://brew.sh/)
+- Use [uv](https://github.com/astral-sh/uv) for Python environment management
+- Install R following [these instructions](https://github.com/broadinstitute/imaging-configs/blob/master/R-instructions.md)
+
+### Git Configuration
+
+```bash
+# Configure shared repositories
+git config --global core.sharedRepository true
+```
+
+**Best practice**: Avoid [amending](https://stackoverflow.com/questions/253055/how-do-i-push-amended-commit-to-the-remote-git-repository) pushed commits
+
+## General Principles
+
+### Troubleshooting
+
+Follow the 30-minute rule: Try to solve issues independently for ~30 minutes before asking for help.
+
+1. Search existing resources (documentation, issues, archives)
+2. Consult official documentation for relevant tools/libraries
+3. Consider using AI tools (ChatGPT, Claude) with appropriate caution
+4. Google error messages or problem descriptions
+5. Post in relevant channels with details of what you've already tried
+6. Document your troubleshooting process for future reference
diff --git a/workflows.md b/workflows.md
