[perf] Port more pre-commit hooks

[lmmx]

I don't see a checklist tracking issue for the remaining pre-commit hooks, but some like check-toml have been implemented and gone from sub-second to instant, so this would likely contribute a further speedup on many existing pre-commit configs.

• Docs: Currently implemented hooks
  • Source: docs/builtin.md

It looks like the specific site of the fast path redirection for these hooks is here in src/builtin/pre_commit_hooks/mod.rs

To support this, I would compare the list in the docs against this repo, specifically in this hooks YAML

pre-commit-hooks

• check-added-large-files
• check-ast
• check-byte-order-marker
• check-builtin-literals
• check-case-conflict
• check-docstring-first
• check-executables-have-shebangs
• check-illegal-windows-names
• check-json
• check-shebang-scripts-are-executable
• pretty-format-json
• check-merge-conflict
• check-symlinks
• check-toml
• check-vcs-permalinks
• check-xml
• check-yaml
• debug-statements
• destroyed-symlinks
• detect-aws-credentials
• detect-private-key
• double-quote-string-fixer
• end-of-file-fixer
• file-contents-sorter
• fix-byte-order-marker
• fix-encoding-pragma
• forbid-new-submodules
• forbid-submodules
• mixed-line-ending
• name-tests-test
• no-commit-to-branch
• requirements-txt-fixer
• sort-simple-yaml
• trailing-whitespace

[j178]

Here is a tracking issue #17

I don't think porting all the pre-commit-hooks is really our goal, so for now I've only ported some of the most commonly used ones.

Some of them just aren't worth it - either they're barely used or too niche. Others are too tightly coupled with Python and can't be implemented natively in Rust, like check-ast and check-builtin-literals. There might still be a few others worth porting, but I may want to wait until we get actual feature requests for them.

[lmmx]

Aha I missed that, thanks. Maybe this issue is still useful as a checklist of coverage?

Appreciate there are a fair number here but some do reliably save my bacon, I'd be interested to take on the debug-statements one for example.

I think the ones that just check text files (like debug-statements would be easier to include). check-ast would require a AST parser (but there are a few now like the one in ruff maybe, ruff_python_ast ruff_python_parser ?

edit here's a standalone rust-script demo of what that would look like: and in fact this being a git-only dependency might actually be viable here since it's not for publication to crates.io (due to the existing path dep)

[lmmx]

In terms of their relative popularity, we can get a sense of that from grep.app (which indexes popular repos, I think above some minimum number of stars, maybe 5)

If I put the results in order of hits on grep.app it does indeed look like the most widely used ones are the ones that've been ported so far :-) Maybe this can help prioritise their development:

• trailing-whitespace 15k
• end-of-file-fixer 15k
• check-yaml 12k
• check-merge-conflict 7.8k
  • feat(check-merge-conflict): implement builtin hook #885
• check-added-large-files 7k
• mixed-line-ending 5k
• debug-statements 5k
  • feat(debug-statements): implement builtin hook #886
• check-toml 5k
• check-json 4.9k
• check-case-conflict 4.8k
  • feat(check-case-conflict): implement builtin hook #888
• detect-private-key 3.3k
  • Implement detect-private-key as builtin hook #893
• check-symlinks 2.8k
  • Implement check-symlinks as builtin hook #895
• check-executables-have-shebangs 2.7k
  • Implement check-executables-have-shebangs as builtin-hook #924 by @Felix-Blom
• check-ast 2.7k
  • feat(check-ast): implement builtin hook #884
• check-docstring-first 2.4k
• requirements-txt-fixer 2.3k
• check-xml 1.6k
  • Implement check-xml as builtin hook #894
• no-commit-to-branch 1.2k
  • Implement no-commit-to-branch as builtin hook #930 by @fllesser
• fix-byte-order-marker 1.2k
• pretty-format-json 1k
  • Implement pretty_format_json as builtin hook #915 by @Felix-Blom
• fix-encoding-pragma 1k
• check-shebang-scripts-are-executable 1k
• name-tests-test 0.8k
• double-quote-string-fixer 0.8k
• destroyed-symlinks 0.7k
• check-builtin-literals 0.7k
  • feat(check-builtin-literals): implement builtin hook #890
• check-byte-order-marker 0.6k
• detect-aws-credentials 0.5k
• check-vcs-permalinks 0.5k
• forbid-new-submodules 0.4k
• sort-simple-yaml 0.2k
• file-contents-sorter 0.2k
• forbid-submodules 0.1k
• check-illegal-windows-names 0.09k

[lmmx]

Another good way to prioritise would be to aim for the ones being used by early adopters of prek

• apache/iceberg-python uses prek in its Makefile but these are all covered or in PRs 👍
• apache/airflow uses:
  • detect-private-key
  • check-builtin-literals
  • check-executables-have-shebangs
  • check-xml
  • pretty-format-json
• goldziher/kreuzberg uses:
  • detect-private-key

prek runs on Airflow in the following order (slowest to fastest):

• check-builtin-literals 0.72s
• check-xml 0.23s
• check-executables-have-shebangs 0.18s
• detect-private-key 0.11s
• pretty-format-json 0.05s

[lmmx]

Arg, sorry @j178 my finger slipped and I clicked "Comment and close issue"! Please reopen 🤦‍♂️