Merge branch 'devel' into numpy-v2

.github/workflows/benchmark.yml

@@ -13,7 +13,12 @@ jobs:
       uses: actions/setup-python@v5
       with:
         python-version: 3.12
-    - run: curl -LsSf https://astral.sh/uv/install.sh | sh
+    - uses: astral-sh/setup-uv@v3
+      with:
+        enable-cache: true
+        cache-dependency-glob: |
+          **/requirements*.txt
+          **/pyproject.toml
     - name: Install dependencies
       run: uv pip install --system .[test,amber,ase,pymatgen,benchmark] rdkit openbabel-wheel
     - name: Run benchmarks

.github/workflows/test.yml

@@ -18,7 +18,13 @@ jobs:
       uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
-    - run: curl -LsSf https://astral.sh/uv/install.sh | sh
+    - uses: astral-sh/setup-uv@v3
+      with:
+        enable-cache: true
+        cache-dependency-glob: |
+          **/requirements*.txt
+          **/pyproject.toml
+        cache-suffix: "py${{ matrix.python-version }}"
     - name: Install dependencies
       run: uv pip install --system .[test,amber,ase,pymatgen] coverage ./tests/plugin rdkit openbabel-wheel
     - name: Test

.pre-commit-config.yaml

@@ -2,7 +2,7 @@
 # See https://pre-commit.com/hooks.html for more hooks
 repos:
 -   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.6.0
+    rev: v5.0.0
     hooks:
     # there are many log files in tests
     # TODO: seperate py files and log files
@@ -21,7 +21,7 @@ repos:
 # Python
 -   repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: v0.5.6
+    rev: v0.7.1
     hooks:
     - id: ruff
       args: ["--fix"]
@@ -36,7 +36,7 @@ repos:
       args: ["--write"]
 # Python inside docs
 -   repo: https://github.com/asottile/blacken-docs
-    rev: 1.18.0
+    rev: 1.19.1
     hooks:
     -   id: blacken-docs
 ci:

.readthedocs.yaml

@@ -3,7 +3,7 @@ version: 2
 build:
   os: ubuntu-22.04
   tools:
-    python: "mambaforge-22.9"
+    python: "mambaforge-23.11"
 conda:
   environment: docs/rtd_environment.yml
 sphinx:

docs/conf.py

@@ -44,12 +44,12 @@
 # ones.
 extensions = [
     "deepmodeling_sphinx",
-    "sphinx_rtd_theme",
+    "sphinx_book_theme",
     "sphinx.ext.mathjax",
     "sphinx.ext.viewcode",
     "sphinx.ext.intersphinx",
     "numpydoc",
-    "m2r2",
+    "myst_parser",
     "sphinxarg.ext",
     "jupyterlite_sphinx",
 ]
@@ -87,7 +87,7 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "sphinx_rtd_theme"
+html_theme = "sphinx_book_theme"
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the

docs/index.rst

@@ -6,22 +6,23 @@
 Welcome to dpdata's documentation!
 ==================================
 
+dpdata is a Python package for manipulating atomistic data of software in computational science.
+
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
 
-   Overview <self>
+   installation
+   systems/index
    try_dpdata
    cli
    formats
    drivers
    minimizers
+   plugin
    api/api
    credits
 
-.. mdinclude:: ../README.md
-
-
 Indices and tables
 ==================
 

docs/installation.md

@@ -0,0 +1,13 @@
+# Installation
+
+DP-GEN only supports Python 3.7 and above. You can [setup a conda/pip environment](https://docs.deepmodeling.com/faq/conda.html), and then use one of the following methods to install DP-GEN:
+
+- Install via pip: `pip install dpdata`
+- Install via conda: `conda install -c conda-forge dpdata`
+- Install from source code: `git clone https://github.com/deepmodeling/dpdata && pip install ./dpdata`
+
+To test if the installation is successful, you may execute
+
+```bash
+dpdata --version
+```

docs/plugin.md

@@ -0,0 +1,9 @@
+# Plugins
+
+One can follow a simple example under `plugin_example/` directory to add their own format by creating and installing plugins.
+It's critical to add the {class}`Format <dpdata.format.Format>` class to `entry_points['dpdata.plugins']` in `pyproject.toml`:
+
+```toml
+[project.entry-points.'dpdata.plugins']
+random = "dpdata_random:RandomFormat"
+```

docs/rtd_environment.yml

@@ -2,6 +2,7 @@ name: dpdata
 channels:
   - conda-forge
 dependencies:
-  - mamba
+  - python <3.13
+  - mamba <2
   - pip:
     - ..[docs]

docs/systems/bond_order_system.md

@@ -0,0 +1,51 @@
+
+## BondOrderSystem
+A new class {class}`BondOrderSystem <dpdata.BondOrderSystem>` which inherits from class {class}`System <dpdata.System>` is introduced in dpdata. This new class contains information of chemical bonds and formal charges (stored in `BondOrderSystem.data['bonds']`, `BondOrderSystem.data['formal_charges']`). Now BondOrderSystem can only read from .mol/.sdf formats, because of its dependency on rdkit (which means rdkit must be installed if you want to use this function). Other formats, such as pdb, must be converted to .mol/.sdf format (maybe with software like open babel).
+```python
+import dpdata
+
+system_1 = dpdata.BondOrderSystem(
+    "tests/bond_order/CH3OH.mol", fmt="mol"
+)  # read from .mol file
+system_2 = dpdata.BondOrderSystem(
+    "tests/bond_order/methane.sdf", fmt="sdf"
+)  # read from .sdf file
+```
+In sdf file, all molecules must be of the same topology (i.e. conformers of the same molecular configuration).
+`BondOrderSystem <dpdata.BondOrderSystem>` also supports initialize from a {class}`rdkit.Chem.rdchem.Mol` object directly.
+```python
+from rdkit import Chem
+from rdkit.Chem import AllChem
+import dpdata
+
+mol = Chem.MolFromSmiles("CC")
+mol = Chem.AddHs(mol)
+AllChem.EmbedMultipleConfs(mol, 10)
+system = dpdata.BondOrderSystem(rdkit_mol=mol)
+```
+
+### Bond Order Assignment
+The {class}`BondOrderSystem <dpdata.BondOrderSystem>` implements a more robust sanitize procedure for rdkit Mol, as defined in {class}`dpdata.rdkit.santizie.Sanitizer`. This class defines 3 level of sanitization process by: low, medium and high. (default is medium).
++ low: use `rdkit.Chem.SanitizeMol()` function to sanitize molecule.
++ medium: before using rdkit, the programm will first assign formal charge of each atom to avoid inappropriate valence exceptions. However, this mode requires the rightness of the bond order information in the given molecule.
++ high: the program will try to fix inappropriate bond orders in aromatic hetreocycles, phosphate, sulfate, carboxyl, nitro, nitrine, guanidine groups. If this procedure fails to sanitize the given molecule, the program will then try to call `obabel` to pre-process the mol and repeat the sanitization procedure. **That is to say, if you wan't to use this level of sanitization, please ensure `obabel` is installed in the environment.**
+According to our test, our sanitization procedure can successfully read 4852 small molecules in the PDBBind-refined-set. It is necessary to point out that the in the molecule file (mol/sdf), the number of explicit hydrogens has to be correct. Thus, we recommend to use
+ `obabel xxx -O xxx -h` to pre-process the file. The reason why we do not implement this hydrogen-adding procedure in dpdata is that we can not ensure its correctness.
+
+```python
+import dpdata
+
+for sdf_file in glob.glob("bond_order/refined-set-ligands/obabel/*sdf"):
+    syst = dpdata.BondOrderSystem(sdf_file, sanitize_level="high", verbose=False)
+```
+### Formal Charge Assignment
+BondOrderSystem implement a method to assign formal charge for each atom based on the 8-electron rule (see below). Note that it only supports common elements in bio-system: B,C,N,O,P,S,As
+```python
+import dpdata
+
+syst = dpdata.BondOrderSystem("tests/bond_order/CH3NH3+.mol", fmt="mol")
+print(syst.get_formal_charges())  # return the formal charge on each atom
+print(syst.get_charge())  # return the total charge of the system
+```
+
+If a valence of 3 is detected on carbon, the formal charge will be assigned to -1. Because for most cases (in alkynyl anion, isonitrile, cyclopentadienyl anion), the formal charge on 3-valence carbon is -1, and this is also consisent with the 8-electron rule.

docs/systems/index.rst

@@ -0,0 +1,11 @@
+Systems
+=======
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   system
+   multi
+   bond_order_system
+   mixed

docs/systems/mixed.md

@@ -0,0 +1,24 @@
+# Mixed Type Format
+
+The format `deepmd/npy/mixed` is the mixed type numpy format for DeePMD-kit, and can be loaded or dumped through class {class}`dpdata.MultiSystems`.
+
+Under this format, systems with the same number of atoms but different formula can be put together
+for a larger system, especially when the frame numbers in systems are sparse.
+
+This also helps to mixture the type information together for model training with type embedding network.
+
+Here are examples using `deepmd/npy/mixed` format:
+
+- Dump a MultiSystems into a mixed type numpy directory:
+```python
+import dpdata
+
+dpdata.MultiSystems(*systems).to_deepmd_npy_mixed("mixed_dir")
+```
+
+- Load a mixed type data into a MultiSystems:
+```python
+import dpdata
+
+dpdata.MultiSystems().load_systems_from_file("mixed_dir", fmt="deepmd/npy/mixed")
+```

docs/systems/multi.md

@@ -0,0 +1,62 @@
+# `MultiSystems`
+
+The Class {class}`dpdata.MultiSystems`  can read data  from a dir which may contains many files of different systems, or from single xyz file which contains different systems.
+
+Use {meth}`dpdata.MultiSystems.from_dir` to read from a  directory, {class}`dpdata.MultiSystems` will walk in the directory
+Recursively  and  find all file with specific file_name. Supports all the file formats that {class}`dpdata.LabeledSystem` supports.
+
+Use {meth}`dpdata.MultiSystems.from_file` to read from single file. Single-file support is available for the `quip/gap/xyz` and `ase/structure` formats.
+
+For example, for `quip/gap xyz` files, single .xyz file may contain many different configurations with different atom numbers and atom type.
+
+The following commands relating to {class}`dpdata.MultiSystems` may be useful.
+```python
+# load data
+
+xyz_multi_systems = dpdata.MultiSystems.from_file(
+    file_name="tests/xyz/xyz_unittest.xyz", fmt="quip/gap/xyz"
+)
+vasp_multi_systems = dpdata.MultiSystems.from_dir(
+    dir_name="./mgal_outcar", file_name="OUTCAR", fmt="vasp/outcar"
+)
+
+# use wildcard
+vasp_multi_systems = dpdata.MultiSystems.from_dir(
+    dir_name="./mgal_outcar", file_name="*OUTCAR", fmt="vasp/outcar"
+)
+
+# print the multi_system infomation
+print(xyz_multi_systems)
+print(xyz_multi_systems.systems)  # return a dictionaries
+
+# print the system infomation
+print(xyz_multi_systems.systems["B1C9"].data)
+
+# dump a system's data to ./my_work_dir/B1C9_raw folder
+xyz_multi_systems.systems["B1C9"].to_deepmd_raw("./my_work_dir/B1C9_raw")
+
+# dump all systems
+xyz_multi_systems.to_deepmd_raw("./my_deepmd_data/")
+```
+
+You may also use the following code to parse muti-system:
+```python
+from dpdata import LabeledSystem, MultiSystems
+from glob import glob
+
+"""
+process multi systems
+"""
+fs = glob("./*/OUTCAR")  # remeber to change here !!!
+ms = MultiSystems()
+for f in fs:
+    try:
+        ls = LabeledSystem(f)
+    except:
+        print(f)
+    if len(ls) > 0:
+        ms.append(ls)
+
+ms.to_deepmd_raw("deepmd")
+ms.to_deepmd_npy("deepmd")
+```