diff --git a/.travis.yml b/.travis.yml index f73b75720..b0963bb18 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,12 +1,10 @@ language: python python: - - "3.6.3" - - "3.6-dev" # 3.6 development branch + - "3.6" + - "3.7" # command to install dependencies env: matrix: - - PYMATGEN_VERSION=2017.9.1 - - PYMATGEN_VERSION=2018.1.19 - PYMATGEN_VERSION=2019.1.13 - PYMATGEN_VERSION=2019.7.30 before_install: diff --git a/README.md b/README.md index 3ebd81725..d9fed8fe8 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,10 @@ ## About DP-GEN + +[![GitHub release](https://img.shields.io/github/release/deepmodeling/dpgen.svg?maxAge=86400)](https://github.com/deepmodeling/dpgen/releases/) +[![arxiv:1910.12690](http://img.shields.io/badge/arXiv-1910.12690-B31B1B.svg?maxAge=86400)](https://arxiv.org/abs/1910.12690) + DP-GEN (Deep Generator) is a software written in Python, delicately designed to generate a deep learning based model of interatomic potential energy and force field. DP-GEN is depedent on DeepMD-kit (https://github.com/deepmodeling/deepmd-kit/blob/master/README.md). With highly scalable interface with common softwares for molecular simulation, DP-GEN is capable to automatically prepare scripts and maintain job queues on HPC machines (High Performance Cluster) and analyze results ### Highlighted features + **Accurate and efficient**: DP-GEN is capable to sample more than tens of million structures and select only a few for first principles calculation. DP-GEN will finally obtain a uniformly accurate model. @@ -58,14 +62,14 @@ One can download the source code of dpgen by ```bash git clone https://github.com/deepmodeling/dpgen.git ``` -then you may install DP-GEN easily by: +then you may install DP-GEN easily by: ```bash cd dpgen pip install --user . ``` With this command, the dpgen executable is install to `$HOME/.local/bin/dpgen`. You may want to export the `PATH` by ```bash -export PATH=$HOME/.local/bin/dpgen:$PATH +export PATH=$HOME/.local/bin:$PATH ``` To test if the installation is successful, you may execute ```bash @@ -123,7 +127,7 @@ You may prepare initial data for bulk systems with VASP by: ```bash dpgen init_bulk PARAM [MACHINE] ``` -The MACHINE configure file is optional. If this parameter exists, then the optimization +The MACHINE configure file is optional. If this parameter exists, then the optimization tasks or MD tasks will be submitted automatically according to MACHINE.json. Basically `init_bulk` can be devided into four parts , denoted as `stages` in `PARAM`: @@ -273,7 +277,7 @@ The bold notation of key (such as **Elements**) means that it's a necessary key. | **layer_numb** | Integer | 3 | Number of equavilent layers of slab. | **vacuum_max** | Float | 9 | Maximal thickness of vacuum (Angstrom). | **vacuum_resol** | List of float | [0.5, 1 ] | Interval of thichness of vacuum. If size of `vacuum_resol` is 1, the interval is fixed to its value. If size of `vacuum_resol` is 2, the interval is `vacuum_resol[0]` before `mid_point`, otherwise `vacuum_resol[1]` after `mid_point`. -| **millers** | List of list of Integer | [[1,0,0]] | Miller indices. +| **millers** | List of list of Integer | [[1,0,0]] | Miller indices. | relax_incar | String | "....../INCAR" | Path of INCAR for relaxation in VASP. **Necessary** if `stages` include 1. | **scale** | List of float | [0.980, 1.000, 1.020] | Scales for transforming cells. | **skip_relax** | Boolean | False | If it's true, you may directly run stage 2 (pertub and scale) using an unrelaxed POSCAR. @@ -506,8 +510,45 @@ The bold notation of key (such aas **type_map**) means that it's a necessary key |**fp_params["kspacing"]** | Float| 0.4 | Sample factor in Brillouin zones. |**fp_params["mixingweight"]** | Float| 0.05 | Proportion a of output Density Matrix to be used for the input Density Matrix of next SCF cycle (linear mixing). |**fp_params["NumberPulay"]** | Integer| 5 | Controls the Pulay convergence accelerator. +| *fp_style == cp2k* +| **fp_params** | Dict | |Parameters for cp2k calculation. find detail in manual.cp2k.org. only the kind section must be set before use. we assume that you have basic knowledge for cp2k input. + + +#### Rules for cp2k input at dictionary form + Converting cp2k input is very simple as dictionary used to dpgen input. You just need follow some simple rule: +- kind section parameter must be provide +- replace `keyword` in cp2k as `keyword` in dict. +- replace `keyword parameter` in cp2k as `value` in dict. +- replace `section name` in cp2k as `keyword` in dict. . The corresponding value is a `dict`. +- repalce `section parameter` in cp2k as `value` with dict. keyword `"_"` +- `repeat section` in cp2k just need to be written once with repeat parameter as list. +Here are examples for setting: + ```python + + #minimal information you should provide for input + #other we have set other parameters in code, if you want to + #use your own paramter, just write a corresponding dictionary + "user_fp_params": { + "FORCE_EVAL":{ + "DFT":{ + "BASIS_SET_FILE_NAME": "path", + "POTENTIAL_FILE_NAME": "path" + } + "SUBSYS":{ + "KIND":{ + "_": ["N","C","H"], + "POTENTIAL": ["GTH-PBE-q5","GTH-PBE-q4", "GTH-PBE-q1"], + "BASIS_SET": ["DZVP-MOLOPT-GTH","DZVP-MOLOPT-GTH","DZVP-MOLOPT-GTH"] + } + } + } + } +``` + + ## Test: Auto-test for Deep Generator +### configure and param.json At this step, we assume that you have prepared some graph files like `graph.*.pb` and the particular pseudopotential `POTCAR`. The main code of this step is @@ -529,6 +570,8 @@ The whole program contains a series of tasks shown as follows. In each task, the + `05.surf`: the surface formation energy +Dpgen auto_test will auto make dir for each task it tests, the dir name is the same as the dir name. And the test results will in a plain text file named result. For example `cat ./01.eos/Al/std-fcc/deepmd/result` + We take Al as an example to show the parameter settings of `param.json`. The first part is the fundamental setting for particular alloy system. @@ -542,6 +585,7 @@ The first part is the fundamental setting for particular alloy system. "task_type":"deepmd", "task":"eos", ``` + You need to add the specified paths of necessary `POTCAR` files in "potcar_map". The different `POTCAR` paths are separated by commas. Then you also need to add the folder path of particular configuration, which contains `POSCAR` file. ``` @@ -578,12 +622,67 @@ The second part is the computational settings for vasp and lammps. According to }, ``` The last part is the optional settings for various tasks mentioned above. You can change the parameters according to actual needs. + + +param.json in a dictionary. + +| Fields | Type | Example | Discription | +| :---------------- | :--------------------- | :------------- | :----------------| +| potcar_map | dict | {"Al": "example/POTCAR"} |a dict like { "element" : "position of POTCAR" } | +| conf_dir | path like string | "confs/Al/std-fcc" | the dir which contains vasp's POSCAR | +| key_id | string| "DZIwdXCXg1fiXXXXXX" |the API key of Material project| +| task_type | string | "vasp" | task type, one of deepmd vasp meam | +| task | string | "equi" | task, one of equi, eos, elastic, vacancy, interstitial, surf or all | +| vasp_params| dict | seeing below | params relating to vasp INCAR| +| lammps_params | dict| seeing below| params relating to lammps | + +The keys in param["vasp_params"] is shown below. + +| Fields | Type | Example | Discription | +| :---------------- | :--------------------- | :---------------- | :----------------| +| ecut | real number | 650 | the plane wave cutoff for grid. | +| ediff | real number | 1e-6 |Tolerance of Density Matrix | +| kspacing | real number | 0.1 | Sample factor in Brillouin zones | +| kgamma | boolen | false | whether generate a Gamma centered grid | +| npar | positive integer | 1 | the number of k-points that are to be treated in parallel | +| kpar | positive integer | 1 | the number of bands that are treated in parallel | + +the keys in param["lammps_params"]. + +| Key | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| model_dir | path like string | "example/Al_model" | the model dir which contains .pb file | +| type_map | list of string | ["Al"] | a list contains the element, usually useful for multiple element situation | +| model_name | boolean | false | | +| model_param_type | boolean | false | | + +### auto_test tasks +#### 00.equi ```json "_comment":"00.equi", "store_stable":true, ``` + `store_stable`:(boolean) whether to store the stable energy and volume +param.json. + +| Field | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| EpA(eV) | real number | -3.7468 | the potential energy of a atom| +| VpA(A^3)| real number | 16.511| theEquilibrium volume of a atom | + +test results +``` +conf_dir: EpA(eV) VpA(A^3) +confs/Al/std-fcc -3.7468 16.511 +``` + +| Field | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| EpA(eV) | real number | -3.7468 | the potential energy of a atom| +| VpA(A^3)| real number | 16.511| theEquilibrium volume of a atom | + +#### 01.eos ```json "_comment": "01.eos", "vol_start": 12, @@ -592,6 +691,23 @@ The last part is the optional settings for various tasks mentioned above. You ca ``` + `vol_start`, `vol_end` and `vol_step` determine the volumetric range and accuracy of the **eos**. +test results +``` +conf_dir:confs/Al/std-fcc +VpA(A^3) EpA(eV) +15.500 -3.7306 +16.000 -3.7429 +16.500 -3.7468 +17.000 -3.7430 +``` + + +| Field | Type| Example| Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| EpA(eV) | list of real number | [15.5,16.0,16.5,17.0] | the potential energy of a atom in quilibrium state| +| VpA(A^3)| list of real number |[-3.7306, -3.7429, -3.746762, -3.7430] | the equilibrium volume of a atom | + +#### 02.elastic ```json "_comment": "02.elastic", "norm_deform": 2e-2, @@ -600,11 +716,61 @@ The last part is the optional settings for various tasks mentioned above. You ca + `norm_deform` and `shear_deform` are the scales of material deformation. This task uses the stress-strain relationship to calculate the elastic constant. +|Key | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| norm_deform | real number | 0.02 | uniaxial deformation range | +| shear_deform | real number | 0.05| shear deformation range | + +test results +``` +conf_dir:confs/Al/std-fcc +130.50 57.45 54.45 4.24 0.00 0.00 +57.61 130.31 54.45 -4.29 -0.00 -0.00 +54.48 54.48 133.32 -0.00 -0.00 -0.00 +4.49 -4.02 -0.89 33.78 0.00 -0.00 +-0.00 -0.00 -0.00 -0.00 33.77 4.29 +0.00 -0.00 -0.00 -0.00 4.62 36.86 +# Bulk Modulus BV = 80.78 GPa +# Shear Modulus GV = 36.07 GPa +# Youngs Modulus EV = 94.19 GPa +# Poission Ratio uV = 0.31 +``` + +| Field | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| elastic module(GPa)| 6*6 matrix of real number| [[130.50 57.45 54.45 4.24 0.00 0.00] [57.61 130.31 54.45 -4.29 -0.00 -0.00] [54.48 54.48 133.32 -0.00 -0.00 -0.00] [4.49 -4.02 -0.89 33.78 0.00 -0.00] [-0.00 -0.00 -0.00 -0.00 33.77 4.29] [0.00 -0.00 -0.00 -0.00 4.62 36.86]]| Voigt-notation elastic module;sequence of row and column is (xx, yy, zz, yz, zx, xy)| +| bulk modulus(GPa) | real number | 80.78 | bulk modulus | +| shear modulus(GPa) | real number | 36.07 | shear modulus | +| Youngs Modulus(GPa) | real number | 94.19 | Youngs Modulus| +| Poission Ratio | real number | 0.31 | Poission Ratio | + + + +#### 03.vacancy ```json "_comment":"03.vacancy", "supercell":[3,3,3], ``` + `supercell`:(list of integer) the supercell size used to generate vacancy defect and interstitial defect + +|Key | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| supercell | list of integer | [3,3,3] | the supercell size used to generate vacancy defect and interstitial defect | + +test result +``` +conf_dir:confs/Al/std-fcc +Structure: Vac_E(eV) E(eV) equi_E(eV) +struct-3x3x3-000: 0.859 -96.557 -97.416 +``` +| Field | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +|Structure| list of string |['struct-3x3x3-000'] | structure name| +| Vac_E(eV) | real number |0.723 | the vacancy formation energy | +| E(eV) | real number | -96.684 | potential energy of the vacancy configuration | +| equi_E(eV) | real number |-97.407 | potential energy of the equilibrium state| + +#### 04.interstitial ```json "_comment":"04.interstitial", "insert_ele":["Al"], @@ -613,6 +779,27 @@ This task uses the stress-strain relationship to calculate the elastic constant. + `insert_ele`:(list of string) the elements used to generate point interstitial defect + `repord-opt`:(boolean) whether to reproduce trajectories of interstitial defect +|Key | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| insert_ele | list of string | ["Al"] | the elements used to generate point interstitial defect | +| reprod-opt | boolean | false | whether to reproduce trajectories of interstitial defect| + +test result +``` +conf_dir:confs/Al/std-fcc +Insert_ele-Struct: Inter_E(eV) E(eV) equi_E(eV) +struct-Al-3x3x3-000: 3.919 -100.991 -104.909 +struct-Al-3x3x3-001: 2.681 -102.229 -104.909 +``` +| Field | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +|Structure| string |'struct-Al-3x3x3-000' | structure name| +| Inter_E(eV) | real number |0.723 | the interstitial formation energy | +| E(eV) | real number | -96.684 | potential energy of the interstitial configuration | +| equi_E(eV) | real number |-97.407 | potential energy of the equilibrium state| + +#### 05.surface + ```json "_comment": "05.surface", "min_slab_size": 10, @@ -629,7 +816,158 @@ This task uses the stress-strain relationship to calculate the elastic constant. + `static-opt`:(boolean) whether to use atomic relaxation to compute surface energy. if false, the structure will be relaxed. + `relax_box`:(boolean) set true if the box is relaxed, otherwise only relax atom positions. +|Key | Type | Example | Discription | +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +| min_slab_size| real number| 10 | the minimum size of slab thickness | +|min_vacuum_size | real number| 11 | the minimum size of the vacuume width | +|pert_xz | real number| 0.01 | the perturbation through xz direction used to compute surface energy | +|max_miller | integer| 2 | the maximum miller index | +|static-opt|boolean| false | whether to use atomic relaxation to compute surface energy. if false, the structure will be relaxed. | +|relax_box | boolean | false | set true if the box is relaxed, otherwise only relax atom positions | + +test result +``` +conf_dir:confs/Al/std-fcc +Miller_Indices: Surf_E(J/m^2) EpA(eV) equi_EpA(eV) +struct-000-m1.1.1m: 0.673 -3.628 -3.747 +struct-001-m2.2.1m: 0.917 -3.592 -3.747 +``` +| Field | Type | Example| Discription| +| :---------------- | :--------------------- | :-------------------------------------- | :-------------------------------------------------------------| +|Miller_Indices| string | struct-000-m1.1.1m | Miller Indices| +|Surf_E(J/m^2)| real number | 0.673 | the surface formation energy | +| EpA(eV) | real number | -3.628 | potential energy of the surface configuration | +| equi_EpA | real number | -3.747 | potential energy of the equilibrium state| + +### The content of the auto_test +To know what actually will dpgen autotest do, including the lammps and vasp script, the input file and atom configuration file auto_test will generate, please refer to https://hackmd.io/@yeql5ephQLaGJGgFgpvIDw/rJY1FO92B + +## Simplify +When you have a dataset containing lots of repeated data, this step will help you simplify your dataset. The workflow contains three stages: train, model_devi, and fp. The train stage and the fp stage are as the same as the run step, and the model_devi stage will calculate model deviations of the rest data that has not been confirmed accurate. Data with small model deviations will be confirmed accurate, while the program will pick data from those with large model deviations to the new dataset. + +Use the following script to start the workflow: +```bash +dpgen simplify param.json machine.json +``` + +Here is an example of `param.json` for QM7 dataset: +```json +{ + "type_map": [ + "C", + "H", + "N", + "O", + "S" + ], + "mass_map": [ + 12.011, + 1.008, + 14.007, + 15.999, + 32.065 + ], + "pick_data": "/scratch/jz748/simplify/qm7", + "init_data_prefix": "", + "init_data_sys": [], + "sys_batch_size": [ + "auto" + ], + "numb_models": 4, + "train_param": "input.json", + "default_training_param": { + "model": { + "type_map": [ + "C", + "H", + "N", + "O", + "S" + ], + "descriptor": { + "type": "se_a", + "sel": [ + 7, + 16, + 3, + 3, + 1 + ], + "rcut_smth": 1.00, + "rcut": 6.00, + "neuron": [ + 25, + 50, + 100 + ], + "resnet_dt": false, + "axis_neuron": 12 + }, + "fitting_net": { + "neuron": [ + 240, + 240, + 240 + ], + "resnet_dt": true + } + }, + "learning_rate": { + "type": "exp", + "start_lr": 0.001, + "decay_steps": 10, + "decay_rate": 0.99 + }, + "loss": { + "start_pref_e": 0.02, + "limit_pref_e": 1, + "start_pref_f": 1000, + "limit_pref_f": 1, + "start_pref_v": 0, + "limit_pref_v": 0, + "start_pref_pf": 0, + "limit_pref_pf": 0 + }, + "training": { + "set_prefix": "set", + "stop_batch": 10000, + "disp_file": "lcurve.out", + "disp_freq": 1000, + "numb_test": 1, + "save_freq": 1000, + "save_ckpt": "model.ckpt", + "load_ckpt": "model.ckpt", + "disp_training": true, + "time_training": true, + "profiling": false, + "profiling_file": "timeline.json" + }, + "_comment": "that's all" + }, + "use_clusters": true, + "fp_style": "gaussian", + "shuffle_poscar": false, + "fp_task_max": 1000, + "fp_task_min": 10, + "fp_pp_path": "/home/jzzeng/", + "fp_pp_files": [], + "fp_params": { + "keywords": "mn15/6-31g** force nosymm scf(maxcyc=512)", + "nproc": 28, + "multiplicity": 1, + "_comment": " that's all " + }, + "init_pick_number":100, + "iter_pick_number":100, + "e_trust_lo":1e10, + "e_trust_hi":1e10, + "f_trust_lo":0.25, + "f_trust_hi":0.45, + "_comment": " that's all " +} +``` +Here `pick_data` is the data to simplify and currently only supports `MultiSystems` containing `System` with `deepmd/npy` format, and `use_clusters` should always be `true`. `init_pick_number` and `iter_pick_number` are the numbers of picked frames. `e_trust_lo`, `e_trust_hi` mean the range of the deviation of the frame energy, and `f_trust_lo` and `f_trust_hi` mean the range of the max deviation of atomic forces in a frame. `fp_style` can only be `gaussian` currently. Other parameters are as the same as those of generator. ## Set up machine @@ -741,7 +1079,10 @@ The following table gives explicit descriptions on keys in param.json. | # Followings are keys in resources | numb_node | Integer | 1 | Node count required for the job | task_per_node | Integer | 4 | Number of CPU cores required -| `numb_gpu` | Integer | 4 | Number of GPUs required +| numb_gpu | Integer | Integer | 4 | Number of GPUs required +| manual_cuda_devices | Interger | 1 | Used with key "manual_cuda_multiplicity" specify the gpu number +| manual_cuda_multiplicity |Interger | 5 | Used in 01.model_devi,used with key "manual_cuda_devices" specify the MD program number running on one GPU at the same time,dpgen will automatically allocate MD jobs on different GPU. This can improve GPU usage for GPU like V100. +| node_cpu | Integer | 4 | Only for LSF. The number of CPU cores on each node that should be allocated to the job. | source_list | List of string | "....../vasp.env" | Environment needed for certain job. For example, if "env" is in the list, 'source env' will be written in the script. | module_list | List of string | [ "Intel/2018", "Anaconda3"] | For example, If "Intel/2018" is in the list, "module load Intel/2018" will be written in the script. | partition | String | "AdminGPU" | Partition / queue in which to run the job. | @@ -749,10 +1090,10 @@ The following table gives explicit descriptions on keys in param.json. mem_limit | Interger | 16 | Maximal memory permitted to apply for the job. | with_mpi | Boolean | true | Deciding whether to use mpi for calculation. If it's true and machine type is Slurm, "srun" will be prefixed to `command` in the script. | qos | "string"| "bigdata" | Deciding priority, dependent on particular settings of your HPC. +| allow_failure | Boolean | false | Allow the command to return a non-zero exit code. | # End of resources | command | String | "lmp_serial" | Executable path of software, such as `lmp_serial`, `lmp_mpi` and `vasp_gpu`, `vasp_std`, etc. | group_size | Integer | 5 | DP-GEN will put these jobs together in one submitting script. -| allow_failure | Boolean | false | Allow the command to return a non-zero exit code. ## Troubleshooting 1. The most common problem is whether two settings correspond with each other, including: @@ -763,7 +1104,7 @@ mem_limit | Interger | 16 | Maximal memory permitted to apply for the job. - Index of `sys_configs` and `sys_idx` 2. Please verify the directories of `sys_configs`. If there isnt's any POSCAR for `01.model_devi` in one iteration, it may happen that you write the false path of `sys_configs`. -3. Correct format of JSON file. +3. Correct format of JSON file. 4. In `02.fp`, total cores you require through `task_per_node` should be devided by `npar` times `kpar`. 5. The frames of one system should be larger than `batch_size` and `numb_test` in `default_training_param`. It happens that one iteration adds only a few structures and causes error in next iteration's training. In this condition, you may let `fp_task_min` be larger than `numb_test`. ## License diff --git a/dpgen/auto_test/cmpt_00_equi.py b/dpgen/auto_test/cmpt_00_equi.py index a716eb0c6..cffa8faf5 100755 --- a/dpgen/auto_test/cmpt_00_equi.py +++ b/dpgen/auto_test/cmpt_00_equi.py @@ -21,40 +21,36 @@ def comput_e_shift(poscar, task_name) : ener_shift += a_natoms[ii] * ener return ener_shift -def comput_lmp_nev(conf_dir, task_name, write_stable = False) : +def comput_lmp_nev(conf_dir, task_name,write_shift = False) : conf_path = re.sub('confs', global_equi_name, conf_dir) conf_path = os.path.abspath(conf_path) poscar = os.path.join(conf_path, 'POSCAR') - if write_stable : - ele_types = vasp.get_poscar_types(poscar) - if len(ele_types) > 1 : - raise RuntimeError('stable energy and volume only for one element, current you have %s from POSCAR' % str(ele_types)) - ener_shift = comput_e_shift(poscar, task_name) + ele_types = vasp.get_poscar_types(poscar) lmp_path = os.path.join(conf_path, task_name) log_lammps = os.path.join(lmp_path, 'log.lammps') if os.path.isfile(log_lammps): natoms, epa, vpa = lammps.get_nev(log_lammps) - epa = (epa * natoms - ener_shift) / natoms - if write_stable : + if write_shift and len(ele_types)>1: + ener_shift = comput_e_shift(poscar, task_name) + shift = (epa * natoms - ener_shift) / natoms + return natoms,epa,vpa,shift + if len(ele_types)==1: stable_dir = 'stables' os.makedirs(stable_dir, exist_ok=True) name_prefix=os.path.join(stable_dir,'%s.%s' % (ele_types[0], task_name)) open(name_prefix + '.e', 'w').write('%.16f\n' % (epa)) open(name_prefix + '.v', 'w').write('%.16f\n' % (vpa)) - return natoms, epa, vpa + return natoms, epa, vpa , None else : - return None, None, None + return None, None, None, None + +def comput_vasp_nev(jdata, conf_dir, write_shift = False) : -def comput_vasp_nev(jdata, conf_dir, write_stable = False) : - conf_path = re.sub('confs', global_equi_name, conf_dir) conf_path = os.path.abspath(conf_path) poscar = os.path.join(conf_path, 'POSCAR') - if write_stable : - ele_types = vasp.get_poscar_types(poscar) - if len(ele_types) > 1 : - raise RuntimeError('stable energy and volume only for one element, current you have %s from POSCAR' % str(ele_types)) + ele_types = vasp.get_poscar_types(poscar) if 'relax_incar' in jdata.keys(): vasp_str='vasp-relax_incar' @@ -62,7 +58,6 @@ def comput_vasp_nev(jdata, conf_dir, write_stable = False) : kspacing = jdata['vasp_params']['kspacing'] vasp_str='vasp-k%.2f' % kspacing - ener_shift = comput_e_shift(poscar, vasp_str) vasp_path = os.path.join(conf_path, vasp_str) outcar = os.path.join(vasp_path, 'OUTCAR') # tag_fin = os.path.join(vasp_path, 'tag_finished') @@ -72,22 +67,25 @@ def comput_vasp_nev(jdata, conf_dir, write_stable = False) : warnings.warn("incomplete job "+vasp_path+" use the last frame") if os.path.isfile(outcar): natoms, epa, vpa = vasp.get_nev(outcar) - epa = (epa * natoms - ener_shift) / natoms - if write_stable : + if write_shift and len(ele_types)>1: + ener_shift = comput_e_shift(poscar, vasp_str) + shift = (epa * natoms - ener_shift) / natoms + return natoms,epa,vpa,shift + if len(ele_types)==1: stable_dir = 'stables' os.makedirs(stable_dir, exist_ok=True) name_prefix=os.path.join(stable_dir,'%s.'% (ele_types[0])+vasp_str) open(name_prefix + '.e', 'w').write('%.16f\n' % (epa)) open(name_prefix + '.v', 'w').write('%.16f\n' % (vpa)) - return natoms, epa, vpa + return natoms, epa, vpa, None else : - return None, None, None + return None, None, None, None def _main(): parser = argparse.ArgumentParser( description="cmpt 00.equi") - parser.add_argument('TASK', type=str, - choices = ['all', 'vasp', 'deepmd', 'meam'], + parser.add_argument('TASK', type=str, + choices = ['all', 'vasp', 'deepmd', 'meam'], help='the task of generation, vasp or lammps') parser.add_argument('PARAM', type=str, help='the json param') @@ -120,4 +118,3 @@ def _main(): if __name__ == '__main__' : _main() - diff --git a/dpgen/auto_test/cmpt_02_elastic.py b/dpgen/auto_test/cmpt_02_elastic.py index fc3480110..b1a0c9229 100755 --- a/dpgen/auto_test/cmpt_02_elastic.py +++ b/dpgen/auto_test/cmpt_02_elastic.py @@ -104,7 +104,7 @@ def cmpt_deepmd_lammps(jdata, conf_dir, task_name) : # et = -et / 1e4 print_et(et) result = os.path.join(task_path,'result') - result_et(et,conf_dir,task_path) + result_et(et,conf_dir,result) if 'upload_username' in jdata.keys() and task_name=='deepmd': upload_username=jdata['upload_username'] util.insert_data('elastic','deepmd',upload_username,result) diff --git a/dpgen/auto_test/cmpt_03_vacancy.py b/dpgen/auto_test/cmpt_03_vacancy.py index 0c3c201e5..a53602d38 100755 --- a/dpgen/auto_test/cmpt_03_vacancy.py +++ b/dpgen/auto_test/cmpt_03_vacancy.py @@ -23,7 +23,7 @@ def comput_e_shift(poscar, task_name) : ref_e_file = os.path.join('stables', ref_e_file) ener = float(open(ref_e_file).read()) ener_shift += a_natoms[ii] * ener - return ener_shift + return ener_shift def cmpt_vasp(jdata, conf_dir, supercell) : @@ -37,7 +37,7 @@ def cmpt_vasp(jdata, conf_dir, supercell) : equi_path = os.path.join(equi_path, vasp_str) equi_path = os.path.abspath(equi_path) equi_outcar = os.path.join(equi_path, 'OUTCAR') - task_path = re.sub('confs', global_task_name, conf_dir) + task_path = re.sub('confs', global_task_name, conf_dir) task_path = os.path.join(task_path, vasp_str) task_path = os.path.abspath(task_path) print("# ", task_path) @@ -51,17 +51,23 @@ def cmpt_vasp(jdata, conf_dir, supercell) : if len(struct_path_list) == 0: print("# cannot find results for conf %s supercell %s" % (conf_dir, supercell)) sys.stdout.write ("Structure: \tVac_E(eV) E(eV) equi_E(eV)\n") - for ii in struct_path_list : - struct_poscar = os.path.join(ii, 'POSCAR') - energy_shift = comput_e_shift(struct_poscar, vasp_str) - structure_dir = os.path.basename(ii) - outcar = os.path.join(ii, 'OUTCAR') - natoms, epa, vpa = vasp.get_nev(outcar) - evac = epa * natoms - equi_epa * natoms - sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa*natoms)) + result = os.path.join(task_path,'result') + with open(result,'w') as fp: + fp.write('conf_dir:%s\n'% (conf_dir)) + fp.write("Structure: \tVac_E(eV) E(eV) equi_E(eV)\n") + for ii in struct_path_list : + struct_poscar = os.path.join(ii, 'POSCAR') + #energy_shift = comput_e_shift(struct_poscar, vasp_str) + structure_dir = os.path.basename(ii) + outcar = os.path.join(ii, 'OUTCAR') + natoms, epa, vpa = vasp.get_nev(outcar) + evac = epa * natoms - equi_epa * natoms + sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa*natoms)) + fp.write("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa*natoms)) + fp.close() # evac = epa * natoms - energy_shift # sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, energy_shift)) - # sys.stdout.write ("%s: %7.3f \n" % (structure_dir, evac)) + # sys.stdout.write ("%s: %7.3f \n" % (structure_dir, evac)) def cmpt_deepmd_lammps(jdata, conf_dir, supercell, task_name) : equi_path = re.sub('confs', global_equi_name, conf_dir) @@ -82,14 +88,20 @@ def cmpt_deepmd_lammps(jdata, conf_dir, supercell, task_name) : if len(struct_path_list) == 0: print("# cannot find results for conf %s supercell %s" % (conf_dir, supercell)) sys.stdout.write ("Structure: \tVac_E(eV) E(eV) equi_E(eV)\n") - for ii in struct_path_list : - struct_poscar = os.path.join(ii, 'POSCAR') - energy_shift = comput_e_shift(struct_poscar, task_name) - structure_dir = os.path.basename(ii) - lmp_log = os.path.join(ii, 'log.lammps') - natoms, epa, vpa = lammps.get_nev(lmp_log) - evac = epa * natoms - equi_epa * natoms - sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa * natoms)) + result = os.path.join(task_path,'result') + with open(result,'w') as fp: + fp.write('conf_dir:%s\n'% (conf_dir)) + fp.write("Structure: \tVac_E(eV) E(eV) equi_E(eV)\n") + for ii in struct_path_list : + struct_poscar = os.path.join(ii, 'POSCAR') + #energy_shift = comput_e_shift(struct_poscar, task_name) + structure_dir = os.path.basename(ii) + lmp_log = os.path.join(ii, 'log.lammps') + natoms, epa, vpa = lammps.get_nev(lmp_log) + evac = epa * natoms - equi_epa * natoms + sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa * natoms)) + fp.write("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa*natoms)) + fp.close() # evac = epa * natoms - energy_shift # sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, energy_shift)) # sys.stdout.write ("%s: %7.3f\n" % (structure_dir, evac)) @@ -112,15 +124,13 @@ def _main() : # print('# generate %s task with conf %s' % (args.TASK, args.CONF)) if args.TASK == 'vasp': - cmpt_vasp(jdata, args.CONF, args.COPY) + cmpt_vasp(jdata, args.CONF, args.COPY) elif args.TASK == 'deepmd' : cmpt_deepmd_lammps(jdata, args.CONF, args.COPY, args.TASK) elif args.TASK == 'meam' : cmpt_deepmd_lammps(jdata, args.CONF, args.COPY, args.TASK) else : raise RuntimeError("unknow task ", args.TASK) - + if __name__ == '__main__' : _main() - - diff --git a/dpgen/auto_test/cmpt_04_interstitial.py b/dpgen/auto_test/cmpt_04_interstitial.py index c87ef741e..6b029909c 100755 --- a/dpgen/auto_test/cmpt_04_interstitial.py +++ b/dpgen/auto_test/cmpt_04_interstitial.py @@ -41,13 +41,19 @@ def _cmpt_vasp(jdata, conf_dir, supercell, insert_ele) : struct_path_list.sort() if len(struct_path_list) == 0: print("# cannot find results for conf %s supercell %s" % (conf_dir, supercell)) - sys.stdout.write ("Insert_ele-Struct: Inter_E(eV)\n") - for ii in struct_path_list : - structure_dir = os.path.basename(ii) - outcar = os.path.join(ii, 'OUTCAR') - natoms, epa, vpa = vasp.get_nev(outcar) - evac = epa * natoms - equi_epa * natoms - sys.stdout.write ("%s: %7.3f \n" % (structure_dir, evac)) + sys.stdout.write ("Insert_ele-Struct: Inter_E(eV) E(eV) equi_E(eV)\n") + result = os.path.join(task_path,'result') + with open(result,'w') as fp: + fp.write('conf_dir:%s\n'% (conf_dir)) + fp.write ("Insert_ele-Struct: Inter_E(eV) E(eV) equi_E(eV)\n") + for ii in struct_path_list : + structure_dir = os.path.basename(ii) + outcar = os.path.join(ii, 'OUTCAR') + natoms, epa, vpa = vasp.get_nev(outcar) + evac = epa * natoms - equi_epa * natoms + sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa * natoms)) + fp.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa * natoms)) + fp.close() def cmpt_deepmd_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_name) : for ii in insert_ele: @@ -63,7 +69,7 @@ def _cmpt_deepmd_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_name) conf_path = os.path.abspath(conf_dir) task_path = re.sub('confs', global_task_name, conf_path) vasp_path = os.path.join(task_path, vasp_str) - lmps_path = os.path.join(task_path, task_name + vasp_str.replace('vasp','')) + lmps_path = os.path.join(task_path, task_name + vasp_str.replace('vasp','')) copy_str = "%sx%sx%s" % (supercell[0], supercell[1], supercell[2]) struct_widecard = os.path.join(vasp_path, 'struct-%s-%s-*' % (insert_ele,copy_str)) vasp_struct = glob.glob(struct_widecard) @@ -122,12 +128,18 @@ def _cmpt_deepmd_lammps(jdata, conf_dir, supercell, insert_ele, task_name) : if len(struct_path_list) == 0: print("# cannot find results for conf %s supercell %s" % (conf_dir, supercell)) sys.stdout.write ("Insert_ele-Struct: Inter_E(eV) E(eV) equi_E(eV)\n") - for ii in struct_path_list : - structure_dir = os.path.basename(ii) - lmp_log = os.path.join(ii, 'log.lammps') - natoms, epa, vpa = lammps.get_nev(lmp_log) - evac = epa * natoms - equi_epa * natoms - sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa * natoms)) + result = os.path.join(task_path,'result') + with open(result,'w') as fp: + fp.write('conf_dir:%s\n'% (conf_dir)) + fp.write ("Insert_ele-Struct: Inter_E(eV) E(eV) equi_E(eV)\n") + for ii in struct_path_list : + structure_dir = os.path.basename(ii) + lmp_log = os.path.join(ii, 'log.lammps') + natoms, epa, vpa = lammps.get_nev(lmp_log) + evac = epa * natoms - equi_epa * natoms + sys.stdout.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa * natoms)) + fp.write ("%s: %7.3f %7.3f %7.3f \n" % (structure_dir, evac, epa * natoms, equi_epa * natoms)) + fp.close() def _main() : parser = argparse.ArgumentParser( @@ -149,7 +161,7 @@ def _main() : # print('# generate %s task with conf %s' % (args.TASK, args.CONF)) if args.TASK == 'vasp': - cmpt_vasp(jdata, args.CONF, args.COPY, args.ELEMENT) + cmpt_vasp(jdata, args.CONF, args.COPY, args.ELEMENT) elif args.TASK == 'deepmd' : cmpt_deepmd_lammps(jdata, args.CONF, args.COPY, args.ELEMENT, args.TASK) elif args.TASK == 'deepmd-reprod' : @@ -160,8 +172,6 @@ def _main() : cmpt_deepmd_reprod_traj(jdata, args.CONF, args.COPY, args.ELEMENT, args.TASK) else : raise RuntimeError("unknow task ", args.TASK) - + if __name__ == '__main__' : _main() - - diff --git a/dpgen/auto_test/cmpt_05_surf.py b/dpgen/auto_test/cmpt_05_surf.py index 2cb1e40f2..fde28911d 100755 --- a/dpgen/auto_test/cmpt_05_surf.py +++ b/dpgen/auto_test/cmpt_05_surf.py @@ -96,7 +96,7 @@ def cmpt_deepmd_lammps(jdata, conf_dir, task_name, static = False) : sys.stdout.write ("%s: \t%7.3f %8.3f %8.3f\n" % (structure_dir, evac, epa, equi_epa)) fp.write("%s:\t %7.3f %8.3f %8.3f\n" % (structure_dir, evac, epa, equi_epa)) fp.close() - if 'upload_username' in jdata.keys() and task_name=='deepm': + if 'upload_username' in jdata.keys() and task_name=='deepmd': upload_username=jdata['upload_username'] util.insert_data('surf','deepmd',upload_username,result) diff --git a/dpgen/auto_test/gen_03_vacancy.py b/dpgen/auto_test/gen_03_vacancy.py index 56bb10b51..93b061ea9 100755 --- a/dpgen/auto_test/gen_03_vacancy.py +++ b/dpgen/auto_test/gen_03_vacancy.py @@ -73,7 +73,7 @@ def make_vasp(jdata, conf_dir, supercell = [1,1,1]) : for fname in potcar_list: with open(fname) as infile: outfile.write(infile.read()) - # gen tasks + # gen tasks copy_str = "%sx%sx%s" % (supercell[0], supercell[1], supercell[2]) cwd = os.getcwd() for ii in range(len(dss)) : @@ -94,11 +94,10 @@ def make_vasp(jdata, conf_dir, supercell = [1,1,1]) : os.chdir(cwd) def make_lammps(jdata, conf_dir, task_type, supercell) : - fp_params = jdata['vasp_params'] - kspacing = fp_params['kspacing'] + fp_params = jdata['lammps_params'] model_dir = fp_params['model_dir'] - type_map = fp_params['type_map'] + type_map = fp_params['type_map'] model_dir = os.path.abspath(model_dir) model_name =fp_params['model_name'] if not model_name and task_type =='deepmd': @@ -110,7 +109,7 @@ def make_lammps(jdata, conf_dir, task_type, supercell) : model_param = {'model_name' : fp_params['model_name'], 'param_type': fp_params['model_param_type']} - + ntypes = len(type_map) conf_path = os.path.abspath(conf_dir) @@ -120,7 +119,8 @@ def make_lammps(jdata, conf_dir, task_type, supercell) : if 'relax_incar' in jdata.keys(): vasp_str='vasp-relax_incar' else: - vasp_str='vasp-k%.2f' % kspacing + kspacing = jdata['vasp_params']['kspacing'] + vasp_str='vasp-k%.2f' % kspacing equi_path = os.path.join(equi_path, vasp_str) equi_contcar = os.path.join(equi_path, 'CONTCAR') assert os.path.exists(equi_contcar),"Please compute the equilibrium state using vasp first" @@ -146,28 +146,28 @@ def make_lammps(jdata, conf_dir, task_type, supercell) : dss = [] for jj in vds : dss.append(jj.generate_defect_structure(supercell)) - # gen tasks + # gen tasks cwd = os.getcwd() # make lammps.in, relax at 0 bar (scale = 1) if task_type=='deepmd': - fc = lammps.make_lammps_press_relax('conf.lmp', - ntypes, - 1, + fc = lammps.make_lammps_press_relax('conf.lmp', + ntypes, + 1, lammps.inter_deepmd, model_name) elif task_type =='meam': - fc = lammps.make_lammps_press_relax('conf.lmp', - ntypes, - 1, + fc = lammps.make_lammps_press_relax('conf.lmp', + ntypes, + 1, lammps.inter_meam, model_param) f_lammps_in = os.path.join(task_path, 'lammps.in') with open(f_lammps_in, 'w') as fp : fp.write(fc) - # gen tasks + # gen tasks copy_str = "%sx%sx%s" % (supercell[0], supercell[1], supercell[2]) cwd = os.getcwd() - + os.chdir(task_path) for ii in model_name : if os.path.exists(ii) : @@ -188,7 +188,7 @@ def make_lammps(jdata, conf_dir, task_type, supercell) : dss[ii].to('POSCAR', 'POSCAR') lammps.cvt_lammps_conf('POSCAR', 'conf.lmp') ptypes = vasp.get_poscar_types('POSCAR') - lammps.apply_type_map('conf.lmp', type_map, ptypes) + lammps.apply_type_map('conf.lmp', type_map, ptypes) # link lammps.in os.symlink(os.path.relpath(f_lammps_in), 'lammps.in') # link models @@ -197,7 +197,7 @@ def make_lammps(jdata, conf_dir, task_type, supercell) : # save supercell np.savetxt('supercell.out', supercell, fmt='%d') os.chdir(cwd) - + def _main() : parser = argparse.ArgumentParser( description="gen 03.vacancy") @@ -221,8 +221,6 @@ def _main() : make_lammps(jdata, args.CONF, args.TASK, args.COPY) else : raise RuntimeError("unknow task ", args.TASK) - + if __name__ == '__main__' : _main() - - diff --git a/dpgen/auto_test/gen_04_interstitial.py b/dpgen/auto_test/gen_04_interstitial.py index f423cbfb6..376be45c6 100755 --- a/dpgen/auto_test/gen_04_interstitial.py +++ b/dpgen/auto_test/gen_04_interstitial.py @@ -23,7 +23,7 @@ def _make_vasp(jdata, conf_dir, supercell, insert_ele) : # get equi poscar if 'relax_incar' in jdata.keys(): vasp_str='vasp-relax_incar' - else: + else: kspacing = jdata['vasp_params']['kspacing'] vasp_str='vasp-k%.2f' % (kspacing) equi_path = re.sub('confs', global_equi_name, conf_path) @@ -65,7 +65,7 @@ def _make_vasp(jdata, conf_dir, supercell, insert_ele) : fc = vasp.make_vasp_relax_incar(ecut, ediff, True, True, True, npar=npar,kpar=kpar, kspacing = kspacing, kgamma = kgamma) with open(os.path.join(task_path, 'INCAR'), 'w') as fp : fp.write(fc) - # gen tasks + # gen tasks copy_str = "%sx%sx%s" % (supercell[0], supercell[1], supercell[2]) cwd = os.getcwd() for ii in range(len(dss)) : @@ -82,7 +82,7 @@ def _make_vasp(jdata, conf_dir, supercell, insert_ele) : with open('POSCAR','r') as fp : lines = fp.read().split('\n') ele_list = lines[5].split() - + os.chdir(cwd) potcar_map = jdata['potcar_map'] potcar_list = [] @@ -90,12 +90,12 @@ def _make_vasp(jdata, conf_dir, supercell, insert_ele) : assert os.path.exists(os.path.abspath(potcar_map[ii])),"No POTCAR in the potcar_map of %s"%(ii) potcar_list.append(os.path.abspath(potcar_map[ii])) os.chdir(struct_path) - + with open('POTCAR', 'w') as outfile: for fname in potcar_list: with open(fname) as infile: outfile.write(infile.read()) - + # link incar os.symlink(os.path.relpath(os.path.join(task_path, 'INCAR')), 'INCAR') # save supercell @@ -103,15 +103,15 @@ def _make_vasp(jdata, conf_dir, supercell, insert_ele) : os.chdir(cwd) -def make_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_type) : +def make_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_type) : for ii in insert_ele : _make_reprod_traj(jdata, conf_dir, supercell, ii, task_type) -def _make_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_type) : - kspacing = jdata['vasp_params']['kspacing'] +def _make_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_type) : + fp_params = jdata['lammps_params'] model_dir = fp_params['model_dir'] - type_map = fp_params['type_map'] + type_map = fp_params['type_map'] model_dir = os.path.abspath(model_dir) model_name =fp_params['model_name'] if not model_name and task_type=='deepmd': @@ -128,24 +128,34 @@ def _make_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_type) : conf_path = os.path.abspath(conf_dir) task_path = re.sub('confs', global_task_name, conf_path) - vasp_path = os.path.join(task_path, 'vasp-k%.2f' % kspacing) - lmps_path = os.path.join(task_path, task_type + '-reprod-k%.2f' % kspacing) + if 'relax_incar' in jdata.keys(): + vasp_str='vasp-relax_incar' + lmps_str= task_type + '-reprod-relax_incar' + else: + kspacing = jdata['vasp_params']['kspacing'] + vasp_str = 'vasp-k%.2f' % (kspacing) + lmps_str = task_type + '-reprod-k%.2f' % (kspacing) + + vasp_path = os.path.join(task_path, vasp_str) + lmps_path = os.path.join(task_path, lmps_str) + os.makedirs(lmps_path, exist_ok = True) copy_str = "%sx%sx%s" % (supercell[0], supercell[1], supercell[2]) struct_widecard = os.path.join(vasp_path, 'struct-%s-%s-*' % (insert_ele,copy_str)) vasp_struct = glob.glob(struct_widecard) + assert len(vasp_struct)>0 ,"Please compute the interstitial defect using vasp first" vasp_struct.sort() cwd=os.getcwd() - + # make lammps.in if task_type =='deepmd': - fc = lammps.make_lammps_eval('conf.lmp', - ntypes, + fc = lammps.make_lammps_eval('conf.lmp', + ntypes, lammps.inter_deepmd, model_name) elif task_type =='meam': - fc = lammps.make_lammps_eval('conf.lmp', - ntypes, + fc = lammps.make_lammps_eval('conf.lmp', + ntypes, lammps.inter_meam, model_param) f_lammps_in = os.path.join(lmps_path, 'lammps.in') @@ -188,7 +198,7 @@ def _make_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_type) : for (ii,jj) in zip(models, model_name) : os.symlink(os.path.relpath(ii), jj) share_models = [os.path.join(ls,ii) for ii in model_name] - + # loop over frames for ii in range(xdat_nframes) : frame_path = 'frame.%06d' % ii @@ -197,10 +207,10 @@ def _make_reprod_traj(jdata, conf_dir, supercell, insert_ele, task_type) : # clear dir for jj in ['conf.lmp'] : if os.path.isfile(jj): - os.remove(jj) + os.remove(jj) for jj in ['lammps.in'] + model_name : if os.path.islink(jj): - os.unlink(jj) + os.unlink(jj) # link lammps in os.symlink(os.path.relpath('../lammps.in'), 'lammps.in') # make conf @@ -222,11 +232,10 @@ def make_lammps(jdata, conf_dir, supercell, insert_ele, task_type) : _make_lammps(jdata, conf_dir, supercell, ii, task_type) def _make_lammps(jdata, conf_dir, supercell, insert_ele, task_type) : - fp_params = jdata['vasp_params'] - kspacing = fp_params['kspacing'] + fp_params = jdata['lammps_params'] model_dir = fp_params['model_dir'] - type_map = fp_params['type_map'] + type_map = fp_params['type_map'] model_dir = os.path.abspath(model_dir) model_name =fp_params['model_name'] if not model_name and task_type=='deepmd': @@ -245,7 +254,12 @@ def _make_lammps(jdata, conf_dir, supercell, insert_ele, task_type) : conf_poscar = os.path.join(conf_path, 'POSCAR') # get equi poscar equi_path = re.sub('confs', global_equi_name, conf_path) - equi_path = os.path.join(equi_path, 'vasp-k%.2f' % kspacing) + if 'relax_incar' in jdata.keys(): + vasp_str='vasp-relax_incar' + else: + kspacing = jdata['vasp_params']['kspacing'] + vasp_str='vasp-k%.2f' % (kspacing) + equi_path = os.path.join(equi_path, vasp_str) equi_contcar = os.path.join(equi_path, 'CONTCAR') #equi_path = os.path.join(equi_path, task_type) #equi_dump = os.path.join(equi_path, 'dump.relax') @@ -269,25 +283,25 @@ def _make_lammps(jdata, conf_dir, supercell, insert_ele, task_type) : dss = [] for jj in vds : dss.append(jj.generate_defect_structure(supercell)) - # gen tasks + # gen tasks cwd = os.getcwd() # make lammps.in, relax at 0 bar (scale = 1) if task_type=='deepmd': - fc = lammps.make_lammps_press_relax('conf.lmp', - ntypes, + fc = lammps.make_lammps_press_relax('conf.lmp', + ntypes, 1, lammps.inter_deepmd, model_name) elif task_type =='meam': - fc = lammps.make_lammps_press_relax('conf.lmp', - ntypes, - 1, + fc = lammps.make_lammps_press_relax('conf.lmp', + ntypes, + 1, lammps.inter_meam, model_param) f_lammps_in = os.path.join(task_path, 'lammps.in') with open(f_lammps_in, 'w') as fp : fp.write(fc) - # gen tasks + # gen tasks copy_str = "%sx%sx%s" % (supercell[0], supercell[1], supercell[2]) cwd = os.getcwd() @@ -298,7 +312,7 @@ def _make_lammps(jdata, conf_dir, supercell, insert_ele, task_type) : for (ii,jj) in zip(models, model_name) : os.symlink(os.path.relpath(ii), jj) share_models = [os.path.join(task_path,ii) for ii in model_name] - + for ii in range(len(dss)) : struct_path = os.path.join(task_path, 'struct-%s-%s-%03d' % (insert_ele,copy_str,ii)) print('# generate %s' % (struct_path)) @@ -311,7 +325,7 @@ def _make_lammps(jdata, conf_dir, supercell, insert_ele, task_type) : dss[ii].to('POSCAR', 'POSCAR') lammps.cvt_lammps_conf('POSCAR', 'conf.lmp') ptypes = vasp.get_poscar_types('POSCAR') - lammps.apply_type_map('conf.lmp', type_map, ptypes) + lammps.apply_type_map('conf.lmp', type_map, ptypes) # link lammps.in os.symlink(os.path.relpath(f_lammps_in), 'lammps.in') # link models @@ -321,7 +335,7 @@ def _make_lammps(jdata, conf_dir, supercell, insert_ele, task_type) : np.savetxt('supercell.out', supercell, fmt='%d') os.chdir(cwd) - + def _main() : parser = argparse.ArgumentParser( description="gen 04.interstitial") @@ -350,8 +364,6 @@ def _main() : make_reprod_traj(jdata, args.CONF, args.COPY, args.ELEMENT, args.TASK) else : raise RuntimeError("unknow task ", args.TASK) - + if __name__ == '__main__' : _main() - - diff --git a/dpgen/auto_test/gen_05_surf.py b/dpgen/auto_test/gen_05_surf.py index 776a0205f..56a275630 100755 --- a/dpgen/auto_test/gen_05_surf.py +++ b/dpgen/auto_test/gen_05_surf.py @@ -12,9 +12,13 @@ def make_vasp(jdata, conf_dir, max_miller = 2, relax_box = False, static = False) : + min_slab_size = jdata['min_slab_size'] + min_vacuum_size = jdata['min_vacuum_size'] + pert_xz = jdata['pert_xz'] + if 'relax_incar' in jdata.keys(): vasp_str='vasp-relax_incar' - else: + else: kspacing = jdata['vasp_params']['kspacing'] vasp_str='vasp-k%.2f' % (kspacing) @@ -23,7 +27,7 @@ def make_vasp(jdata, conf_dir, max_miller = 2, relax_box = False, static = False # conf_poscar = os.path.join(conf_path, 'POSCAR') equi_path = re.sub('confs', global_equi_name, conf_dir) equi_path = os.path.join(equi_path, vasp_str) - equi_path = os.path.abspath(equi_path) + equi_path = os.path.abspath(equi_path) equi_contcar = os.path.join(equi_path, 'CONTCAR') assert os.path.exists(equi_contcar),"Please compute the equilibrium state using vasp first" task_path = re.sub('confs', global_task_name, conf_dir) @@ -64,9 +68,6 @@ def make_vasp(jdata, conf_dir, max_miller = 2, relax_box = False, static = False kpar = fp_params['kpar'] kspacing = fp_params['kspacing'] kgamma = fp_params['kgamma'] - min_slab_size = jdata['min_slab_size'] - min_vacuum_size = jdata['min_vacuum_size'] - pert_xz = jdata['pert_xz'] fc = vasp.make_vasp_static_incar(ecut, ediff, npar=npar,kpar=kpar, kspacing = kspacing, kgamma = kgamma) else : if 'relax_incar' in jdata.keys(): @@ -82,9 +83,6 @@ def make_vasp(jdata, conf_dir, max_miller = 2, relax_box = False, static = False kpar = fp_params['kpar'] kspacing = fp_params['kspacing'] kgamma = fp_params['kgamma'] - min_slab_size = jdata['min_slab_size'] - min_vacuum_size = jdata['min_vacuum_size'] - pert_xz = jdata['pert_xz'] fc = vasp.make_vasp_relax_incar(ecut, ediff, True, relax_box, False, npar=npar,kpar=kpar, kspacing = kspacing, kgamma = kgamma) with open(os.path.join(task_path, 'INCAR'), 'w') as fp : fp.write(fc) @@ -101,7 +99,7 @@ def make_vasp(jdata, conf_dir, max_miller = 2, relax_box = False, static = False for fname in potcar_list: with open(fname) as infile: outfile.write(infile.read()) - # gen tasks + # gen tasks cwd = os.getcwd() for ii in range(len(all_slabs)) : slab = all_slabs[ii] @@ -127,10 +125,10 @@ def make_vasp(jdata, conf_dir, max_miller = 2, relax_box = False, static = False cwd = os.getcwd() def make_lammps(jdata, conf_dir, max_miller = 2, static = False, relax_box = False, task_type = 'wrong-task') : - kspacing = jdata['vasp_params']['kspacing'] + fp_params = jdata['lammps_params'] model_dir = fp_params['model_dir'] - type_map = fp_params['type_map'] + type_map = fp_params['type_map'] model_dir = os.path.abspath(model_dir) model_name =fp_params['model_name'] if not model_name and task_type=='deepmd': @@ -153,13 +151,14 @@ def make_lammps(jdata, conf_dir, max_miller = 2, static = False, relax_box = Fal # conf_poscar = os.path.join(conf_path, 'POSCAR') if 'relax_incar' in jdata.keys(): vasp_str='vasp-relax_incar' - else: + else: + kspacing = jdata['vasp_params']['kspacing'] vasp_str='vasp-k%.2f' % (kspacing) equi_path = re.sub('confs', global_equi_name, conf_dir) equi_path = os.path.join(equi_path, vasp_str) - equi_path = os.path.abspath(equi_path) - equi_contcar = os.path.join(equi_path, 'CONTCAR') + equi_path = os.path.abspath(equi_path) + equi_contcar = os.path.join(equi_path, 'CONTCAR') assert os.path.exists(equi_contcar),"Please compute the equilibrium state using vasp first" task_path = re.sub('confs', global_task_name, conf_dir) task_path = os.path.abspath(task_path) @@ -182,27 +181,27 @@ def make_lammps(jdata, conf_dir, max_miller = 2, static = False, relax_box = Fal # make lammps.in if task_type =='deepmd': if static : - fc = lammps.make_lammps_eval('conf.lmp', - ntypes, + fc = lammps.make_lammps_eval('conf.lmp', + ntypes, lammps.inter_deepmd, model_name) else : - fc = lammps.make_lammps_equi('conf.lmp', - ntypes, + fc = lammps.make_lammps_equi('conf.lmp', + ntypes, lammps.inter_deepmd, - model_name, + model_name, change_box = relax_box) elif task_type =='meam': if static : - fc = lammps.make_lammps_eval('conf.lmp', - ntypes, + fc = lammps.make_lammps_eval('conf.lmp', + ntypes, lammps.inter_meam, model_param) else : - fc = lammps.make_lammps_equi('conf.lmp', - ntypes, + fc = lammps.make_lammps_equi('conf.lmp', + ntypes, lammps.inter_meam, - model_param, + model_param, change_box = relax_box) f_lammps_in = os.path.join(task_path, 'lammps.in') with open(f_lammps_in, 'w') as fp : @@ -233,7 +232,7 @@ def make_lammps(jdata, conf_dir, max_miller = 2, static = False, relax_box = Fal vasp.regulate_poscar('POSCAR', 'POSCAR') lammps.cvt_lammps_conf('POSCAR', 'conf.lmp') ptypes = vasp.get_poscar_types('POSCAR') - lammps.apply_type_map('conf.lmp', type_map, ptypes) + lammps.apply_type_map('conf.lmp', type_map, ptypes) # record miller np.savetxt('miller.out', slab.miller_index, fmt='%d') # link lammps.in @@ -273,8 +272,6 @@ def _main() : make_lammps(jdata, args.CONF, args.MAX_MILLER, static = True, relax_box = args.relax_box, task_type = args.TASK) else : raise RuntimeError("unknow task ", args.TASK) - + if __name__ == '__main__' : _main() - - diff --git a/dpgen/auto_test/lib/lammps.py b/dpgen/auto_test/lib/lammps.py index f3788067f..b6a21f44a 100644 --- a/dpgen/auto_test/lib/lammps.py +++ b/dpgen/auto_test/lib/lammps.py @@ -395,7 +395,7 @@ def poscar_from_last_dump(dump, poscar_out, deepmd_type_map) : os.remove('tmp_dump') with open(poscar_out, 'r') as fp: lines = fp.read().split('\n') - types = [ deepmd_type_map[int(ii.split('_')[1])-1] for ii in lines[5].split()] + types = [ deepmd_type_map[int(ii.split('_')[1])] for ii in lines[5].split()] lines[5] = " ".join(types) with open(poscar_out, 'w') as fp: lines = fp.write("\n".join(lines)) diff --git a/dpgen/auto_test/lib/vasp.py b/dpgen/auto_test/lib/vasp.py index 5022e4b74..f2610de82 100644 --- a/dpgen/auto_test/lib/vasp.py +++ b/dpgen/auto_test/lib/vasp.py @@ -148,7 +148,7 @@ def get_nev(fname) : ener = _get_energies(lines)[-1] return natoms, ener/natoms, vol/natoms except OutcarItemError: - return natoms, None, None + raise OutcarItemError("cannot find the result, please check the OUTCAR") # print(fname, natoms, vol, ener) def get_stress(fname) : diff --git a/dpgen/auto_test/run.py b/dpgen/auto_test/run.py index ce76ca69d..a44206909 100644 --- a/dpgen/auto_test/run.py +++ b/dpgen/auto_test/run.py @@ -29,7 +29,7 @@ from dpgen.auto_test.lib.utils import create_path from dpgen.auto_test.lib.utils import copy_file_list from dpgen.auto_test.lib.utils import replace - +from dpgen.dispatcher.Dispatcher import make_dispatcher from dpgen.auto_test.lib.utils import log_iter from dpgen.auto_test.lib.utils import record_iter from dpgen.auto_test.lib.utils import log_iter @@ -51,69 +51,6 @@ lammps_task_type=['deepmd','meam','eam'] -def _run(machine, - machine_type, - ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files): - - print("group_size",group_size) - if ssh_sess == None and machine_type == 'ucloud': - print("The first situation!") - ucloud_submit_jobs(machine, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) - elif machine_type == 'slurm' : - print("The second situation!") - group_slurm_jobs(ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files, - forward_task_deference =False) - elif machine_type == 'pbs' : - group_slurm_jobs(ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files, - remote_job = PBSJob, - forward_task_deference =False) - elif machine_type == 'local' : - group_local_jobs(ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) - else : - raise RuntimeError("unknow machine type") - - - def gen_equi(task_type,jdata,mdata): conf_dir=jdata['conf_dir'] cwd=os.getcwd() @@ -138,7 +75,7 @@ def run_equi(task_type,jdata,mdata): mdata=decide_fp_machine(mdata) forward_files = ['INCAR', 'POTCAR'] - backward_files = ['OUTCAR','CONTCAR','OSZICAR'] + backward_files = ['OUTCAR', 'autotest.out' , 'CONTCAR','OSZICAR'] common_files=['POSCAR'] #lammps @@ -146,7 +83,7 @@ def run_equi(task_type,jdata,mdata): mdata = decide_model_devi_machine(mdata) forward_files = ['conf.lmp', 'lammps.in'] - backward_files = ['dump.relax','log.lammps', 'model_devi.log'] + backward_files = ['dump.relax','log.lammps', 'autotest.out'] fp_params = jdata['lammps_params'] model_dir = fp_params['model_dir'] @@ -166,34 +103,38 @@ def run_equi(task_type,jdata,mdata): raise RuntimeError ("unknow task %s, something wrong" % task_type) run_tasks = util.collect_task(all_task,task_type) - + if len(run_tasks)==0: return machine,machine_type,ssh_sess,resources,command,group_size=util.get_machine_info(mdata,task_type) + disp = make_dispatcher(machine) + disp.run_jobs(resources, + command, + work_path, + run_tasks, + group_size, + common_files, + forward_files, + backward_files, + outlog='autotest.out', + errlog='autotest.err') - _run(machine, - machine_type, - ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) def cmpt_equi(task_type,jdata,mdata): conf_dir=jdata['conf_dir'] - stable=jdata['store_stable'] + cmpt_shift=jdata['alloy_shift'] #vasp if task_type=="vasp": - n, e, v = cmpt_00_equi.comput_vasp_nev(jdata, conf_dir, stable) + n, e, v, s = cmpt_00_equi.comput_vasp_nev(jdata, conf_dir,cmpt_shift) #lammps elif task_type in lammps_task_type: - n, e, v = cmpt_00_equi.comput_lmp_nev(conf_dir, task_type, stable) + n, e, v, s = cmpt_00_equi.comput_lmp_nev(conf_dir, task_type,cmpt_shift) else : raise RuntimeError ("unknow task %s, something wrong" % task_type) - print('conf_dir:\t EpA(eV) VpA(A^3)') - print("%s\t %8.4f %7.3f " % (conf_dir, e, v)) + if cmpt_shift: + print('conf_dir:\t EpA(eV) VpA(A^3) ener_shift(eV)') + print("%s\t %8.4f %7.3f %8.4f" % (conf_dir, e, v, s)) + else: + print('conf_dir:\t EpA(eV) VpA(A^3)') + print("%s\t %8.4f %7.3f " % (conf_dir, e, v)) def gen_eos(task_type,jdata,mdata): conf_dir=jdata['conf_dir'] @@ -223,7 +164,7 @@ def run_eos(task_type,jdata,mdata): mdata=decide_fp_machine(mdata) forward_files = ['INCAR', 'POSCAR','POTCAR'] - backward_files = ['OUTCAR','OSZICAR'] + backward_files = ['OUTCAR', 'autotest.out' , 'OSZICAR'] common_files=['INCAR','POTCAR'] #lammps @@ -240,7 +181,7 @@ def run_eos(task_type,jdata,mdata): else: models = [os.path.join(model_dir,ii) for ii in model_name] forward_files = ['conf.lmp', 'lammps.in']+model_name - backward_files = ['log.lammps', 'model_devi.log'] + backward_files = ['log.lammps', 'autotest.out'] common_files=['lammps.in']+model_name if len(model_name)>1 and task_type == 'deepmd': @@ -250,19 +191,19 @@ def run_eos(task_type,jdata,mdata): raise RuntimeError ("unknow task %s, something wrong" % task_type) run_tasks = util.collect_task(all_task,task_type) - + if len(run_tasks)==0: return machine,machine_type,ssh_sess,resources,command,group_size=util.get_machine_info(mdata,task_type) - _run(machine, - machine_type, - ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) + disp = make_dispatcher(machine) + disp.run_jobs(resources, + command, + work_path, + run_tasks, + group_size, + common_files, + forward_files, + backward_files, + outlog='autotest.out', + errlog='autotest.err') def cmpt_eos(task_type,jdata,mdata): conf_dir=jdata['conf_dir'] @@ -299,7 +240,7 @@ def run_elastic(task_type,jdata,mdata): mdata=decide_fp_machine(mdata) forward_files = ['INCAR', 'POSCAR','POTCAR','KPOINTS'] - backward_files = ['OUTCAR','CONTCAR','OSZICAR'] + backward_files = ['OUTCAR', 'autotest.out' , 'CONTCAR','OSZICAR'] common_files=['INCAR','POTCAR','KPOINTS'] #lammps @@ -316,7 +257,7 @@ def run_elastic(task_type,jdata,mdata): else: models = [os.path.join(model_dir,ii) for ii in model_name] forward_files = ['conf.lmp', 'lammps.in','strain.out']+model_name - backward_files = ['log.lammps', 'model_devi.log'] + backward_files = ['log.lammps', 'autotest.out'] common_files=['lammps.in']+model_name if len(model_name)>1 and task_type == 'deepmd': @@ -326,18 +267,19 @@ def run_elastic(task_type,jdata,mdata): raise RuntimeError ("unknow task %s, something wrong" % task_type) run_tasks = util.collect_task(all_task,task_type) + if len(run_tasks)==0: return machine,machine_type,ssh_sess,resources,command,group_size=util.get_machine_info(mdata,task_type) - _run(machine, - machine_type, - ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) + disp = make_dispatcher(machine) + disp.run_jobs(resources, + command, + work_path, + run_tasks, + group_size, + common_files, + forward_files, + backward_files, + outlog='autotest.out', + errlog='autotest.err') def cmpt_elastic(task_type,jdata,mdata): conf_dir=jdata['conf_dir'] @@ -372,7 +314,7 @@ def run_vacancy(task_type,jdata,mdata): mdata=decide_fp_machine(mdata) forward_files = ['INCAR', 'POSCAR','POTCAR'] - backward_files = ['OUTCAR','OSZICAR'] + backward_files = ['OUTCAR', 'autotest.out' , 'OSZICAR'] common_files=['INCAR','POTCAR'] #lammps @@ -390,7 +332,7 @@ def run_vacancy(task_type,jdata,mdata): models = [os.path.join(model_dir,ii) for ii in model_name] common_files = model_name forward_files = ['conf.lmp', 'lammps.in']+model_name - backward_files = ['log.lammps','model_devi.log'] + backward_files = ['log.lammps','autotest.out'] common_files=['lammps.in']+model_name if len(model_name)>1 and task_type == 'deepmd': @@ -400,18 +342,19 @@ def run_vacancy(task_type,jdata,mdata): raise RuntimeError ("unknow task %s, something wrong" % task_type) run_tasks = util.collect_task(all_task,task_type) + if len(run_tasks)==0: return machine,machine_type,ssh_sess,resources,command,group_size=util.get_machine_info(mdata,task_type) - _run(machine, - machine_type, - ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) + disp = make_dispatcher(machine) + disp.run_jobs(resources, + command, + work_path, + run_tasks, + group_size, + common_files, + forward_files, + backward_files, + outlog='autotest.out', + errlog='autotest.err') def cmpt_vacancy(task_type,jdata,mdata): conf_dir=jdata['conf_dir'] @@ -455,7 +398,7 @@ def run_interstitial(task_type,jdata,mdata): mdata=decide_fp_machine(mdata) forward_files = ['INCAR', 'POSCAR','POTCAR'] - backward_files = ['OUTCAR','XDATCAR','OSZICAR'] + backward_files = ['OUTCAR', 'autotest.out' , 'XDATCAR','OSZICAR'] common_files=['INCAR'] #lammps @@ -488,7 +431,7 @@ def run_interstitial(task_type,jdata,mdata): else: models = [os.path.join(model_dir,ii) for ii in model_name] forward_files = ['conf.lmp', 'lammps.in']+model_name - backward_files = ['log.lammps', 'model_devi.log'] + backward_files = ['log.lammps', 'autotest.out'] common_files=['lammps.in']+model_name if len(model_name)>1 and task_type == 'deepmd': @@ -498,37 +441,37 @@ def run_interstitial(task_type,jdata,mdata): raise RuntimeError ("unknow task %s, something wrong" % task_type) machine,machine_type,ssh_sess,resources,command,group_size=util.get_machine_info(mdata,task_type) - + disp = make_dispatcher(machine) if reprod_opt: for ii in work_path: run_tasks=[] for jj in run_tasks_: if ii in jj: run_tasks.append(os.path.basename(jj)) - _run(machine, - machine_type, - ssh_sess, - resources, - command, - ii, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) + + disp.run_jobs(resources, + command, + ii, + run_tasks, + group_size, + common_files, + forward_files, + backward_files, + outlog='autotest.out', + errlog='autotest.err') else: run_tasks = util.collect_task(all_task,task_type) - _run(machine, - machine_type, - ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) + if len(run_tasks)==0: return + disp.run_jobs(resources, + command, + work_path, + run_tasks, + group_size, + common_files, + forward_files, + backward_files, + outlog='autotest.out', + errlog='autotest.err') def cmpt_interstitial(task_type,jdata,mdata): conf_dir=jdata['conf_dir'] @@ -577,7 +520,7 @@ def run_surf(task_type,jdata,mdata): mdata=decide_fp_machine(mdata) forward_files = ['INCAR', 'POSCAR','POTCAR'] - backward_files = ['OUTCAR','OSZICAR'] + backward_files = ['OUTCAR', 'autotest.out' , 'OSZICAR'] common_files=['INCAR','POTCAR'] #lammps @@ -594,7 +537,7 @@ def run_surf(task_type,jdata,mdata): else: models = [os.path.join(model_dir,ii) for ii in model_name] forward_files = ['conf.lmp', 'lammps.in']+model_name - backward_files = ['log.lammps','model_devi.log'] + backward_files = ['log.lammps','autotest.out'] common_files=['lammps.in']+model_name if len(model_name)>1 and task_type == 'deepmd': @@ -604,18 +547,19 @@ def run_surf(task_type,jdata,mdata): raise RuntimeError ("unknow task %s, something wrong" % task_type) run_tasks = util.collect_task(all_task,task_type) + if len(run_tasks)==0: return machine,machine_type,ssh_sess,resources,command,group_size=util.get_machine_info(mdata,task_type) - _run(machine, - machine_type, - ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) + disp = make_dispatcher(machine) + disp.run_jobs(resources, + command, + work_path, + run_tasks, + group_size, + common_files, + forward_files, + backward_files, + outlog='autotest.out', + errlog='autotest.err') def cmpt_surf(task_type,jdata,mdata): conf_dir=jdata['conf_dir'] @@ -661,20 +605,20 @@ def run_phonon(task_type,jdata,mdata): run_tasks = util.collect_task(all_task,task_type) forward_files = ['INCAR', 'POTCAR','KPOINTS'] - backward_files = ['OUTCAR','OSZICAR','vasprun.xml'] + backward_files = ['OUTCAR', 'autotest.out' , 'OSZICAR','vasprun.xml'] common_files=['POSCAR'] - _run(machine, - machine_type, - ssh_sess, - resources, - command, - work_path, - run_tasks, - group_size, - common_files, - forward_files, - backward_files) + disp = make_dispatcher(machine) + disp.run_jobs(resources, + command, + work_path, + run_tasks, + group_size, + common_files, + forward_files, + backward_files, + outlog='autotest.out', + errlog='autotest.err') #lammps elif task_type in lammps_task_type: None diff --git a/dpgen/data/gen.py b/dpgen/data/gen.py index c8e98ff54..73bb8e25f 100644 --- a/dpgen/data/gen.py +++ b/dpgen/data/gen.py @@ -553,9 +553,9 @@ def run_vasp_relax(jdata, mdata, dispatcher): return relax_run_tasks = relax_tasks - for ii in relax_tasks : - if not _vasp_check_fin(ii): - relax_run_tasks.append(ii) + #for ii in relax_tasks : + # if not _vasp_check_fin(ii): + # relax_run_tasks.append(ii) run_tasks = [os.path.basename(ii) for ii in relax_run_tasks] #dlog.info(run_tasks) diff --git a/dpgen/data/reaction.py b/dpgen/data/reaction.py new file mode 100644 index 000000000..5e1f4f120 --- /dev/null +++ b/dpgen/data/reaction.py @@ -0,0 +1,231 @@ +""" +input: trajectory +00: ReaxFF MD (lammps) +01: build dataset (mddatasetbuilder) +02: fp (gaussian) +03: convert to deepmd data +output: data +""" + +import argparse +import glob +import json +import os +import random + +import dpdata +from dpgen import dlog +from dpgen.dispatcher.Dispatcher import make_dispatcher +from dpgen.generator.run import create_path, make_fp_task_name +from dpgen.util import sepline + +reaxff_path = "00.reaxff" +build_path = "01.build" +fp_path = "02.fp" +data_path = "03.data" + +trj_path = "lammpstrj" +ff_path = "ffield.reax" +data_init_path = "data.init" +control_path = "lmp_control" +lmp_path = "in.lmp" +dataset_name = "dpgen_init" + + +def link_reaxff(jdata): + create_path(reaxff_path) + task_path = os.path.join(reaxff_path, "task.000") + create_path(task_path) + + rdata = jdata['reaxff'] + os.symlink(os.path.abspath(rdata["data"]), os.path.abspath( + os.path.join(task_path, data_init_path))) + os.symlink(os.path.abspath(rdata["ff"]), os.path.abspath( + os.path.join(task_path, ff_path))) + os.symlink(os.path.abspath(rdata["control"]), os.path.abspath( + os.path.join(task_path, control_path))) + with open(os.path.join(task_path, lmp_path), 'w') as f: + f.write(make_lmp(jdata)) + + +def make_lmp(jdata): + rdata = jdata['reaxff'] + lmp_string = """units real +atom_style charge +read_data data.init +pair_style reax/c lmp_control +pair_coeff * * ffield.reax {type_map} +velocity all create {temp} {rand} +fix 1 all nvt temp {temp} {temp} {tau_t} +fix 2 all qeq/reax 1 0.0 10.0 1.0e-6 reax/c +dump 1 all custom {dump_freq} lammpstrj id type x y z +timestep {dt} +run {nstep} +""".format( + type_map=" ".join(jdata['type_map']), + temp=rdata['temp'], + rand=random.randrange(1000000-1)+1, + tau_t=rdata['tau_t'], + dump_freq=rdata['dump_freq'], + dt=rdata['dt'], + nstep=rdata['nstep'] + ) + return lmp_string + + +def run_reaxff(jdata, mdata, dispatcher, log_file="reaxff_log"): + work_path = reaxff_path + reaxff_command = "{} -in {}".format(mdata["reaxff_command"], lmp_path) + run_tasks = glob.glob(os.path.join(work_path, 'task.*')) + run_tasks.sort() + run_tasks = [os.path.basename(ii) for ii in run_tasks] + + dispatcher.run_jobs(mdata['reaxff_resources'], + [reaxff_command], + work_path, + run_tasks, + 1, + [], + [ff_path, data_init_path, control_path, lmp_path], + [trj_path], + outlog=log_file, + errlog=log_file) + + +def link_trj(jdata): + """link lammpstrj""" + create_path(build_path) + task_path = os.path.join(build_path, "task.000") + create_path(task_path) + + os.symlink(os.path.abspath(os.path.join(reaxff_path, "task.000", trj_path)), os.path.abspath( + os.path.join(task_path, trj_path))) + + +def run_build_dataset(jdata, mdata, dispatcher, log_file="build_log"): + work_path = build_path + build_command = "{cmd} -n {dataset_name} -a {type_map} -d {lammpstrj} -c {cutoff} -s {dataset_size} -k \"{qmkeywords}\" --nprocjob {nprocjob} --nproc {nproc}".format( + cmd=mdata["build_command"], + type_map=" ".join(jdata["type_map"]), + lammpstrj=trj_path, + cutoff=jdata["cutoff"], + dataset_size=jdata["dataset_size"], + qmkeywords=jdata["qmkeywords"], + nprocjob=mdata["fp_resources"]["task_per_node"], + nproc=mdata["build_resources"]["task_per_node"], + dataset_name=dataset_name + ) + run_tasks = glob.glob(os.path.join(work_path, 'task.*')) + run_tasks.sort() + run_tasks = [os.path.basename(ii) for ii in run_tasks] + + dispatcher.run_jobs(mdata['build_resources'], + [build_command], + work_path, + run_tasks, + 1, + [], + [trj_path], + [f"dataset_{dataset_name}_gjf"], + outlog=log_file, + errlog=log_file) + + +def link_fp_input(): + all_input_file = glob.glob(os.path.join( + build_path, "task.*", f"dataset_{dataset_name}_gjf", "*", "*.gjf")) + work_path = fp_path + create_path(work_path) + + for ii, fin in enumerate(all_input_file): + dst_path = os.path.join(work_path, make_fp_task_name(0, ii)) + create_path(dst_path) + os.symlink(os.path.abspath(fin), os.path.abspath( + os.path.join(dst_path, "input"))) + + +def run_fp(jdata, + mdata, + dispatcher, + log_file="output", + forward_common_files=[]): + fp_command = mdata['fp_command'] + fp_group_size = mdata['fp_group_size'] + work_path = fp_path + + fp_tasks = glob.glob(os.path.join(work_path, 'task.*')) + fp_tasks.sort() + if len(fp_tasks) == 0: + return + + fp_run_tasks = fp_tasks + + run_tasks = [os.path.basename(ii) for ii in fp_run_tasks] + + dispatcher.run_jobs(mdata['fp_resources'], + [fp_command], + work_path, + run_tasks, + fp_group_size, + [], + ["input"], + [log_file], + outlog=log_file, + errlog=log_file) + + +def convert_data(jdata): + s = dpdata.MultiSystems(*[dpdata.LabeledSystem(x, fmt="gaussian/log") + for x in glob.glob(os.path.join(fp_path, "*", "output"))], + type_map=jdata["type_map"]) + s.to_deepmd_npy(data_path) + dlog.info("Initial data is avaiable in %s" % os.path.abspath(data_path)) + + +def gen_init_reaction(args): + try: + import ruamel + from monty.serialization import loadfn, dumpfn + warnings.simplefilter( + 'ignore', ruamel.yaml.error.MantissaNoDotYAML1_1Warning) + jdata = loadfn(args.PARAM) + if args.MACHINE is not None: + mdata = loadfn(args.MACHINE) + except: + with open(args.PARAM, 'r') as fp: + jdata = json.load(fp) + if args.MACHINE is not None: + with open(args.MACHINE, "r") as fp: + mdata = json.load(fp) + + record = "record.reaction" + iter_rec = -1 + numb_task = 7 + if os.path.isfile(record): + with open(record) as frec: + for line in frec: + iter_rec = int(line.strip()) + dlog.info("continue from task %02d" % iter_rec) + for ii in range(numb_task): + sepline(str(ii), '-') + if ii <= iter_rec: + continue + elif ii == 0: + link_reaxff(jdata) + elif ii == 1: + dispatcher = make_dispatcher(mdata["reaxff_machine"]) + run_reaxff(jdata, mdata, dispatcher) + elif ii == 2: + link_trj(jdata) + elif ii == 3: + dispatcher = make_dispatcher(mdata["build_machine"]) + run_build_dataset(jdata, mdata, dispatcher) + elif ii == 4: + link_fp_input() + elif ii == 5: + dispatcher = make_dispatcher(mdata["fp_machine"]) + run_fp(jdata, mdata, dispatcher) + elif ii == 6: + convert_data(jdata) + with open(record, "a") as frec: + frec.write(str(ii)+'\n') diff --git a/dpgen/data/surf.py b/dpgen/data/surf.py index f28f6e3ba..78c6f918c 100644 --- a/dpgen/data/surf.py +++ b/dpgen/data/surf.py @@ -60,17 +60,29 @@ def replace (file_name, pattern, subst) : global_dirname_04 = '02.md' def out_dir_name(jdata) : - cell_type = jdata['cell_type'] - elements = jdata['elements'] super_cell = jdata['super_cell'] - ele_str = "surf." - for ii in elements: - ele_str = ele_str + ii.lower() - cell_str = "%02d" % (super_cell[0]) - for ii in range(1,len(super_cell)) : - cell_str = cell_str + ("x%02d" % super_cell[ii]) - return ele_str + '.' + cell_type + '.' + cell_str + from_poscar= jdata.get('from_poscar',False) + + if from_poscar: + from_poscar_path = jdata['from_poscar_path'] + poscar_name = os.path.basename(from_poscar_path) + cell_str = "%02d" % (super_cell[0]) + for ii in range(1,len(super_cell)) : + cell_str = cell_str + ("x%02d" % super_cell[ii]) + return poscar_name + '.' + cell_str + else: + cell_type = jdata['cell_type'] + elements = jdata['elements'] + super_cell = jdata['super_cell'] + + ele_str = "surf." + for ii in elements: + ele_str = ele_str + ii.lower() + cell_str = "%02d" % (super_cell[0]) + for ii in range(1,len(super_cell)) : + cell_str = cell_str + ("x%02d" % super_cell[ii]) + return ele_str + '.' + cell_type + '.' + cell_str def class_cell_type(jdata) : ct = jdata['cell_type'] @@ -177,33 +189,46 @@ def poscar_elong (poscar_in, poscar_out, elong) : fout.write("".join(lines)) def make_unit_cell (jdata) : - latt = jdata['latt'] + + from_poscar= jdata.get('from_poscar',False) + if not from_poscar: + latt = jdata['latt'] + cell_type = class_cell_type(jdata) + out_dir = jdata['out_dir'] path_uc = os.path.join(out_dir, global_dirname_02) - cell_type = class_cell_type(jdata) cwd = os.getcwd() # for ii in scale : # path_work = create_path(os.path.join(path_uc, '%.3f' % ii)) path_work = create_path(path_uc) os.chdir(path_work) - with open('POSCAR.unit', 'w') as fp: - fp.write (cell_type.poscar_unit(latt)) + if not from_poscar: + with open('POSCAR.unit', 'w') as fp: + fp.write (cell_type.poscar_unit(latt)) os.chdir(cwd) def make_super_cell_pymatgen (jdata) : - make_unit_cell(jdata) + make_unit_cell(jdata) out_dir = jdata['out_dir'] path_uc = os.path.join(out_dir, global_dirname_02) - from_path = path_uc - from_file = os.path.join(from_path, 'POSCAR.unit') - ss = Structure.from_file(from_file) - # ase only support X type element - for i in range(len(ss)): - ss[i]='X' - ss=AseAtomsAdaptor.get_atoms(ss) + from_poscar= jdata.get('from_poscar',False) + + if from_poscar: + from_poscar_path = jdata['from_poscar_path'] + poscar_name = os.path.basename(from_poscar_path) + ss = Structure.from_file(poscar_name) + else: + from_path = path_uc + from_file = os.path.join(from_path, 'POSCAR.unit') + ss = Structure.from_file(from_file) + # ase only support X type element + for i in range(len(ss)): + ss[i]='X' + + ss=AseAtomsAdaptor.get_atoms(ss) all_millers = jdata['millers'] path_sc = os.path.join(out_dir, global_dirname_02) @@ -256,6 +281,7 @@ def place_element (jdata) : super_cell = jdata['super_cell'] cell_type = class_cell_type(jdata) elements = jdata['elements'] + from_poscar= jdata.get('from_poscar',False) path_sc = os.path.join(out_dir, global_dirname_02) path_pe = os.path.join(out_dir, global_dirname_02) path_sc = os.path.abspath(path_sc) @@ -285,7 +311,10 @@ def place_element (jdata) : path_work = os.path.join(path_surf, comb_name) create_path(path_work) pos_out = os.path.join(path_work, 'POSCAR') - poscar_ele(pos_in, pos_out, elements, ii) + if from_poscar: + shutil.copy2( pos_in, pos_out) + else: + poscar_ele(pos_in, pos_out, elements, ii) poscar_shuffle(pos_out, pos_out) def make_vasp_relax (jdata) : @@ -528,7 +557,7 @@ def gen_init_surf(args): out_dir = out_dir_name(jdata) jdata['out_dir'] = out_dir dlog.info ("# working dir %s" % out_dir) - + if args.MACHINE is not None: # Decide a proper machine mdata = decide_fp_machine(mdata) diff --git a/dpgen/dispatcher/ALI.py b/dpgen/dispatcher/ALI.py new file mode 100644 index 000000000..5ceb5ad8d --- /dev/null +++ b/dpgen/dispatcher/ALI.py @@ -0,0 +1,146 @@ +from aliyunsdkecs.request.v20140526.DescribeInstancesRequest import DescribeInstancesRequest +from aliyunsdkcore.client import AcsClient +from aliyunsdkcore.acs_exception.exceptions import ClientException +from aliyunsdkcore.acs_exception.exceptions import ServerException +from aliyunsdkecs.request.v20140526.RunInstancesRequest import RunInstancesRequest +from aliyunsdkecs.request.v20140526.DeleteInstancesRequest import DeleteInstancesRequest +import time, json, os, glob +from dpgen.dispatcher.Dispatcher import Dispatcher, _split_tasks +from os.path import join + +class ALI(): + def __init__(self, adata, mdata_resources, mdata_machine, nchunks, work_path): + self.ip_list = None + self.instance_list = None + self.dispatchers = None + self.adata = adata + self.mdata_resources = mdata_resources + self.mdata_machine = mdata_machine + self.nchunks = nchunks + self.work_path = work_path + + def init(self): + if self.check_restart(): + pass + else: + self.create_machine() + self.dispatchers = self.make_dispatchers() + + def check_restart(self): + dispatchers = [] + instance_list = [] + if len(glob.glob(os.path.join(self.work_path, 'jr.*.json'))) == self.nchunks: + for ii in range(self.nchunks): + with open(os.path.join(self.work_path, 'jr.%.06d.json' % ii)) as fp: + job_record = json.load(fp) + key = list(job_record.keys())[0] + ip, instance_id = job_record[key]['context'][-2], job_record[key]['context'][-1] + instance_list.append(instance_id) + profile = self.mdata_machine.copy() + profile['hostname'] = ip + profile['instance_id'] = instance_id + disp = Dispatcher(profile, context_type='ssh', batch_type='shell', job_record='jr.%.06d.json' % ii) + max_check = 10 + cnt = 0 + while not disp.session._check_alive(): + cnt += 1 + if cnt == max_check: + break + if cnt != max_check: + dispatchers.append(disp) + restart = False + if len(dispatchers) == self.nchunks: + restart = True + self.dispatchers = dispatchers + self.instance_list = instance_list + return restart + + def run_jobs(self, + resources, + command, + work_path, + tasks, + group_size, + forward_common_files, + forward_task_files, + backward_task_files, + forward_task_deference = True, + outlog = 'log', + errlog = 'err'): + task_chunks = _split_tasks(tasks, group_size) + job_handlers = [] + for ii in range(self.nchunks): + job_handler = self.dispatchers[ii].submit_jobs(resources, + command, + work_path, + task_chunks[ii], + group_size, + forward_common_files, + forward_task_files, + backward_task_files, + forward_task_deference, + outlog, + errlog) + job_handlers.append(job_handler) + while True: + cnt = 0 + for ii in range(self.nchunks): + if self.dispatchers[ii].all_finished(job_handlers[ii]): + cnt += 1 + if cnt == self.nchunks: + break + else: + time.sleep(10) + self.delete_machine() + + def make_dispatchers(self): + dispatchers = [] + for ii in range(self.nchunks): + profile = self.mdata_machine.copy() + profile['hostname'] = self.ip_list[ii] + profile['instance_id'] = self.instance_list[ii] + disp = Dispatcher(profile, context_type='ssh', batch_type='shell', job_record='jr.%.06d.json' % ii) + dispatchers.append(disp) + return dispatchers + + def create_machine(self): + AccessKey_ID = self.adata["AccessKey_ID"] + AccessKey_Secret = self.adata["AccessKey_Secret"] + strategy = self.adata["pay_strategy"] + pwd = self.adata["password"] + regionID = self.mdata_machine['regionID'] + template_name = '%s_%s_%s' % (self.mdata_resources['partition'], self.mdata_resources['numb_gpu'], strategy) + instance_name = self.adata["instance_name"] + client = AcsClient(AccessKey_ID,AccessKey_Secret, regionID) + request = RunInstancesRequest() + request.set_accept_format('json') + request.set_UniqueSuffix(True) + request.set_Password(pwd) + request.set_InstanceName(instance_name) + request.set_Amount(self.nchunks) + request.set_LaunchTemplateName(template_name) + response = client.do_action_with_exception(request) + response = json.loads(response) + self.instance_list = response["InstanceIdSets"]["InstanceIdSet"] + time.sleep(50) + request = DescribeInstancesRequest() + request.set_accept_format('json') + request.set_InstanceIds(self.instance_list) + response = client.do_action_with_exception(request) + response = json.loads(response) + ip = [] + for i in range(len(response["Instances"]["Instance"])): + ip.append(response["Instances"]["Instance"][i]["PublicIpAddress"]['IpAddress'][0]) + self.ip_list = ip + + def delete_machine(self): + AccessKey_ID = self.adata["AccessKey_ID"] + AccessKey_Secret = self.adata["AccessKey_Secret"] + regionID = self.mdata_machine['regionID'] + client = AcsClient(AccessKey_ID,AccessKey_Secret, regionID) + request = DeleteInstancesRequest() + request.set_accept_format('json') + request.set_InstanceIds(self.instance_list) + request.set_Force(True) + response = client.do_action_with_exception(request) + diff --git a/dpgen/dispatcher/AWS.py b/dpgen/dispatcher/AWS.py index ddb91dad6..9bbd3ca85 100644 --- a/dpgen/dispatcher/AWS.py +++ b/dpgen/dispatcher/AWS.py @@ -5,15 +5,14 @@ from dpgen.dispatcher.JobStatus import JobStatus from dpgen import dlog +try: + import boto3 +except ModuleNotFoundError: + pass +else: + batch_client = boto3.client('batch') class AWS(Batch): - try: - import boto3 - except ModuleNotFoundError: - pass - else: - batch_client = boto3.client('batch') - _query_max_results = 1000 _query_time_interval = 30 _job_id_map_status = {} _jobQueue = "" @@ -44,19 +43,19 @@ def AWS_check_status(cls, job_id=""): """ query_dict ={} if datetime.now().timestamp() > cls._query_next_allow_time: - cls.batch_client = boto3.client('batch') cls._query_next_allow_time=datetime.now().timestamp()+cls._query_time_interval for status in ['SUBMITTED', 'PENDING', 'RUNNABLE', 'STARTING', 'RUNNING','SUCCEEDED', 'FAILED']: - status_response = cls.batch_client.list_jobs(jobQueue=cls._jobQueue, jobStatus=status, maxResults=cls._query_max_results) - status_list=status_response.get('jobSummaryList', []) - for job_dict in status_list: - cls._job_id_map_status.update({job_dict['jobId']: cls.map_aws_status_to_dpgen_status(job_dict['status'])}) - # for job in cls._job_id_map_status: - # cls._job_id_map_status[job]=query_dict.get(job, JobStatus.unknown) + nextToken = '' + while nextToken is not None: + status_response = batch_client.list_jobs(jobQueue=cls._jobQueue, jobStatus=status, maxResults=100, nextToken=nextToken) + status_list=status_response.get('jobSummaryList') + nextToken = status_response.get('nextToken', None) + for job_dict in status_list: + cls._job_id_map_status.update({job_dict['jobId']: cls.map_aws_status_to_dpgen_status(job_dict['status'])}) dlog.debug('20000:_map: %s' %(cls._job_id_map_status)) dlog.debug('62000:job_id:%s, _query: %s, _map: %s' %(job_id, query_dict, cls._job_id_map_status)) if job_id: - return cls._job_id_map_status.get(job_id, JobStatus.unknown) + return cls._job_id_map_status.get(job_id) return cls._job_id_map_status @@ -67,7 +66,7 @@ def job_id(self): except AttributeError: if self.context.check_file_exists(self.job_id_name): self._job_id = self.context.read_file(self.job_id_name) - response_list = self.__class__.batch_client.describe_jobs(jobs=[self._job_id]).get('jobs') + response_list = batch_client.describe_jobs(jobs=[self._job_id]).get('jobs') try: response = response_list[0] jobQueue = response['jobQueue'] @@ -135,7 +134,7 @@ def do_submit(self, """ jobName = os.path.join(self.context.remote_root,job_dirs.pop())[1:].replace('/','-').replace('.','_') jobName += ("_" + str(self.context.job_uuid)) - response = self.__class__.batch_client.submit_job(jobName=jobName, + response = batch_client.submit_job(jobName=jobName, jobQueue=res['jobQueue'], jobDefinition=res['jobDefinition'], parameters={'task_command':script_str}, diff --git a/dpgen/dispatcher/Batch.py b/dpgen/dispatcher/Batch.py index 4da2d196b..0b6d16b71 100644 --- a/dpgen/dispatcher/Batch.py +++ b/dpgen/dispatcher/Batch.py @@ -74,9 +74,13 @@ def sub_script(self, # loop over commands self.cmd_cnt = 0 try: - self.manual_gpu = res['manual_cuda_devices'] - except: - self.manual_gpu = 0 + self.manual_cuda_devices = res['manual_cuda_devices'] + except KeyError: + self.manual_cuda_devices = 0 + try: + self.manual_cuda_multiplicity = res['manual_cuda_multiplicity'] + except KeyError: + self.manual_cuda_multiplicity = 1 for ii in range(len(cmd)): # for one command ret += self._sub_script_inner(job_dirs, @@ -136,7 +140,7 @@ def _sub_script_inner(self, for ii,jj in zip(job_dirs, args) : ret += 'cd %s\n' % ii ret += 'test $? -ne 0 && exit\n\n' - if self.manual_gpu <= 0: + if self.manual_cuda_devices <= 0: ret += 'if [ ! -f tag_%d_finished ] ;then\n' % idx ret += ' %s 1>> %s 2>> %s \n' % (self.sub_script_cmd(cmd, jj, res), outlog, errlog) if res['allow_failure'] is False: @@ -147,11 +151,11 @@ def _sub_script_inner(self, else : # do not support task-wise restart tmp_cmd = ' %s 1>> %s 2>> %s ' % (self.sub_script_cmd(cmd, jj, res), outlog, errlog) - ret += 'CUDA_VISIBLE_DEVICES=%d %s &\n\n' % ((self.cmd_cnt % self.manual_gpu), tmp_cmd) + ret += 'CUDA_VISIBLE_DEVICES=%d %s &\n\n' % ((self.cmd_cnt % self.manual_cuda_devices), tmp_cmd) self.cmd_cnt += 1 ret += 'cd %s\n' % self.context.remote_root ret += 'test $? -ne 0 && exit\n' - if self.manual_gpu > 0 and self.cmd_cnt % self.manual_gpu == 0: + if self.manual_cuda_devices > 0 and self.cmd_cnt % (self.manual_cuda_devices * self.manual_cuda_multiplicity) == 0: ret += '\nwait\n\n' ret += '\nwait\n\n' return ret diff --git a/dpgen/dispatcher/Dispatcher.py b/dpgen/dispatcher/Dispatcher.py index 39a988543..915d7e201 100644 --- a/dpgen/dispatcher/Dispatcher.py +++ b/dpgen/dispatcher/Dispatcher.py @@ -1,4 +1,4 @@ -import os,sys,time,random +import os,sys,time,random,json,glob from dpgen.dispatcher.LocalContext import LocalSession from dpgen.dispatcher.LocalContext import LocalContext @@ -13,8 +13,6 @@ from dpgen.dispatcher.JobStatus import JobStatus from dpgen import dlog from hashlib import sha1 -from monty.serialization import dumpfn,loadfn - def _split_tasks(tasks, group_size): @@ -35,7 +33,8 @@ class Dispatcher(object): def __init__ (self, remote_profile, context_type = 'local', - batch_type = 'slurm'): + batch_type = 'slurm', + job_record = 'jr.json'): self.remote_profile = remote_profile if context_type == 'local': self.session = LocalSession(remote_profile) @@ -63,7 +62,7 @@ def __init__ (self, self.batch = AWS else : raise RuntimeError('unknown batch ' + batch_type) - + self.jrname = job_record def run_jobs(self, resources, @@ -77,30 +76,59 @@ def run_jobs(self, forward_task_deference = True, outlog = 'log', errlog = 'err') : + job_handler = self.submit_jobs(resources, + command, + work_path, + tasks, + group_size, + forward_common_files, + forward_task_files, + backward_task_files, + forward_task_deference, + outlog, + errlog) + while not self.all_finished(job_handler) : + time.sleep(10) + # delete path map file when job finish + # _pmap.delete() + + + def submit_jobs(self, + resources, + command, + work_path, + tasks, + group_size, + forward_common_files, + forward_task_files, + backward_task_files, + forward_task_deference = True, + outlog = 'log', + errlog = 'err') : + self.backward_task_files = backward_task_files # task_chunks = [ # [os.path.basename(j) for j in tasks[i:i + group_size]] \ # for i in range(0, len(tasks), group_size) # ] task_chunks = _split_tasks(tasks, group_size) - _pmap=PMap(work_path) - path_map=_pmap.load() - _fr = FinRecord(work_path, len(task_chunks)) - + task_chunks_str = ['+'.join(ii) for ii in task_chunks] + task_hashes = [sha1(ii.encode('utf-8')).hexdigest() for ii in task_chunks_str] + job_record = JobRecord(work_path, task_chunks, fname = self.jrname) + nchunks = len(task_chunks) + job_list = [] - task_chunks_=['+'.join(ii) for ii in task_chunks] - job_fin = _fr.get_record() - assert(len(job_fin) == len(task_chunks)) - for ii,chunk in enumerate(task_chunks) : - if not job_fin[ii] : - # map chunk info. to uniq id - chunk_sha1 = sha1(task_chunks_[ii].encode('utf-8')).hexdigest() - # if hash in map, recover job, else start a new job - if chunk_sha1 in path_map: - # job_uuid = path_map[chunk_sha1][1].split('/')[-1] - job_uuid = path_map[chunk_sha1][2] - dlog.debug("load uuid %s for chunk %s" % (job_uuid, task_chunks_[ii])) - else: + for ii in range(nchunks) : + cur_chunk = task_chunks[ii] + cur_hash = task_hashes[ii] + if not job_record.check_finished(cur_hash): + # chunk is not finished + # check if chunk is submitted + submitted = job_record.check_submitted(cur_hash) + if not submitted: job_uuid = None + else : + job_uuid = job_record.get_uuid(cur_hash) + dlog.debug("load uuid %s for chunk %s" % (job_uuid, cur_hash)) # communication context, bach system context = self.context(work_path, self.session, job_uuid) batch = self.batch(context, uuid_names = self.uuid_names) @@ -109,102 +137,163 @@ def run_jobs(self, if not rjob['context'].check_file_exists('tag_upload'): rjob['context'].upload('.', forward_common_files) - rjob['context'].upload(chunk, + rjob['context'].upload(cur_chunk, forward_task_files, dereference = forward_task_deference) rjob['context'].write_file('tag_upload', '') - dlog.debug('uploaded files for %s' % task_chunks_[ii]) + dlog.debug('uploaded files for %s' % task_chunks_str[ii]) # submit new or recover old submission - if job_uuid is None: - rjob['batch'].submit(chunk, command, res = resources, outlog=outlog, errlog=errlog) + if not submitted: + rjob['batch'].submit(cur_chunk, command, res = resources, outlog=outlog, errlog=errlog) job_uuid = rjob['context'].job_uuid - dlog.debug('assigned uudi %s for %s ' % (job_uuid, task_chunks_[ii])) - dlog.info('new submission of %s' % job_uuid) + dlog.debug('assigned uuid %s for %s ' % (job_uuid, task_chunks_str[ii])) + dlog.info('new submission of %s for chunk %s' % (job_uuid, cur_hash)) else: - rjob['batch'].submit(chunk, command, res = resources, outlog=outlog, errlog=errlog, restart = True) - dlog.info('restart from old submission %s ' % job_uuid) - # record job and its hash + rjob['batch'].submit(cur_chunk, command, res = resources, outlog=outlog, errlog=errlog, restart = True) + dlog.info('restart from old submission %s for chunk %s' % (job_uuid, cur_hash)) + # record job and its remote context job_list.append(rjob) - path_map[chunk_sha1] = [context.local_root, context.remote_root, job_uuid] + ip = None + instance_id = None + if "type" in self.remote_profile: + if self.remote_profile['type'] == 'ALI': + ip = self.remote_profile['hostname'] + instance_id = self.remote_profile['instance_id'] + job_record.record_remote_context(cur_hash, + context.local_root, + context.remote_root, + job_uuid, + ip, + instance_id) else : # finished job, append a None to list job_list.append(None) - _pmap.dump(path_map) - - assert(len(job_list) == len(task_chunks)) - fcount = [0]*len(job_list) - while not all(job_fin) : - dlog.debug('checking jobs') - for idx,rjob in enumerate(job_list) : - if not job_fin[idx] : - status = rjob['batch'].check_status() - job_uuid = rjob['context'].job_uuid - if status == JobStatus.terminated : - fcount[idx] += 1 - if fcount[idx] > 3: - raise RuntimeError('Job %s failed for more than 3 times' % job_uuid) - dlog.info('job %s terminated, submit again'% job_uuid) - dlog.debug('try %s times for %s'% (fcount[idx], job_uuid)) - rjob['batch'].submit(task_chunks[idx], command, res = resources, outlog=outlog, errlog=errlog,restart=True) - elif status == JobStatus.finished : - dlog.info('job %s finished' % job_uuid) - rjob['context'].download(task_chunks[idx], backward_task_files) - rjob['context'].clean() - job_fin[idx] = True - _fr.write_record(job_fin) - time.sleep(10) - # delete path map file when job finish - _pmap.delete() + job_record.dump() + assert(len(job_list) == nchunks) + job_handler = { + 'task_chunks': task_chunks, + 'job_list': job_list, + 'job_record': job_record, + 'command': command, + 'resources': resources, + 'outlog': outlog, + 'errlog': errlog, + 'backward_task_files': backward_task_files + } + return job_handler -class FinRecord(object): - def __init__ (self, path, njobs, fname = 'fin.record'): + def all_finished(self, + job_handler): + task_chunks = job_handler['task_chunks'] + task_chunks_str = ['+'.join(ii) for ii in task_chunks] + task_hashes = [sha1(ii.encode('utf-8')).hexdigest() for ii in task_chunks_str] + job_list = job_handler['job_list'] + job_record = job_handler['job_record'] + command = job_handler['command'] + resources = job_handler['resources'] + outlog = job_handler['outlog'] + errlog = job_handler['errlog'] + backward_task_files = job_handler['backward_task_files'] + dlog.debug('checking jobs') + nchunks = len(task_chunks) + for idx in range(nchunks) : + cur_hash = task_hashes[idx] + rjob = job_list[idx] + if not job_record.check_finished(cur_hash) : + # chunk not finished according to record + status = rjob['batch'].check_status() + job_uuid = rjob['context'].job_uuid + dlog.debug('checked job %s' % job_uuid) + if status == JobStatus.terminated : + job_record.increase_nfail(cur_hash) + if job_record.check_nfail(cur_hash) > 3: + raise RuntimeError('Job %s failed for more than 3 times' % job_uuid) + dlog.info('job %s terminated, submit again'% job_uuid) + dlog.debug('try %s times for %s'% (job_record.check_nfail(cur_hash), job_uuid)) + rjob['batch'].submit(task_chunks[idx], command, res = resources, outlog=outlog, errlog=errlog,restart=True) + elif status == JobStatus.finished : + dlog.info('job %s finished' % job_uuid) + rjob['context'].download(task_chunks[idx], backward_task_files) + rjob['context'].clean() + job_record.record_finish(cur_hash) + job_record.dump() + job_record.dump() + return job_record.check_all_finished() + + +class JobRecord(object): + def __init__ (self, path, task_chunks, fname = 'job_record.json', ip=None): self.path = os.path.abspath(path) self.fname = os.path.join(self.path, fname) - self.njobs = njobs - - def get_record(self): + self.task_chunks = task_chunks if not os.path.exists(self.fname): - return [False] * self.njobs + self._new_record() else : - with open(self.fname) as fp: - return [bool(int(ii)) for ii in fp.read().split()] + self.load() - def write_record(self, job_fin): + def check_submitted(self, chunk_hash): + self.valid_hash(chunk_hash) + return self.record[chunk_hash]['context'] is not None + + def record_remote_context(self, + chunk_hash, + local_root, + remote_root, + job_uuid, + ip=None, + instance_id=None): + self.valid_hash(chunk_hash) + self.record[chunk_hash]['context'] = [local_root, remote_root, job_uuid, ip, instance_id] + + def get_uuid(self, chunk_hash): + self.valid_hash(chunk_hash) + return self.record[chunk_hash]['context'][2] + + def check_finished(self, chunk_hash): + self.valid_hash(chunk_hash) + return self.record[chunk_hash]['finished'] + + def check_all_finished(self): + flist = [self.record[ii]['finished'] for ii in self.record] + return all(flist) + + def record_finish(self, chunk_hash): + self.valid_hash(chunk_hash) + self.record[chunk_hash]['finished'] = True + + def check_nfail(self,chunk_hash): + self.valid_hash(chunk_hash) + return self.record[chunk_hash]['fail_count'] + + def increase_nfail(self,chunk_hash): + self.valid_hash(chunk_hash) + self.record[chunk_hash]['fail_count'] += 1 + + def valid_hash(self, chunk_hash): + if chunk_hash not in self.record.keys(): + raise RuntimeError('chunk hash %s not in record, a invalid record may be used, please check file %s' % (chunk_hash, self.fname)) + + def dump(self): with open(self.fname, 'w') as fp: - for ii in job_fin: - if ii: - fp.write('1 ') - else: - fp.write('0 ') - - -class PMap(object): - ''' - Path map class to operate {read,write,delte} the pmap.json file - ''' - - def __init__(self,path,fname="pmap.json"): - self.f_path_map=os.path.join(path,fname) - - def load(self): - f_path_map=self.f_path_map - if os.path.isfile(f_path_map): - path_map=loadfn(f_path_map) - else: - path_map={} - return path_map - - def dump(self,pmap,indent=4): - f_path_map=self.f_path_map - dumpfn(pmap,f_path_map,indent=indent) - - def delete(self): - f_path_map=self.f_path_map - try: - os.remove(f_path_map) - except: - pass + json.dump(self.record, fp, indent=4) + + def load(self): + with open(self.fname) as fp: + self.record = json.load(fp) + + def _new_record(self): + task_chunks_str=['+'.join(ii) for ii in self.task_chunks] + task_hash = [sha1(ii.encode('utf-8')).hexdigest() for ii in task_chunks_str] + self.record = {} + for ii,jj in zip(task_hash, self.task_chunks): + self.record[ii] = { + 'context': None, + 'finished': False, + 'fail_count': 0, + 'task_chunk': jj, + } + def make_dispatcher(mdata): try: diff --git a/dpgen/dispatcher/LocalContext.py b/dpgen/dispatcher/LocalContext.py index e9e304418..4549b307a 100644 --- a/dpgen/dispatcher/LocalContext.py +++ b/dpgen/dispatcher/LocalContext.py @@ -113,7 +113,10 @@ def download(self, elif (os.path.exists(rfile)) and (os.path.exists(lfile)) : # both exists, replace! dlog.info('find existing %s, replacing by %s' % (lfile, rfile)) - shutil.rmtree(lfile) + if os.path.isdir(lfile): + shutil.rmtree(lfile) + elif os.path.isfile(lfile) or os.path.islink(lfile): + os.remove(lfile) shutil.move(rfile, lfile) else : raise RuntimeError('should not reach here!') diff --git a/dpgen/dispatcher/SSHContext.py b/dpgen/dispatcher/SSHContext.py index e580c0f5a..185ade392 100644 --- a/dpgen/dispatcher/SSHContext.py +++ b/dpgen/dispatcher/SSHContext.py @@ -138,7 +138,7 @@ def block_checkcall(self, stdin, stdout, stderr = self.ssh.exec_command(('cd %s ;' % self.remote_root) + cmd) exit_status = stdout.channel.recv_exit_status() if exit_status != 0: - raise RuntimeError("Get error code %d in calling %s through ssh with job: %s . message:", + raise RuntimeError("Get error code %d in calling %s through ssh with job: %s . message: %s" % (exit_status, cmd, self.job_uuid, stderr.read().decode('utf-8'))) return stdin, stdout, stderr diff --git a/dpgen/dispatcher/Shell.py b/dpgen/dispatcher/Shell.py index dd285c229..81bff523c 100644 --- a/dpgen/dispatcher/Shell.py +++ b/dpgen/dispatcher/Shell.py @@ -69,7 +69,7 @@ def sub_script_head(self, resources) : ret += ('module load %s\n' % ii) ret += ('\n') for ii in source_list : - ret += 'source %s\n' + ret += ('source %s\n' % ii) ret += ('\n') return ret diff --git a/dpgen/generator/lib/cp2k.py b/dpgen/generator/lib/cp2k.py index 93476d864..bbe877786 100644 --- a/dpgen/generator/lib/cp2k.py +++ b/dpgen/generator/lib/cp2k.py @@ -1,41 +1,139 @@ import dpdata import numpy as np -def make_section(section_name, section_value = None): - if section_value == None : - temp_section = '&' + section_name + '\n' - temp_section += '&END ' + section_name + '\n' - else : - temp_section = '&' + section_name + ' ' + section_value + '\n' - temp_section += '&END ' + section_name + '\n' - return temp_section - -def section_add_subsection(section_string, subsection_string): - section_string, section_end = section_string.rsplit('\n', 2)[0:2] - section_string += '\n' + subsection_string + section_end + '\n' - return section_string - -def section_add_keyword_and_value(section_string, keyword, keyword_value): - section_string, section_end = section_string.rsplit('\n', 2)[0:2] - section_string += '\n' + keyword + ' ' + keyword_value + '\n' + section_end + '\n' - return section_string +default_config={ + "GLOBAL": { + "PROJECT": "DPGEN" + }, + "FORCE_EVAL": { + "METHOD": "QS", + "STRESS_TENSOR": "ANALYTICAL", + "DFT": { + "BASIS_SET_FILE_NAME": "./cp2k_basis_pp_file/BASIS_MOLOPT", + "POTENTIAL_FILE_NAME": "./cp2k_basis_pp_file/GTH_POTENTIALS", + "CHARGE": 0, + "UKS": "F", + "MULTIPLICITY": 1, + "MGRID": { + "CUTOFF": 400, + "REL_CUTOFF": 50, + "NGRIDS": 4 + }, + "QS": { + "EPS_DEFAULT": "1.0E-12" + }, + "SCF": { + "SCF_GUESS": "ATOMIC", + "EPS_SCF": "1.0E-6", + "MAX_SCF": 50, + "OT": { + "MINIMIZER": "DIIS", + "PRECONDITIONER": "FULL_SINGLE_INVERSE" + } + }, + "XC": { + "XC_FUNCTIONAL": { + "_": "PBE" + } + + } + }, + "SUBSYS": { + "CELL":{ + "A": "10 .0 .0", + "B": ".0 10 .0", + "C": ".0 .0 10" + }, + "COORD": {"@include": "coord.xyz"}, + "KIND": { + "_": ["H","C","N"], + "POTENTIAL": ["GTH-PBE-q1","GTH-PBE-q4", "GTH-PBE-q5"], + "BASIS_SET": ["DZVP-MOLOPT-GTH","DZVP-MOLOPT-GTH","DZVP-MOLOPT-GTH"] + } + }, + "PRINT": { + "FORCES": { + "_": "ON" + } + } +} +} + +def update_dict(old_d, update_d): + """ + a method to recursive update dict + :old_d: old dictionary + :update_d: some update value written in dictionary form + """ + import collections + for k, v in update_d.items(): + if (k in old_d and isinstance(old_d[k], dict) and isinstance(update_d[k], collections.Mapping)): + update_dict(old_d[k], update_d[k]) + else: + old_d[k] = update_d[k] + +def iterdict(d, out_list, flag=None): + """ + :doc: a recursive expansion of dictionary into cp2k input + :k: current key + :v: current value + :d: current dictionary under expansion + :flag: used to record dictionary state. if flag is None, + it means we are in top level dict. flag is a string. + """ + for k,v in d.items(): + k=str(k) # cast key into string + #if value is dictionary + if isinstance(v, dict): + # flag == None, it is now in top level section of cp2k + if flag==None : + out_list.append("&"+k) + out_list.append("&END "+k) + iterdict(v, out_list, k) + # flag is not None, now it has name of section + else: + index = out_list.index("&END " + flag) + out_list.insert(index, "&"+k) + out_list.insert(index+1,"&END "+k ) + iterdict(v, out_list, k) + elif isinstance(v, list): +# print("we have encountered the repeat section!") + index = out_list.index("&"+flag) + # delete the current constructed repeat section + del out_list[index:index+2] + # do a loop over key and corresponding list + k_tmp_list = [] + v_list_tmp_list = [] + for k_tmp, v_tmp in d.items(): + k_tmp_list.append(str(k_tmp)) + v_list_tmp_list.append(v_tmp) + for repeat_keyword in zip(*v_list_tmp_list): + out_list.insert(index,"&" + flag) + out_list.insert(index+1, "&END " + flag) + for idx, k_tmp in enumerate(k_tmp_list): + if k_tmp == "_": + out_list[index] = "&" + flag + " " + repeat_keyword[idx] + else: + out_list.insert(index+1, k_tmp+" "+repeat_keyword[idx]) + + break + + else: + v=str(v) + if flag==None : + out_list.append(k+" "+v) + print (k,":",v) + else: + if k == "_": + index = out_list.index("&" + flag) + out_list[index] = ("&" + flag + " " + v) + + else: + index = out_list.index("&END "+flag) + out_list.insert(index, k+" "+v) -def make_cp2k_xyz(sys_data): - #get structral information - atom_names = sys_data['atom_names'] - atom_types = sys_data['atom_types'] - - #write coordinate to xyz file used by cp2k input - coord_list = sys_data['coords'][0] - u = np.array(atom_names) - atom_list = u[atom_types] - x = '\n' - for kind, coord in zip(atom_list, coord_list) : - x += str(kind) + ' ' + str(coord[:])[1:-1] + '\n' - return x def make_cp2k_input(sys_data, fp_params): - #covert cell to cell string cell = sys_data['cells'][0] cell = np.reshape(cell, [3,3]) @@ -46,90 +144,45 @@ def make_cp2k_input(sys_data, fp_params): cell_c = np.array2string(cell[2,:]) cell_c = cell_c[1:-1] + #get update from user + user_config=fp_params + #get update from cell + cell_config={"FORCE_EVAL":{ + "SUBSYS":{ + "CELL":{ + "A": cell_a, + "B": cell_b, + "C": cell_c + } + } + } + } - #made global section - global_section = make_section('GLOBAL') - global_section = section_add_keyword_and_value(global_section, 'PROJECT', 'DPGEN') - - #made force_eval section - mgrid_section = make_section('MGRID') - mgrid_section = section_add_keyword_and_value(mgrid_section, 'CUTOFF', fp_params['cutoff']) - mgrid_section = section_add_keyword_and_value(mgrid_section, 'REL_CUTOFF', fp_params['rel_cutoff']) - mgrid_section = section_add_keyword_and_value(mgrid_section, 'NGRIDS', '4') - - qs_section = make_section('QS') - qs_section = section_add_keyword_and_value(qs_section, 'EPS_DEFAULT', '1.0E-12') - - ot_section = make_section('OT') - ot_section = section_add_keyword_and_value(ot_section, 'MINIMIZER', 'DIIS') - ot_section = section_add_keyword_and_value(ot_section, 'PRECONDITIONER', 'FULL_SINGLE_INVERSE') - - scf_section = make_section('SCF') - scf_section = section_add_keyword_and_value(scf_section, 'SCF_GUESS', 'ATOMIC') - scf_section = section_add_keyword_and_value(scf_section, 'EPS_SCF', '1.0E-6') - scf_section = section_add_keyword_and_value(scf_section, 'MAX_SCF', '50') - scf_section = section_add_subsection(scf_section, ot_section) - - - xc_functional_section = make_section('XC_FUNCTIONAL', fp_params['functional']) + update_dict(default_config, user_config) + update_dict(default_config, cell_config) + #output list + input_str = [] + iterdict(default_config, input_str) + string="\n".join(input_str) + return string - pair_potential_section = make_section('PAIR_POTENTIAL') - pair_potential_section = section_add_keyword_and_value(pair_potential_section, 'TYPE', 'DFTD3') - pair_potential_section = section_add_keyword_and_value(pair_potential_section, 'PARAMETER_FILE_NAME', fp_params['pair_potential_path']) - pair_potential_section = section_add_keyword_and_value(pair_potential_section, 'REFERENCE_FUNCTIONAL', fp_params['pair_ref_functional']) - vdw_potential_section = make_section('VDW_POTENTIAL') - vdw_potential_section = section_add_keyword_and_value(vdw_potential_section, 'DISPERSION_FUNCTIONAL', 'PAIR_POTENTIAL') - vdw_potential_section = section_add_subsection(vdw_potential_section, pair_potential_section) +def make_cp2k_xyz(sys_data): + #get structral information + atom_names = sys_data['atom_names'] + atom_types = sys_data['atom_types'] - xc_section = make_section('XC') - xc_section = section_add_subsection(xc_section, xc_functional_section) - xc_section = section_add_subsection(xc_section, vdw_potential_section) - - - dft_section = make_section('DFT') - dft_section = section_add_keyword_and_value(dft_section, 'BASIS_SET_FILE_NAME', fp_params['basis_path']) - dft_section = section_add_keyword_and_value(dft_section, 'POTENTIAL_FILE_NAME', fp_params['pp_path']) - dft_section = section_add_keyword_and_value(dft_section, 'CHARGE', '0') - dft_section = section_add_keyword_and_value(dft_section, 'UKS', 'F') - dft_section = section_add_keyword_and_value(dft_section, 'MULTIPLICITY', '1') - dft_section = section_add_subsection(dft_section, mgrid_section) - dft_section = section_add_subsection(dft_section, qs_section) - dft_section = section_add_subsection(dft_section, scf_section) - dft_section = section_add_subsection(dft_section, xc_section) - - cell_section = make_section('CELL') - cell_section = section_add_keyword_and_value(cell_section, 'A', cell_a) - cell_section = section_add_keyword_and_value(cell_section, 'B', cell_b) - cell_section = section_add_keyword_and_value(cell_section, 'C', cell_c) - - coord_section = make_section('COORD') - coord_section = section_add_keyword_and_value(coord_section, '@include', 'coord.xyz') - - subsys_section = make_section('SUBSYS') - subsys_section = section_add_subsection(subsys_section, cell_section) - subsys_section = section_add_subsection(subsys_section, coord_section) - - for kind, basis, potential in zip(fp_params['element_list'], fp_params['basis_list'], fp_params['pp_list']) : - kind_section = make_section('KIND', kind) - kind_section = section_add_keyword_and_value(kind_section, 'BASIS_SET', basis) - kind_section = section_add_keyword_and_value(kind_section, 'POTENTIAL', potential) - subsys_section = section_add_subsection(subsys_section, kind_section) - - forces_section = make_section('FORCES', 'ON') - - print_section = make_section('PRINT') - print_section = section_add_subsection(print_section, forces_section) + #write coordinate to xyz file used by cp2k input + coord_list = sys_data['coords'][0] + u = np.array(atom_names) + atom_list = u[atom_types] + x = '\n' + for kind, coord in zip(atom_list, coord_list) : + x += str(kind) + ' ' + str(coord[:])[1:-1] + '\n' + return x - force_eval_section = make_section('FORCE_EVAL') - force_eval_section = section_add_keyword_and_value(force_eval_section, 'METHOD', 'QS') - force_eval_section = section_add_keyword_and_value(force_eval_section, 'STRESS_TENSOR', 'ANALYTICAL') - force_eval_section = section_add_subsection(force_eval_section, dft_section) - force_eval_section = section_add_subsection(force_eval_section, subsys_section) - force_eval_section = section_add_subsection(force_eval_section, print_section) - return global_section + force_eval_section diff --git a/dpgen/generator/lib/gaussian.py b/dpgen/generator/lib/gaussian.py index 2669e5f56..79a420e27 100644 --- a/dpgen/generator/lib/gaussian.py +++ b/dpgen/generator/lib/gaussian.py @@ -48,7 +48,7 @@ def _crd2frag(symbols, crds, pbc=False, cell=None, return_bonds=False): bond = mol.GetBond(ii) a = bond.GetBeginAtom().GetId() b = bond.GetEndAtom().GetId() - bo = bond.GetBO() + bo = bond.GetBondOrder() if a >= atomnumber and b >= atomnumber: # duplicated continue @@ -76,7 +76,11 @@ def _crd2mul(symbols, crds): mol = openbabel.OBMol() conv.ReadString(mol, xyzstring) gjfstring = conv.WriteString(mol) - mul = int(gjfstring.split('\n')[4].split()[1]) + try: + mul = int(gjfstring.split('\n')[4].split()[1]) + except IndexError: + # openbabel 3.0 + mul = int(gjfstring.split('\n')[5].split()[1]) return mul @@ -176,6 +180,7 @@ def make_gaussian_input(sys_data, fp_params): def take_cluster(old_conf_name, type_map, idx, jdata): cutoff = jdata['cluster_cutoff'] + cutoff_hard = jdata.get('cluster_cutoff_hard', None) sys = dpdata.System(old_conf_name, fmt = 'lammps/dump', type_map = type_map) atom_names = sys['atom_names'] atom_types = sys['atom_types'] @@ -190,27 +195,42 @@ def take_cluster(old_conf_name, type_map, idx, jdata): distances = all_atoms.get_distances(idx, range(len(all_atoms)), mic=True) distancescutoff = distances < cutoff cutoff_atoms_idx = np.where(distancescutoff)[0] + if cutoff_hard is not None: + distancescutoff_hard = distances < cutoff_hard + cutoff_atoms_idx_hard = np.where(distancescutoff_hard)[0] # make cutoff atoms in molecules taken_atoms_idx = [] added = [] for ii in range(frag_numb): frag_atoms_idx = np.where(frag_index == ii)[0] + if cutoff_hard is not None: + # drop atoms out of the hard cutoff anyway + frag_atoms_idx = np.intersect1d(frag_atoms_idx, cutoff_atoms_idx_hard) if np.any(np.isin(frag_atoms_idx, cutoff_atoms_idx)): if 'cluster_minify' in jdata and jdata['cluster_minify']: - # currently support C, H + # support for organic species take_frag_idx=[] for aa in frag_atoms_idx: if np.any(np.isin(aa, cutoff_atoms_idx)): + # atom is in the soft cutoff + # pick up anyway take_frag_idx.append(aa) elif np.count_nonzero(np.logical_and(distancescutoff, graph.toarray()[aa]==1)): + # atom is between the hard cutoff and the soft cutoff + # and has a single bond with the atom inside if all_atoms[aa].symbol == 'H': + # for atom H: just add it take_frag_idx.append(aa) - elif all_atoms[aa].symbol == 'C': + else: + # for other atoms (C, O, etc.): replace it with a ghost H atom near_atom_idx = np.nonzero(np.logical_and(distancescutoff, graph.toarray()[aa]>0))[0][0] vector = all_atoms[aa].position - all_atoms[near_atom_idx].position new_position = all_atoms[near_atom_idx].position + vector / np.linalg.norm(vector) * 1.09 added.append(Atom('H', new_position)) elif np.count_nonzero(np.logical_and(distancescutoff, graph.toarray()[aa]>1)): + # if that atom has a double bond with the atom inside + # just pick up the whole fragment (within the hard cutoff) + # TODO: use a more fantastic method take_frag_idx=frag_atoms_idx break else: diff --git a/dpgen/generator/run.py b/dpgen/generator/run.py index 7fa034773..ceac94ca1 100644 --- a/dpgen/generator/run.py +++ b/dpgen/generator/run.py @@ -21,6 +21,7 @@ import warnings import shutil import time +import copy import dpdata import numpy as np import subprocess as sp @@ -51,7 +52,12 @@ from dpgen.remote.group_jobs import group_slurm_jobs from dpgen.remote.group_jobs import group_local_jobs from dpgen.remote.decide_machine import decide_train_machine, decide_fp_machine, decide_model_devi_machine -from dpgen.dispatcher.Dispatcher import Dispatcher, make_dispatcher +from dpgen.dispatcher.Dispatcher import Dispatcher, _split_tasks, make_dispatcher +try: + from dpgen.dispatcher.ALI import ALI +except ImportError as e: + dlog.info(e) + pass from dpgen.util import sepline from dpgen import ROOT_PATH from pymatgen.io.vasp import Incar,Kpoints,Potcar @@ -341,8 +347,7 @@ def detect_batch_size(batch_size, system=None): def run_train (iter_index, jdata, - mdata, - dispatcher) : + mdata) : # load json param numb_models = jdata['numb_models'] # train_param = jdata['train_param'] @@ -356,7 +361,8 @@ def run_train (iter_index, deepmd_path = mdata['deepmd_path'] else: # 1.x - python_path = mdata['python_path'] + python_path = mdata.get('python_path', None) + train_command = mdata.get('train_command', 'dp') train_resources = mdata['train_resources'] # paths @@ -379,12 +385,21 @@ def run_train (iter_index, commands.append(command) command = os.path.join(deepmd_path, 'bin/dp_frz') commands.append(command) - else: + elif python_path: # 1.x command = '%s -m deepmd train %s' % (python_path, train_input_file) commands.append(command) command = '%s -m deepmd freeze' % python_path commands.append(command) + else: + ## Commands are like `dp train` and `dp freeze` + ## train_command should not be None + assert(train_command) + command = '%s train %s' % (train_command, train_input_file) + commands.append(command) + command = '%s freeze' % train_command + commands.append(command) + #_tasks = [os.path.basename(ii) for ii in all_task] # run_tasks = [] @@ -415,17 +430,21 @@ def run_train (iter_index, for single_sys in os.listdir(os.path.join(ii)): trans_comm_data += glob.glob(os.path.join(ii, single_sys, 'set.*')) trans_comm_data += glob.glob(os.path.join(ii, single_sys, 'type.raw')) + trans_comm_data += glob.glob(os.path.join(ii, single_sys, 'nopbc')) else: trans_comm_data += glob.glob(os.path.join(ii, 'set.*')) trans_comm_data += glob.glob(os.path.join(ii, 'type.raw')) + trans_comm_data += glob.glob(os.path.join(ii, 'nopbc')) for ii in fp_data : if jdata.get('use_clusters', False): for single_sys in os.listdir(os.path.join(ii)): trans_comm_data += glob.glob(os.path.join(ii, single_sys, 'set.*')) trans_comm_data += glob.glob(os.path.join(ii, single_sys, 'type.raw')) + trans_comm_data += glob.glob(os.path.join(ii, single_sys, 'nopbc')) else: trans_comm_data += glob.glob(os.path.join(ii, 'set.*')) trans_comm_data += glob.glob(os.path.join(ii, 'type.raw')) + trans_comm_data += glob.glob(os.path.join(ii, 'nopbc')) os.chdir(cwd) try: @@ -433,6 +452,14 @@ def run_train (iter_index, except: train_group_size = 1 + task_chunks = _split_tasks(run_tasks, train_group_size) + nchunks = len(task_chunks) + if "ali_auth" in mdata: + dispatcher = ALI(mdata['ali_auth'], mdata['train_resources'], mdata['train_machine'], nchunks, work_path) + dispatcher.init() + else: + dispatcher = make_dispatcher(mdata['train_machine']) + dispatcher.run_jobs(mdata['train_resources'], commands, work_path, @@ -444,7 +471,6 @@ def run_train (iter_index, outlog = 'train.log', errlog = 'train.log') - def post_train (iter_index, jdata, mdata) : @@ -494,34 +520,94 @@ def parse_cur_job(cur_job) : dt = None return ensemble, nsteps, trj_freq, temps, press, pka_e, dt +def expand_matrix_values(target_list, cur_idx = 0): + nvar = len(target_list) + if cur_idx == nvar : + return [[]] + else : + res = [] + prev = expand_matrix_values(target_list, cur_idx+1) + for ii in target_list[cur_idx]: + tmp = copy.deepcopy(prev) + for jj in tmp: + jj.insert(0, ii) + res.append(jj) + return res + +def parse_cur_job_revmat(cur_job, use_plm = False): + templates = [cur_job['template']['lmp']] + if use_plm : + templates.append(cur_job['template']['plm']) + revise_keys = [] + revise_values = [] + if 'rev_mat' not in cur_job.keys(): + cur_job['rev_mat'] = {} + if 'lmp' not in cur_job['rev_mat'].keys(): + cur_job['rev_mat']['lmp'] = {} + for ii in cur_job['rev_mat']['lmp'].keys(): + revise_keys.append(ii) + revise_values.append(cur_job['rev_mat']['lmp'][ii]) + n_lmp_keys = len(revise_keys) + if use_plm: + if 'plm' not in cur_job['rev_mat'].keys(): + cur_job['rev_mat']['plm'] = {} + for ii in cur_job['rev_mat']['plm'].keys(): + revise_keys.append(ii) + revise_values.append(cur_job['rev_mat']['plm'][ii]) + revise_matrix = expand_matrix_values(revise_values) + return revise_keys, revise_matrix, n_lmp_keys + + +def find_only_one_key(lmp_lines, key): + found = [] + for idx in range(len(lmp_lines)): + words = lmp_lines[idx].split() + nkey = len(key) + if len(words) >= nkey and words[:nkey] == key : + found.append(idx) + if len(found) > 1: + raise RuntimeError('found %d keywords %s' % (len(found), key)) + if len(found) == 0: + raise RuntimeError('failed to find keyword %s' % (key)) + return found[0] + + +def revise_lmp_input_model(lmp_lines, task_model_list, trj_freq, deepmd_version = '1'): + idx = find_only_one_key(lmp_lines, ['pair_style', 'deepmd']) + graph_list = ' '.join(task_model_list) + if LooseVersion(deepmd_version) < LooseVersion('1'): + lmp_lines[idx] = "pair_style deepmd %s %d model_devi.out\n" % (graph_list, trj_freq) + else: + lmp_lines[idx] = "pair_style deepmd %s out_freq %d out_file model_devi.out\n" % (graph_list, trj_freq) + return lmp_lines + + +def revise_lmp_input_dump(lmp_lines, trj_freq): + idx = find_only_one_key(lmp_lines, ['dump', 'dpgen_dump']) + lmp_lines[idx] = "dump dpgen_dump all custom %d traj/*.lammpstrj id type x y z\n" % trj_freq + return lmp_lines + + +def revise_lmp_input_plm(lmp_lines, in_plm, out_plm = 'output.plumed'): + idx = find_only_one_key(lmp_lines, ['fix', 'dpgen_plm']) + lmp_lines[idx] = "fix dpgen_plm all plumed plumedfile %s outfile %s\n" % (in_plm, out_plm) + return lmp_lines + + +def revise_by_keys(lmp_lines, keys, values): + for kk,vv in zip(keys, values): + for ii in range(len(lmp_lines)): + lmp_lines[ii] = lmp_lines[ii].replace(kk, str(vv)) + return lmp_lines + + def make_model_devi (iter_index, jdata, mdata) : - use_ele_temp = jdata.get('use_ele_temp', 0) - model_devi_dt = jdata['model_devi_dt'] - model_devi_neidelay = None - if 'model_devi_neidelay' in jdata : - model_devi_neidelay = jdata['model_devi_neidelay'] - model_devi_taut = 0.1 - if 'model_devi_taut' in jdata : - model_devi_taut = jdata['model_devi_taut'] - model_devi_taup = 0.5 - if 'model_devi_taup' in jdata : - model_devi_taup = jdata['model_devi_taup'] model_devi_jobs = jdata['model_devi_jobs'] if (iter_index >= len(model_devi_jobs)) : return False cur_job = model_devi_jobs[iter_index] - # ensemble = model_devi_jobs['ensemble'] - # nsteps = model_devi_jobs['nsteps'] - # trj_freq = model_devi_jobs['trj_freq'] - # job_names = get_job_names (model_devi_jobs) - # assert (iter_index < len(job_names)) - # cur_job_name = job_names[iter_index] - # cur_job = model_devi_jobs[cur_job_name] - ensemble, nsteps, trj_freq, temps, press, pka_e, dt = parse_cur_job(cur_job) - if dt is not None : - model_devi_dt = dt if "sys_configs_prefix" in jdata: sys_configs = [] for sys_list in jdata["sys_configs"]: @@ -544,15 +630,11 @@ def make_model_devi (iter_index, cur_systems.sort() cur_systems = [os.path.abspath(ii) for ii in cur_systems] conf_systems.append (cur_systems) - mass_map = jdata['mass_map'] iter_name = make_iter_name(iter_index) train_path = os.path.join(iter_name, train_name) train_path = os.path.abspath(train_path) models = glob.glob(os.path.join(train_path, "graph*pb")) - task_model_list = [] - for ii in models: - task_model_list.append(os.path.join('..', os.path.basename(ii))) work_path = os.path.join(iter_name, model_devi_name) create_path(work_path) for mm in models : @@ -586,6 +668,136 @@ def make_model_devi (iter_index, conf_counter += 1 sys_counter += 1 + input_mode = "native" + if "template" in cur_job: + input_mode = "revise_template" + use_plm = jdata.get('model_devi_plumed', False) + if input_mode == "native": + _make_model_devi_native(iter_index, jdata, mdata, conf_systems) + elif input_mode == "revise_template": + _make_model_devi_revmat(iter_index, jdata, mdata, conf_systems) + else: + raise RuntimeError('unknown model_devi input mode', input_mode) + + return True + + +def _make_model_devi_revmat(iter_index, jdata, mdata, conf_systems): + model_devi_jobs = jdata['model_devi_jobs'] + if (iter_index >= len(model_devi_jobs)) : + return False + cur_job = model_devi_jobs[iter_index] + sys_idx = expand_idx(cur_job['sys_idx']) + if (len(sys_idx) != len(list(set(sys_idx)))) : + raise RuntimeError("system index should be uniq") + mass_map = jdata['mass_map'] + use_plm = jdata.get('model_devi_plumed', False) + trj_freq = _get_param_alias(cur_job, ['t_freq', 'trj_freq','traj_freq']) + + rev_keys, rev_mat, num_lmp = parse_cur_job_revmat(cur_job, use_plm = use_plm) + lmp_templ = cur_job['template']['lmp'] + lmp_templ = os.path.abspath(lmp_templ) + if use_plm: + plm_templ = cur_job['template']['plm'] + plm_templ = os.path.abspath(plm_templ) + + iter_name = make_iter_name(iter_index) + train_path = os.path.join(iter_name, train_name) + train_path = os.path.abspath(train_path) + models = glob.glob(os.path.join(train_path, "graph*pb")) + task_model_list = [] + for ii in models: + task_model_list.append(os.path.join('..', os.path.basename(ii))) + work_path = os.path.join(iter_name, model_devi_name) + try: + mdata["deepmd_version"] + except: + mdata = set_version(mdata) + deepmd_version = mdata['deepmd_version'] + + sys_counter = 0 + for ss in conf_systems: + conf_counter = 0 + task_counter = 0 + for cc in ss : + for ii in range(len(rev_mat)): + rev_item = rev_mat[ii] + task_name = make_model_devi_task_name(sys_idx[sys_counter], task_counter) + conf_name = make_model_devi_conf_name(sys_idx[sys_counter], conf_counter) + '.lmp' + task_path = os.path.join(work_path, task_name) + # create task path + create_path(task_path) + create_path(os.path.join(task_path, 'traj')) + # link conf + loc_conf_name = 'conf.lmp' + os.symlink(os.path.join(os.path.join('..','confs'), conf_name), + os.path.join(task_path, loc_conf_name) ) + cwd_ = os.getcwd() + # chdir to task path + os.chdir(task_path) + shutil.copyfile(lmp_templ, 'input.lammps') + # revise input of lammps + with open('input.lammps') as fp: + lmp_lines = fp.readlines() + lmp_lines = revise_lmp_input_model(lmp_lines, task_model_list, trj_freq, deepmd_version = deepmd_version) + lmp_lines = revise_lmp_input_dump(lmp_lines, trj_freq) + lmp_lines = revise_by_keys(lmp_lines, rev_keys[:num_lmp], rev_item[:num_lmp]) + # revise input of plumed + if use_plm: + lmp_lines = revise_lmp_input_plm(lmp_lines, 'input.plumed') + shutil.copyfile(plm_templ, 'input.plumed') + with open('input.plumed') as fp: + plm_lines = fp.readlines() + plm_lines = revise_by_keys(plm_lines, rev_keys[num_lmp:], rev_item[num_lmp:]) + with open('input.plumed', 'w') as fp: + fp.write(''.join(plm_lines)) + # dump input of lammps + with open('input.lammps', 'w') as fp: + fp.write(''.join(lmp_lines)) + with open('job.json', 'w') as fp: + job = {} + for ii,jj in zip(rev_keys, rev_item) : job[ii] = jj + json.dump(job, fp, indent = 4) + os.chdir(cwd_) + task_counter += 1 + conf_counter += 1 + sys_counter += 1 + + +def _make_model_devi_native(iter_index, jdata, mdata, conf_systems): + model_devi_jobs = jdata['model_devi_jobs'] + if (iter_index >= len(model_devi_jobs)) : + return False + cur_job = model_devi_jobs[iter_index] + ensemble, nsteps, trj_freq, temps, press, pka_e, dt = parse_cur_job(cur_job) + if dt is not None : + model_devi_dt = dt + sys_idx = expand_idx(cur_job['sys_idx']) + if (len(sys_idx) != len(list(set(sys_idx)))) : + raise RuntimeError("system index should be uniq") + + use_ele_temp = jdata.get('use_ele_temp', 0) + model_devi_dt = jdata['model_devi_dt'] + model_devi_neidelay = None + if 'model_devi_neidelay' in jdata : + model_devi_neidelay = jdata['model_devi_neidelay'] + model_devi_taut = 0.1 + if 'model_devi_taut' in jdata : + model_devi_taut = jdata['model_devi_taut'] + model_devi_taup = 0.5 + if 'model_devi_taup' in jdata : + model_devi_taup = jdata['model_devi_taup'] + mass_map = jdata['mass_map'] + + iter_name = make_iter_name(iter_index) + train_path = os.path.join(iter_name, train_name) + train_path = os.path.abspath(train_path) + models = glob.glob(os.path.join(train_path, "graph*pb")) + task_model_list = [] + for ii in models: + task_model_list.append(os.path.join('..', os.path.basename(ii))) + work_path = os.path.join(iter_name, model_devi_name) + sys_counter = 0 for ss in conf_systems: conf_counter = 0 @@ -666,16 +878,15 @@ def make_model_devi (iter_index, conf_counter += 1 sys_counter += 1 - return True def run_model_devi (iter_index, jdata, - mdata, - dispatcher) : + mdata) : #rmdlog.info("This module has been run !") lmp_exec = mdata['lmp_command'] model_devi_group_size = mdata['model_devi_group_size'] model_devi_resources = mdata['model_devi_resources'] + use_plm = jdata.get('model_devi_plumed', False) iter_name = make_iter_name(iter_index) work_path = os.path.join(iter_name, model_devi_name) @@ -688,8 +899,6 @@ def run_model_devi (iter_index, fp = open (os.path.join(work_path, 'cur_job.json'), 'r') cur_job = json.load (fp) - ensemble, nsteps, trj_freq, temps, press, pka_e, dt = parse_cur_job(cur_job) - nframes = nsteps // trj_freq + 1 run_tasks_ = all_task # for ii in all_task: @@ -708,7 +917,18 @@ def run_model_devi (iter_index, model_names = [os.path.basename(ii) for ii in all_models] forward_files = ['conf.lmp', 'input.lammps', 'traj'] backward_files = ['model_devi.out', 'model_devi.log', 'traj'] + if use_plm: + forward_files += ['input.plumed'] + backward_files += ['output.plumed'] + cwd = os.getcwd() + task_chunks = _split_tasks(run_tasks, model_devi_group_size) + nchunks = len(task_chunks) + if "ali_auth" in mdata: + dispatcher = ALI(mdata['ali_auth'], mdata['model_devi_resources'], mdata['model_devi_machine'], nchunks, work_path) + dispatcher.init() + else: + dispatcher = make_dispatcher(mdata['model_devi_machine']) dispatcher.run_jobs(mdata['model_devi_resources'], commands, work_path, @@ -1267,11 +1487,10 @@ def _cp2k_check_fin(ii): def run_fp_inner (iter_index, jdata, mdata, - dispatcher, forward_files, backward_files, check_fin, - log_file = "log", + log_file = "fp.log", forward_common_files=[]) : fp_command = mdata['fp_command'] fp_group_size = mdata['fp_group_size'] @@ -1290,7 +1509,14 @@ def run_fp_inner (iter_index, # if not check_fin(ii) : # fp_run_tasks.append(ii) run_tasks = [os.path.basename(ii) for ii in fp_run_tasks] - + cwd = os.getcwd() + task_chunks = _split_tasks(run_tasks, fp_group_size) + nchunks = len(task_chunks) + if "ali_auth" in mdata: + dispatcher = ALI(mdata['ali_auth'], mdata['fp_resources'], mdata['fp_machine'], nchunks, work_path) + dispatcher.init() + else: + dispatcher = make_dispatcher(mdata['fp_machine']) dispatcher.run_jobs(mdata['fp_resources'], [fp_command], work_path, @@ -1303,10 +1529,10 @@ def run_fp_inner (iter_index, errlog = log_file) + def run_fp (iter_index, jdata, - mdata, - dispatcher) : + mdata) : fp_style = jdata['fp_style'] fp_pp_files = jdata['fp_pp_files'] @@ -1322,24 +1548,24 @@ def run_fp (iter_index, forward_files.append('KPOINTS') else: forward_common_files=[] - run_fp_inner(iter_index, jdata, mdata, dispatcher, forward_files, backward_files, _vasp_check_fin, + run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _vasp_check_fin, forward_common_files=forward_common_files) elif fp_style == "pwscf" : forward_files = ['input'] + fp_pp_files backward_files = ['output'] - run_fp_inner(iter_index, jdata, mdata, dispatcher, forward_files, backward_files, _qe_check_fin, log_file = 'output') + run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _qe_check_fin, log_file = 'output') elif fp_style == "siesta": forward_files = ['input'] + fp_pp_files backward_files = ['output'] - run_fp_inner(iter_index, jdata, mdata, dispatcher, forward_files, backward_files, _siesta_check_fin, log_file='output') + run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _siesta_check_fin, log_file='output') elif fp_style == "gaussian": forward_files = ['input'] backward_files = ['output'] - run_fp_inner(iter_index, jdata, mdata, dispatcher, forward_files, backward_files, _gaussian_check_fin, log_file = 'output') + run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _gaussian_check_fin, log_file = 'output') elif fp_style == "cp2k": forward_files = ['input.inp', 'coord.xyz'] backward_files = ['output'] - run_fp_inner(iter_index, jdata, mdata, dispatcher, forward_files, backward_files, _cp2k_check_fin, log_file = 'output') + run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _cp2k_check_fin, log_file = 'output') else : raise RuntimeError ("unsupported fp style") @@ -1633,17 +1859,21 @@ def post_fp (iter_index, def set_version(mdata): if 'deepmd_path' in mdata: deepmd_version = '0.1' - elif 'python_path' in mdata: - deepmd_version = '1' + #elif 'python_path' in mdata: + # deepmd_version = '1' + #elif 'train_command' in mdata: + # deepmd_version = '1' elif 'train' in mdata: if 'deepmd_path' in mdata['train'][0]: deepmd_version = '0.1' - elif 'python_path' in mdata['train'][0]: - deepmd_version = '1' else: - deepmd_version = '0.1' + deepmd_version = '1' + # elif 'python_path' in mdata['train'][0]: + # deepmd_version = '1' + # elif 'command' in mdata['train']: + # deepmd_version = '1' else: - deepmd_version = '0.1' + deepmd_version = '1' # set mdata['deepmd_version'] = deepmd_version return mdata @@ -1701,15 +1931,14 @@ def run_iter (param_file, machine_file) : if ii * max_tasks + jj <= iter_rec[0] * max_tasks + iter_rec[1] : continue task_name="task %02d"%jj - sepline(task_name,'-') + sepline("{} {}".format(iter_name, task_name),'-') if jj == 0 : log_iter ("make_train", ii, jj) make_train (ii, jdata, mdata) elif jj == 1 : log_iter ("run_train", ii, jj) mdata = decide_train_machine(mdata) - disp = make_dispatcher(mdata['train_machine']) - run_train (ii, jdata, mdata, disp) + run_train (ii, jdata, mdata) elif jj == 2 : log_iter ("post_train", ii, jj) post_train (ii, jdata, mdata) @@ -1721,8 +1950,8 @@ def run_iter (param_file, machine_file) : elif jj == 4 : log_iter ("run_model_devi", ii, jj) mdata = decide_model_devi_machine(mdata) - disp = make_dispatcher(mdata['model_devi_machine']) - run_model_devi (ii, jdata, mdata, disp) + run_model_devi (ii, jdata, mdata) + elif jj == 5 : log_iter ("post_model_devi", ii, jj) post_model_devi (ii, jdata, mdata) @@ -1732,8 +1961,7 @@ def run_iter (param_file, machine_file) : elif jj == 7 : log_iter ("run_fp", ii, jj) mdata = decide_fp_machine(mdata) - disp = make_dispatcher(mdata['fp_machine']) - run_fp (ii, jdata, mdata, disp) + run_fp (ii, jdata, mdata) elif jj == 8 : log_iter ("post_fp", ii, jj) post_fp (ii, jdata) diff --git a/dpgen/main.py b/dpgen/main.py index c3a37260f..f5aff6583 100644 --- a/dpgen/main.py +++ b/dpgen/main.py @@ -9,9 +9,12 @@ from dpgen.generator.run import gen_run from dpgen.data.gen import gen_init_bulk from dpgen.data.surf import gen_init_surf +from dpgen.data.reaction import gen_init_reaction +from dpgen.simplify.simplify import gen_simplify from dpgen.auto_test.run import gen_test from dpgen.database.run import db_run from dpgen.tools.run_report import run_report +from dpgen.tools.auto_gen_param import auto_gen_param from dpgen import info, __version__, __date__ @@ -54,6 +57,13 @@ def main(): parser_init_bulk.add_argument('MACHINE', type=str,default=None,nargs="?", help="machine file, json/yaml format") parser_init_bulk.set_defaults(func=gen_init_bulk) + + parser_auto_gen_param = subparsers.add_parser( + "auto_gen_param", help="auto gen param.json") + # parser_auto_gen_param.add_argument('meltpoint', type=float, help="melt point") + parser_auto_gen_param.add_argument('PARAM', type=str, + help="parameter file, json/yaml format") + parser_auto_gen_param.set_defaults(func=auto_gen_param) # parser_init.add_argument("-p",'--parameter', type=str, dest='param', # help="parameter file, json/yaml format") # parser_init.add_argument("-s","--stage", type=int, dest='stage', @@ -67,6 +77,14 @@ def main(): # help="directory to process (default to .)") # parser_init.set_defaults(func=gen_data) + parser_init_reaction = subparsers.add_parser( + "init_reaction", help="Generating initial data for reactive systems.") + parser_init_reaction.add_argument('PARAM', type=str, + help="parameter file, json/yaml format") + parser_init_reaction.add_argument('MACHINE', type=str,default=None,nargs="?", + help="machine file, json/yaml format") + parser_init_reaction.set_defaults(func=gen_init_reaction) + # run parser_run = subparsers.add_parser( "run", @@ -97,6 +115,18 @@ def main(): help="being loud") parser_rr.set_defaults(func=run_report) + # simplify + parser_run = subparsers.add_parser( + "simplify", + help="Simplify data.") + parser_run.add_argument('PARAM', type=str, + help="parameter file, json/yaml format") + parser_run.add_argument('MACHINE', type=str, + help="machine file, json/yaml format") + parser_run.add_argument('-d','--debug', action='store_true', + help="log debug info") + parser_run.set_defaults(func=gen_simplify) + # test parser_test = subparsers.add_parser("test", help="Auto-test for Deep Potential.") parser_test.add_argument('PARAM', type=str, diff --git a/dpgen/remote/decide_machine.py b/dpgen/remote/decide_machine.py index 3a0e3ecf3..8484731be 100644 --- a/dpgen/remote/decide_machine.py +++ b/dpgen/remote/decide_machine.py @@ -13,7 +13,6 @@ def decide_train_machine(mdata): if 'train' in mdata: continue_flag = False - ## decide whether to use an existing machine if 'record.machine' in os.listdir(): try: with open('record.machine', 'r') as _infile: @@ -32,7 +31,7 @@ def decide_train_machine(mdata): continue_flag = True except: pass - if "hostname" not in mdata["train"][0]["machine"]: + if ("hostname" not in mdata["train"][0]["machine"]) or (len(mdata["train"]) == 1): mdata["train_machine"] = mdata["train"][0]["machine"] mdata["train_resources"] = mdata["train"][0]["resources"] if 'deepmd_path' in mdata["train"][0]: @@ -43,8 +42,10 @@ def decide_train_machine(mdata): mdata["train_group_size"] = mdata["train"][0]["group_size"] if 'deepmd_version' in mdata["train"][0]: mdata["deepmd_version"] = mdata["train"][0]["deepmd_version"] + if 'command' in mdata["train"][0]: + mdata["train_command"] = mdata["train"][0]["command"] continue_flag = True - + pd_flag = False pd_count_list =[] # pd for pending job in slurm @@ -133,13 +134,14 @@ def decide_model_devi_machine(mdata): continue_flag = True except: pass - if "hostname" not in mdata["model_devi"][0]["machine"]: + if ("hostname" not in mdata["model_devi"][0]["machine"]) or (len(mdata["model_devi"]) == 1): mdata["model_devi_machine"] = mdata["model_devi"][0]["machine"] mdata["model_devi_resources"] = mdata["model_devi"][0]["resources"] mdata["lmp_command"] = mdata["model_devi"][0]["command"] #if "group_size" in mdata["train"][0]: mdata["model_devi_group_size"] = mdata["model_devi"][0]["group_size"] continue_flag = True + pd_count_list =[] pd_flag = False if not continue_flag: @@ -209,13 +211,15 @@ def decide_fp_machine(mdata): continue_flag = True except: pass - if "hostname" not in mdata["fp"][0]["machine"]: + if ("hostname" not in mdata["fp"][0]["machine"]) or (len(mdata["fp"]) == 1): mdata["fp_machine"] = mdata["fp"][0]["machine"] mdata["fp_resources"] = mdata["fp"][0]["resources"] mdata["fp_command"] = mdata["fp"][0]["command"] #if "group_size" in mdata["train"][0]: mdata["fp_group_size"] = mdata["fp"][0]["group_size"] continue_flag = True + + pd_count_list =[] pd_flag = False if not continue_flag: @@ -256,6 +260,5 @@ def decide_fp_machine(mdata): profile['group_size'] = mdata['fp_group_size'] profile['command'] = mdata['fp_command'] json.dump(profile, _outfile, indent = 4) -# print("mdata", mdata) return mdata diff --git a/dpgen/simplify/__init__.py b/dpgen/simplify/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/dpgen/simplify/simplify.py b/dpgen/simplify/simplify.py new file mode 100644 index 000000000..ba99b0b6c --- /dev/null +++ b/dpgen/simplify/simplify.py @@ -0,0 +1,422 @@ +"""Simplify dataset (minimize the dataset size). + +Init: +pick up init data from dataset randomly + +Iter: +00: train models (same as generator) +01: calculate model deviations of the rest dataset, pick up data with proper model deviaiton +02: fp (optional, if the original dataset do not have fp data, same as generator) +""" +import logging +import queue +import os +import json +import argparse +import pickle +import glob + +import dpdata +import numpy as np + +from dpgen import dlog +from dpgen import SHORT_CMD +from dpgen.util import sepline +from dpgen.remote.decide_machine import decide_train_machine +from dpgen.dispatcher.Dispatcher import Dispatcher, make_dispatcher +from dpgen.generator.run import make_train, run_train, post_train, run_fp, post_fp, fp_name, model_devi_name, train_name +# TODO: maybe the following functions can be moved to dpgen.util +from dpgen.generator.lib.utils import log_iter, make_iter_name, create_path, record_iter +from dpgen.remote.decide_machine import decide_train_machine, decide_fp_machine, decide_model_devi_machine +from dpgen.generator.lib.gaussian import make_gaussian_input + + +picked_data_name = "data.picked" +rest_data_name = "data.rest" +accurate_data_name = "data.accurate" +detail_file_name_prefix = "details" + + +def get_system_cls(jdata): + if jdata.get("labeled", False): + return dpdata.LabeledSystem + return dpdata.System + + +def get_systems(path, jdata): + system = get_system_cls(jdata) + systems = dpdata.MultiSystems( + *[system(os.path.join(path, s), fmt='deepmd/npy') for s in os.listdir(path)]) + return systems + + +def init_pick(iter_index, jdata, mdata): + """pick up init data from dataset randomly""" + pick_data = jdata['pick_data'] + init_pick_number = jdata['init_pick_number'] + # use MultiSystems with System + # TODO: support System and LabeledSystem + # TODO: support other format + systems = get_systems(pick_data, jdata) + # label the system + labels = [] + for key, system in systems.systems.items(): + labels.extend([(key, j) for j in range(len(system))]) + + # random pick + iter_name = make_iter_name(iter_index) + create_path(iter_name) + work_path = os.path.join(iter_name, model_devi_name) + create_path(work_path) + idx = np.arange(len(labels)) + np.random.shuffle(idx) + pick_idx = idx[:init_pick_number] + rest_idx = idx[init_pick_number:] + + # dump the init data + picked_systems = dpdata.MultiSystems() + for j in pick_idx: + sys_name, sys_id = labels[j] + picked_systems.append(systems[sys_name][sys_id]) + sys_data_path = os.path.join(work_path, picked_data_name) + + picked_systems.to_deepmd_raw(sys_data_path) + picked_systems.to_deepmd_npy(sys_data_path, set_size=init_pick_number) + + # dump the rest data + rest_systems = dpdata.MultiSystems() + for j in rest_idx: + sys_name, sys_id = labels[j] + rest_systems.append(systems[sys_name][sys_id]) + sys_data_path = os.path.join(work_path, rest_data_name) + rest_systems.to_deepmd_raw(sys_data_path) + rest_systems.to_deepmd_npy(sys_data_path, set_size=rest_idx.size) + + +def make_model_devi(iter_index, jdata, mdata): + """calculate the model deviation of the rest idx""" + iter_name = make_iter_name(iter_index) + work_path = os.path.join(iter_name, model_devi_name) + create_path(work_path) + # link the model + train_path = os.path.join(iter_name, train_name) + train_path = os.path.abspath(train_path) + models = glob.glob(os.path.join(train_path, "graph*pb")) + for mm in models: + model_name = os.path.basename(mm) + os.symlink(mm, os.path.join(work_path, model_name)) + # link the last rest data + last_iter_name = make_iter_name(iter_index-1) + rest_data_path = os.path.join(last_iter_name, model_devi_name, rest_data_name) + if not os.path.exists(rest_data_path): + return False + for jj, subsystem in enumerate(os.listdir(rest_data_path)): + task_name = "task.%03d.%06d" % (0, jj) + task_path = os.path.join(work_path, task_name) + create_path(task_path) + os.symlink(os.path.abspath(os.path.join(rest_data_path, subsystem)), + os.path.abspath(os.path.join(task_path, rest_data_name))) + return True + + +def run_model_devi(iter_index, jdata, mdata, dispatcher): + """submit dp test tasks""" + iter_name = make_iter_name(iter_index) + work_path = os.path.join(iter_name, model_devi_name) + # generate command + commands = [] + tasks = glob.glob(os.path.join(work_path, "task.*")) + run_tasks = [os.path.basename(ii) for ii in tasks] + # get models + models = glob.glob(os.path.join(work_path, "graph*pb")) + model_names = [os.path.basename(ii) for ii in models] + task_model_list = [] + for ii in model_names: + task_model_list.append(os.path.join('..', ii)) + # get max data size + data_size = max([len(dpdata.System(os.path.join( + task, rest_data_name), fmt="deepmd/npy")) for task in tasks]) + # models + commands = [] + detail_file_names = [] + for ii, mm in enumerate(task_model_list): + detail_file_name = "{prefix}.{ii}".format( + prefix=detail_file_name_prefix, + ii=ii, + ) + # TODO: support 0.x? + command = "{python} -m deepmd test -m {model} -s {system} -n {numb_test} -d {detail_file}".format( + python=mdata['python_test_path'], + model=mm, + system=rest_data_name, + numb_test=data_size, + detail_file=detail_file_name, + ) + commands.append(command) + detail_file_names.append(detail_file_name) + # submit + try: + model_devi_group_size = mdata['model_devi_group_size'] + except: + model_devi_group_size = 1 + + forward_files = [rest_data_name] + backward_files = sum([[pf+".e.out", pf+".f.out", pf+".v.out"] for pf in detail_file_names], []) + + dispatcher.run_jobs(mdata['model_devi_resources'], + commands, + work_path, + run_tasks, + model_devi_group_size, + model_names, + forward_files, + backward_files, + outlog='model_devi.log', + errlog='model_devi.log') + + +def post_model_devi(iter_index, jdata, mdata): + """calculate the model deviation""" + iter_name = make_iter_name(iter_index) + work_path = os.path.join(iter_name, model_devi_name) + tasks = glob.glob(os.path.join(work_path, "task.*")) + + e_trust_lo = jdata['e_trust_lo'] + e_trust_hi = jdata['e_trust_hi'] + f_trust_lo = jdata['f_trust_lo'] + f_trust_hi = jdata['f_trust_hi'] + + sys_accurate = dpdata.MultiSystems() + sys_candinate = dpdata.MultiSystems() + sys_failed = dpdata.MultiSystems() + + for task in tasks: + # e.out + details_e = glob.glob(os.path.join(task, "{}.*.e.out".format(detail_file_name_prefix))) + e_all = np.array([np.loadtxt(detail_e, ndmin=2)[:, 1] for detail_e in details_e]) + e_std = np.std(e_all, axis=0) + n_frame = e_std.size + + # f.out + details_f = glob.glob(os.path.join(task, "{}.*.f.out".format(detail_file_name_prefix))) + f_all = np.array([np.loadtxt(detail_f, ndmin=2)[:, 3:6].reshape((n_frame, -1, 3)) for detail_f in details_f]) + # (n_model, n_frame, n_atom, 3) + f_std = np.std(f_all, axis=0) + # (n_frame, n_atom, 3) + f_std = np.linalg.norm(f_std, axis=2) + # (n_frame, n_atom) + f_std = np.max(f_std, axis=1) + # (n_frame,) + + system_cls = get_system_cls(jdata) + for subsys, e_devi, f_devi in zip(system_cls(os.path.join(task, rest_data_name), fmt='deepmd/npy'), e_std, f_std): + if (e_devi < e_trust_hi and e_devi >= e_trust_lo) or (f_devi < f_trust_hi and f_devi >= f_trust_lo) : + sys_candinate.append(subsys) + elif (e_devi >= e_trust_hi ) or (f_devi >= f_trust_hi ): + sys_failed.append(subsys) + elif (e_devi < e_trust_lo and f_devi < f_trust_lo ): + sys_accurate.append(subsys) + counter = {"candidate": sys_candinate.get_nframes(), "accurate": sys_accurate.get_nframes(), "failed": sys_failed.get_nframes()} + fp_sum = sum(counter.values()) + for cc_key, cc_value in counter.items(): + dlog.info("{0:9s} : {1:6d} in {2:6d} {3:6.2f} %".format(cc_key, cc_value, fp_sum, cc_value/fp_sum*100)) + + # label the candidate system + labels = [] + for key, system in sys_candinate.systems.items(): + labels.extend([(key, j) for j in range(len(system))]) + # candinate: pick up randomly + iter_pick_number = jdata['iter_pick_number'] + idx = np.arange(counter['candidate']) + np.random.shuffle(idx) + pick_idx = idx[:iter_pick_number] + rest_idx = idx[iter_pick_number:] + + # dump the picked candinate data + picked_systems = dpdata.MultiSystems() + for j in pick_idx: + sys_name, sys_id = labels[j] + picked_systems.append(sys_candinate[sys_name][sys_id]) + sys_data_path = os.path.join(work_path, picked_data_name) + + picked_systems.to_deepmd_raw(sys_data_path) + picked_systems.to_deepmd_npy(sys_data_path, set_size=iter_pick_number) + + # dump the rest data (not picked candinate data and failed data) + rest_systems = dpdata.MultiSystems() + for j in rest_idx: + sys_name, sys_id = labels[j] + rest_systems.append(sys_candinate[sys_name][sys_id]) + rest_systems += sys_failed + sys_data_path = os.path.join(work_path, rest_data_name) + rest_systems.to_deepmd_raw(sys_data_path) + rest_systems.to_deepmd_npy(sys_data_path, set_size=rest_idx.size) + + # dump the accurate data -- to another directory + sys_data_path = os.path.join(work_path, accurate_data_name) + sys_accurate.to_deepmd_raw(sys_data_path) + sys_accurate.to_deepmd_npy(sys_data_path, set_size=sys_accurate.get_nframes()) + + +def make_fp(iter_index, jdata, mdata): + iter_name = make_iter_name(iter_index) + work_path = os.path.join(iter_name, fp_name) + create_path(work_path) + picked_data_path = os.path.join(iter_name, model_devi_name, picked_data_name) + if jdata.get("labeled", False): + dlog.info("already labeled, skip make_fp and link data directly") + os.symlink(os.path.abspath(picked_data_path), os.path.abspath( + os.path.join(work_path, "task.%03d" % 0))) + os.symlink(os.path.abspath(picked_data_path), os.path.abspath( + os.path.join(work_path, "data.%03d" % 0))) + return + systems = get_systems(picked_data_path, jdata) + fp_style = jdata['fp_style'] + if 'user_fp_params' in jdata.keys() : + fp_params = jdata['user_fp_params'] + else: + fp_params = jdata['fp_params'] + jj = 0 + for system in systems: + for subsys in system: + sys_data = subsys.data + task_name = "task.%03d.%06d" % (0, jj) + task_path = os.path.join(work_path, task_name) + create_path(task_path) + if fp_style == "gaussian" : + ret = make_gaussian_input(sys_data, fp_params) + with open(os.path.join(task_path, 'input'), 'w') as fp: + fp.write(ret) + else : + # TODO: support other formats + raise RuntimeError ("unsupported fp style") + jj += 1 + + +def run_iter(param_file, machine_file): + """ init (iter 0): init_pick + + tasks (iter > 0): + 00 make_train (same as generator) + 01 run_train (same as generator) + 02 post_train (same as generator) + 03 make_model_devi + 04 run_model_devi + 05 post_model_devi + 06 make_fp + 07 run_fp (same as generator) + 08 post_fp (same as generator) + """ + # TODO: function of handling input json should be combined as one function + try: + import ruamel + from monty.serialization import loadfn, dumpfn + warnings.simplefilter( + 'ignore', ruamel.yaml.error.MantissaNoDotYAML1_1Warning) + jdata = loadfn(param_file) + mdata = loadfn(machine_file) + except: + with open(param_file, 'r') as fp: + jdata = json.load(fp) + with open(machine_file, 'r') as fp: + mdata = json.load(fp) + + if jdata.get('pretty_print', False): + fparam = SHORT_CMD+'_' + \ + param_file.split('.')[0]+'.'+jdata.get('pretty_format', 'json') + dumpfn(jdata, fparam, indent=4) + fmachine = SHORT_CMD+'_' + \ + machine_file.split('.')[0]+'.'+jdata.get('pretty_format', 'json') + dumpfn(mdata, fmachine, indent=4) + + if mdata.get('handlers', None): + if mdata['handlers'].get('smtp', None): + que = queue.Queue(-1) + queue_handler = logging.handlers.QueueHandler(que) + smtp_handler = logging.handlers.SMTPHandler( + **mdata['handlers']['smtp']) + listener = logging.handlers.QueueListener(que, smtp_handler) + dlog.addHandler(queue_handler) + listener.start() + + max_tasks = 10000 + numb_task = 9 + record = "record.dpgen" + iter_rec = [0, -1] + if os.path.isfile(record): + with open(record) as frec: + for line in frec: + iter_rec = [int(x) for x in line.split()] + dlog.info("continue from iter %03d task %02d" % + (iter_rec[0], iter_rec[1])) + + cont = True + ii = -1 + while cont: + ii += 1 + iter_name = make_iter_name(ii) + sepline(iter_name, '=') + for jj in range(numb_task): + if ii * max_tasks + jj <= iter_rec[0] * max_tasks + iter_rec[1]: + continue + task_name = "task %02d" % jj + sepline("{} {}".format(iter_name, task_name), '-') + jdata['model_devi_jobs'] = [{} for _ in range(ii+1)] + if ii == 0 and jj < 6: + if jj == 0: + log_iter("init_pick", ii, jj) + init_pick(ii, jdata, mdata) + dlog.info("first iter, skip step 1-5") + elif jj == 0: + log_iter("make_train", ii, jj) + make_train(ii, jdata, mdata) + elif jj == 1: + log_iter("run_train", ii, jj) + mdata = decide_train_machine(mdata) + disp = make_dispatcher(mdata['train_machine']) + run_train(ii, jdata, mdata) + elif jj == 2: + log_iter("post_train", ii, jj) + post_train(ii, jdata, mdata) + elif jj == 3: + log_iter("make_model_devi", ii, jj) + cont = make_model_devi(ii, jdata, mdata) + if not cont or ii >= jdata.get("stop_iter", ii+1): + break + elif jj == 4: + log_iter("run_model_devi", ii, jj) + mdata = decide_model_devi_machine(mdata) + disp = make_dispatcher(mdata['model_devi_machine']) + run_model_devi(ii, jdata, mdata, disp) + elif jj == 5: + log_iter("post_model_devi", ii, jj) + post_model_devi(ii, jdata, mdata) + elif jj == 6: + log_iter("make_fp", ii, jj) + make_fp(ii, jdata, mdata) + elif jj == 7: + log_iter("run_fp", ii, jj) + if jdata.get("labeled", False): + dlog.info("already have labeled data, skip run_fp") + else: + mdata = decide_fp_machine(mdata) + disp = make_dispatcher(mdata['fp_machine']) + run_fp(ii, jdata, mdata) + elif jj == 8: + log_iter("post_fp", ii, jj) + if jdata.get("labeled", False): + dlog.info("already have labeled data, skip post_fp") + else: + post_fp(ii, jdata) + else: + raise RuntimeError("unknown task %d, something wrong" % jj) + record_iter(record, ii, jj) + + +def gen_simplify(args): + if args.PARAM and args.MACHINE: + if args.debug: + dlog.setLevel(logging.DEBUG) + dlog.info("start simplifying") + run_iter(args.PARAM, args.MACHINE) + dlog.info("finished") diff --git a/dpgen/tools/auto_gen_param.py b/dpgen/tools/auto_gen_param.py index 75faa7337..cd6252d3d 100755 --- a/dpgen/tools/auto_gen_param.py +++ b/dpgen/tools/auto_gen_param.py @@ -224,7 +224,7 @@ def get_init_data_sys(scan_dir='./', init_file_name='type.raw'): def get_basic_param_json(melt_point, - out_param_filename='param.json', + out_param_filename='param_basic.json', scan_dir="./", file_name='POSCAR', init_file_name='type.raw', @@ -273,6 +273,7 @@ def get_basic_param_json(melt_point, with open(out_param_filename, 'w') as p: json.dump(param_dict, p, indent=4) + return param_dict def _main(): parser = argparse.ArgumentParser(description='Collect data from inputs and generate basic param.json') parser.add_argument("melt_point", type=float, help="melt_point") @@ -282,5 +283,14 @@ def _main(): if __name__=='__main__': _main() - + +def auto_gen_param(args): + if args.PARAM: + with open(args.PARAM) as p: + j = json.load(p) + melt_point = j['melt_point'] + print('param_basic.json', get_basic_param_json(melt_point=melt_point)) + else: + raise RuntimeError('must provide melt point or PARAM') + #%% diff --git a/dpgen/tools/run_report.py b/dpgen/tools/run_report.py index ec0ef6dbd..17751c9c7 100755 --- a/dpgen/tools/run_report.py +++ b/dpgen/tools/run_report.py @@ -4,8 +4,8 @@ import numpy as np import subprocess as sp from dpgen.tools.stat_sys import stat_sys -from dpgen.tools.stat_iter import stat_iter, stat_time - +from dpgen.tools.stat_iter import stat_iter +from dpgen.tools.stat_time import stat_time def run_report(args): report_count = 0 diff --git a/dpgen/tools/stat_iter.py b/dpgen/tools/stat_iter.py index 228b051d4..519486727 100644 --- a/dpgen/tools/stat_iter.py +++ b/dpgen/tools/stat_iter.py @@ -22,10 +22,10 @@ def stat_iter(target_folder, num, relative_path_doc = line.strip().split(' ') path_doc = os.path.abspath(relative_path_doc) num = int(num) - prefix, iter_dirname, stage, out_filename = path_doc.rsplit('/',3) + prefix, iter_dirname, stage, out_filename = path_doc.rsplit('/',3) # pylint: disable=unused-variable pk_id, out_filename = path_doc.rsplit('/', 1) - iter = int(iter_dirname.split('.')[-1]) - out_id = int(out_filename.strip().split('.')[-2]) + iter = int(iter_dirname.split('.')[-1]) # pylint: disable=unused-variable + out_id = int(out_filename.strip().split('.')[-2]) # pylint: disable=unused-variable out_type = out_filename.strip().split('.')[0] iter_dict[pk_id][out_type] += num # for ii in @@ -39,7 +39,7 @@ def stat_iter(target_folder, if line: # [/home/felix/workplace/SiC/iter.000002/02.fp/task.018.000040/OUTCAR] path_doc = os.path.abspath(line) - pk_id, task_dirname, OUTCAR_filename=path_doc.rsplit('/', 2) + pk_id, task_dirname, OUTCAR_filename=path_doc.rsplit('/', 2) # pylint: disable=unused-variable try: _sys = dpdata.LabeledSystem(path_doc, type_map = jdata['type_map'] ) except: @@ -65,13 +65,5 @@ def stat_iter(target_folder, f":OUTCAR_not_convergence:{value['OUTCAR_not_convergence']}" f":reff:{value['reff']}") -def stat_time(target_folder, - param_file = 'param.json', - verbose = True, - mute = False): - script = os.path.join(os.path.dirname(__file__), 'update_time.sh') - output = subprocess.run([f'bash {script} {target_folder}'], - shell=True,stdout=subprocess.PIPE).stdout - data = output.decode() - print(data) + diff --git a/dpgen/tools/stat_time.py b/dpgen/tools/stat_time.py new file mode 100755 index 000000000..8e3a286fc --- /dev/null +++ b/dpgen/tools/stat_time.py @@ -0,0 +1,105 @@ +import subprocess +import os +def stat_time(target_folder, + param_file = 'param.json', + verbose = True, + mute = False): + train_dirs = subprocess.run([f"ls -d -1 {target_folder}/iter.??????/00.train/", ], + shell=True,stdout=subprocess.PIPE).stdout.decode().strip().split('\n') + for dir in train_dirs: + abs_dir = os.path.abspath(dir) + stage = os.path.basename(os.path.dirname(dir)) + train_time_logs = subprocess.run([f"grep -H --text 'wall time' {dir}/???/train.log", ], + shell=True,stdout=subprocess.PIPE).stdout.decode().strip().split('\n') + upload_task_dir_num = subprocess.run([f"ls -1 -d {dir}/??? |wc -l", ], + shell=True, stdout=subprocess.PIPE).stdout.decode().strip('\n') + total_core_sec = float(0) + + # assume training on single GPU + paral_cores = 1 + finished_task_file_num = len(train_time_logs) + # gpu_type_set = set([]) + for log in train_time_logs: + # log example : + # .//iter.000000/00.train//003/train.log:# DEEPMD: wall time: 7960.265 s + # print(log.split(':')) + file_path, text1, text2, wall_time = log.split(':') # pylint: disable=unused-variable + abs_file_path = os.path.abspath(file_path) + # stage=='00.train' + + wall_time_sec = float(wall_time.strip('s').strip(' ')) + total_core_sec += wall_time_sec * paral_cores + + # r'd\nja\1lgje' leading 'r' means dont treat '\' as Escape character + # gpu_type = subprocess.run([fr"grep -e 'physical GPU' {abs_file_path} |sed -n -E -e 's|^.*name: (.*), pci.*|\1|p'", ], + # shell=True,stdout=subprocess.PIPE).stdout.decode().strip().split('\n').pop() + # gpu_type_set.add(gpu_type) + + total_core_hour = total_core_sec * paral_cores / 3600 + print(f"{stage}:{abs_dir}" + f"paral_cores:{paral_cores}" + f":upload_task_dir_num:{upload_task_dir_num}" + f":finished_task_file_num:{finished_task_file_num}" + f":total_core_hour:{total_core_hour:.3f}") + + model_devi_dirs = subprocess.run([f"ls -d -1 {target_folder}/iter.??????/01.model_devi/", ], + shell=True,stdout=subprocess.PIPE).stdout.decode().strip().split('\n') + # print(model_devi_dirs) + for dir in model_devi_dirs: + abs_dir = os.path.abspath(dir) + stage = os.path.basename(os.path.dirname(dir)) + # print(dir) + model_devi_time_logs = subprocess.run([f"grep -H --text 'wall time' {dir}/task.*/log.lammps", ], + shell=True,stdout=subprocess.PIPE).stdout.decode().strip().split('\n') + upload_task_dir_num = subprocess.run([f"ls -1 -d {dir}/task.* |wc -l", ], + shell=True, stdout=subprocess.PIPE).stdout.decode().strip('\n') + total_core_sec = float(0) + finished_task_file_num = len(model_devi_time_logs) + # assume model_devi lammps job running on GPUs , set paral_cores==1 + paral_cores = 1 + for log in model_devi_time_logs: + # log example: + # .//iter.000002/01.model_devi//task.018.000075/log.lammps:Total wall time: 0:00:39 + # print(log) + file_path, text1, hour, min, sec = log.split(':') # pylint: disable=unused-variable + abs_file_path = os.path.abspath(file_path) + wall_time_sec = 3600*int(hour) + 60*int(min) + 1*int(sec) + total_core_sec += wall_time_sec * paral_cores + total_core_hour = total_core_sec / 3600 + + print(f"{stage}:{abs_dir}" + f":paral_cores:{paral_cores}" + f":upload_task_dir_num:{upload_task_dir_num}" + f":finished_task_file_num:{finished_task_file_num}" + f":total_core_hour:{total_core_hour:.3f}") + + fp_dirs = subprocess.run([f"ls -d -1 {target_folder}/iter.??????/02.fp/", ], + shell=True,stdout=subprocess.PIPE).stdout.decode().strip().split('\n') + for dir in fp_dirs: + abs_dir = os.path.abspath(dir) + stage = os.path.basename(os.path.dirname(dir)) + fp_time_logs = subprocess.run([f"grep -H --text 'CPU time' {dir}/task.*/OUTCAR", ], + shell=True,stdout=subprocess.PIPE).stdout.decode().strip().split('\n') + upload_task_dir_num = subprocess.run([f"ls -1 -d {dir}/task.* |wc -l", ], + shell=True, stdout=subprocess.PIPE).stdout.decode().strip('\n') + total_core_sec = float(0) + finished_task_file_num = len(fp_time_logs) + for log in fp_time_logs: + # log example: + # .//iter.000002/02.fp//task.018.000048/OUTCAR: Total CPU time used (sec): 288.395 + file_path, text1, sec = log.split(':') + abs_file_path = os.path.abspath(file_path) + wall_time_sec = float(sec) + paral_cores = subprocess.run([fr"head -n 1000 {abs_file_path} | grep 'running on' | sed -n -E -e 's|running on\s+([0-9]+)+\s.*|\1|p' ", ], + shell=True,stdout=subprocess.PIPE).stdout.decode().strip() + total_core_sec += wall_time_sec * int(paral_cores) + total_core_hour = total_core_sec /3600 + + print(f"{stage}:{abs_dir}" + f":paral_cores:{paral_cores}" + f":upload_task_dir_num:{upload_task_dir_num}" + f":finished_task_file_num:{finished_task_file_num}" + f":total_core_hour:{total_core_hour:.3f}") + +if __name__=='__main__': + stat_time(target_folder="./") \ No newline at end of file diff --git a/dpgen/tools/update_time.sh b/dpgen/tools/update_time.sh deleted file mode 100755 index 29a08e311..000000000 --- a/dpgen/tools/update_time.sh +++ /dev/null @@ -1,88 +0,0 @@ -#!/bin/bash - -model_devi_paral_cores=1 - -if [[ -n $1 ]] -then - target_dir=$1 -else - target_dir="./" -fi - -if [[ -a time.log ]] -then - rm time.log -fi -for train_dir in `ls -d -1 $target_dir/iter.??????/00.train/`;do -sec=0 -tothour=0 -upload_task_dir_num=0 -recycle_task_file_num=0 -# echo $train_dir -upload_task_dir_num=$(ls -1 -d $train_dir/??? |wc -l) -if [[ -a train_time.log ]] -then - rm train_time.log -fi -grep -H --text 'wall time' $train_dir/???/train.log > train_time.log -recycle_task_file_num=$(wc -l < train_time.log) - while read line; do -mysec=$(echo "$line" |cut -d: -f4 |sed 's/s\| //g') -sec=$(echo "$mysec + $sec" | bc) - done < train_time.log -# echo $hour:$min:$sec -tothour=$(echo "scale=3; $sec/3600"|bc) -echo "00.train:$(realpath $train_dir):paral_cores:GPUV100:upload_task_dir_num:$upload_task_dir_num:recycle_task_file_num:$recycle_task_file_num:total core hour:$tothour" | tee -a time.log -done - -for model_devi_dir in `ls -d -1 $target_dir/iter.??????/01.model_devi/`;do -sec=0 -min=0 -hour=0 -tothour=0 -upload_task_dir_num=0 -recycle_task_file_num=0 -# echo $model_devi_dir -upload_task_dir_num=$(ls -1 -d $model_devi_dir/task.* |wc -l) -if [[ -a model_devi_time.log ]] -then - rm model_devi_time.log -fi -grep -H --text 'wall' $model_devi_dir/task.*/log.lammps > model_devi_time.log -recycle_task_file_num=$(wc -l < model_devi_time.log) - while read line; do -mysec=$(echo "$line" |cut -d: -f5) -sec=$(echo "$mysec + $sec" | bc) -mymin=$(echo "$line" |cut -d: -f4) -min=$(echo "$mymin + $min" | bc) -myhour=$(echo "$line" |cut -d: -f3) -hour=$(echo "$myhour + $hour" | bc) - done < model_devi_time.log -# echo $hour:$min:$sec -tothour=$(echo "scale=3; ($hour*3600+$min*60+$sec)*$model_devi_paral_cores/3600"|bc) -echo "01.model_devi:$(realpath $model_devi_dir):paral_cores:$model_devi_paral_cores:upload_task_dir_num:$upload_task_dir_num:recycle_task_file_num:$recycle_task_file_num:total core hour:$tothour" | tee -a time.log -done - -for fp_dir in `ls -d -1 $target_dir/iter.??????/02.fp/`;do -core_sec=0 -tothour=0 -upload_task_dir_num=0 -recycle_task_file_num=0 -# echo $fp_dir -upload_task_dir_num=$(ls -1 -d $fp_dir/task.* |wc -l) -if [[ -a fp_time.log ]] -then - rm fp_time.log -fi -grep -H --text 'CPU time' $fp_dir/task.*/OUTCAR > fp_time.log -recycle_task_file_num=$(wc -l < fp_time.log) - while read line;do -mysec=$(echo "$line" |cut -d: -f3 |sed 's| ||g') -file_name=$(echo "$line" | cut -d: -f1) -fp_paral_cores=$(grep 'total cores' $file_name |grep -o '[0-9]*') -core_sec=$(echo "$mysec * $fp_paral_cores + $core_sec" | bc) - done < fp_time.log -tothour=$(echo "scale=3; $core_sec/3600"|bc) -echo "02.fp:$(realpath $fp_dir):paral_cores:$fp_paral_cores:upload_task_dir_num:$upload_task_dir_num:recycle_task_file_num:$recycle_task_file_num:total core hour:$tothour" | tee -a time.log -done -wc -l $target_dir/iter.??????/02.fp/*out> candi_fail_accu.log diff --git a/examples/.DS_Store b/examples/.DS_Store deleted file mode 100644 index c3ea4212b..000000000 Binary files a/examples/.DS_Store and /dev/null differ diff --git a/examples/init/reaction.json b/examples/init/reaction.json new file mode 100644 index 000000000..46e327a9f --- /dev/null +++ b/examples/init/reaction.json @@ -0,0 +1,19 @@ +{ + "type_map": [ + "H", + "O" + ], + "reaxff": { + "data": "data.hydrogen", + "ff": "ffield.reax.cho", + "control": "lmp_control", + "temp": 3000, + "tau_t": 100, + "dt": 0.1, + "nstep": 10000, + "dump_freq": 100 + }, + "cutoff": 3.5, + "dataset_size": 100, + "qmkeywords": "b3lyp/6-31g** force" +} \ No newline at end of file diff --git a/examples/machine/machine-aws.json b/examples/machine/DeePMD-kit-0.12/machine-aws.json similarity index 100% rename from examples/machine/machine-aws.json rename to examples/machine/DeePMD-kit-0.12/machine-aws.json diff --git a/examples/machine/machine-local.json b/examples/machine/DeePMD-kit-0.12/machine-local.json similarity index 60% rename from examples/machine/machine-local.json rename to examples/machine/DeePMD-kit-0.12/machine-local.json index 05a0f2811..057db2722 100644 --- a/examples/machine/machine-local.json +++ b/examples/machine/DeePMD-kit-0.12/machine-local.json @@ -1,19 +1,15 @@ { "_comment": "training on localhost ", - "deepmd_path": "/home/wanghan/local/deepmd/0.10.1/", + "_comment" : "This is for DeePMD-kit 0.12.4", + "deepmd_path": "/home/wanghan/local/deepmd/0.12.4/", "train_machine": { - "machine_type": "local", - "hostname" : "127.0.0.1", - "port" : 22, - "username": "wanghan", - "work_path" : "/home/wanghan/tmp/subs/", - "_comment" : "that's all" + "batch": "shell", + "work_path" : "/home/wanghan/tmp/subs/" }, "train_resources": { "envs": { "PYTHONPATH" : "/home/wanghan/local/tensorflow/1.8.py/lib/python3.6/site-packages/" - }, - "_comment": "that's all" + } }, @@ -21,25 +17,18 @@ "lmp_command": "/home/wanghan/local/bin/lmp_mpi_010", "model_devi_group_size": 5, "model_devi_machine": { - "machine_type": "local", - "hostname" : "127.0.0.1", - "port" : 22, - "username": "wanghan", - "work_path" : "/home/wanghan/tmp/subs/", - "_comment" : "that's all" + "batch": "shell", + "_comment" : "If lazy_local is true, calculations are done directly in current folders.", + "lazy_local" : true }, "model_devi_resources": { - "_comment": "that's all" }, "_comment": "fp on localhost ", "fp_command": "/home/wanghan/local/bin/vasp_std", "fp_group_size": 2, "fp_machine": { - "machine_type": "local", - "hostname" : "127.0.0.1", - "port" : 22, - "username": "wanghan", + "batch": "local", "work_path" : "/home/wanghan/tmp/subs/", "_comment" : "that's all" }, diff --git a/examples/machine/machine-lsf.json b/examples/machine/DeePMD-kit-0.12/machine-lsf.json similarity index 96% rename from examples/machine/machine-lsf.json rename to examples/machine/DeePMD-kit-0.12/machine-lsf.json index f10ed82fb..d8ebd61ed 100644 --- a/examples/machine/machine-lsf.json +++ b/examples/machine/DeePMD-kit-0.12/machine-lsf.json @@ -2,7 +2,7 @@ "train": [ { "machine": { - "machine_type": "lsf", + "batch": "lsf", "hostname": "localhost", "port": 22, "username": "ypliu", @@ -32,7 +32,7 @@ "model_devi": [ { "machine": { - "machine_type": "lsf", + "batch": "lsf", "hostname": "localhost", "port": 22, "username": "ypliu", @@ -64,7 +64,7 @@ "fp": [ { "machine": { - "machine_type": "lsf", + "batch": "lsf", "hostname": "localhost", "port": 22, "username": "ypliu", diff --git a/examples/machine/machine-slurm-vasp-multi.json b/examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-multi.json similarity index 95% rename from examples/machine/machine-slurm-vasp-multi.json rename to examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-multi.json index 17f5bad48..e24838077 100644 --- a/examples/machine/machine-slurm-vasp-multi.json +++ b/examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-multi.json @@ -2,7 +2,7 @@ "train": [ { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "localhost", "port": 22, "username": "1600017784", @@ -25,7 +25,7 @@ }, { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "localhost", "port": 22, "username": "1600017784", @@ -49,7 +49,7 @@ { "deepmd_path": "/data2/publicsoft/deepmd-kit/0.12.4-s/", "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "115.27.161.2", "port": 22, "username": "anguse", @@ -78,7 +78,7 @@ "model_devi": [ { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "localhost", "port": 22, "username": "1600017784", @@ -102,7 +102,7 @@ }, { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "localhost", "port": 22, "username": "1600017784", @@ -126,7 +126,7 @@ }, { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "115.27.161.2", "port": 22, "username": "anguse", @@ -156,7 +156,7 @@ "fp": [ { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "localhost", "port": 22, "username": "1600017784", @@ -183,7 +183,7 @@ }, { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "162.105.133.134", "port": 22, "username": "1600017784", @@ -211,7 +211,7 @@ }, { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "162.105.133.134", "port": 22, "username": "1600017784", diff --git a/examples/machine/machine-slurm-vasp-multi.yaml b/examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-multi.yaml similarity index 95% rename from examples/machine/machine-slurm-vasp-multi.yaml rename to examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-multi.yaml index c90df9a09..5bd30d186 100644 --- a/examples/machine/machine-slurm-vasp-multi.yaml +++ b/examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-multi.yaml @@ -1,7 +1,7 @@ --- train: - machine: - machine_type: slurm + batch: slurm hostname: localhost port: 22 username: '1600017784' @@ -19,7 +19,7 @@ train: qos: bigdata deepmd_path: "/gpfs/share/software/deepmd-kit/0.12.4/gpu/gcc/4.9.0/tf1120-lowprec" - machine: - machine_type: slurm + batch: slurm hostname: localhost port: 22 username: '1600017784' @@ -38,7 +38,7 @@ train: deepmd_path: "/gpfs/share/software/deepmd-kit/0.12.4/gpu/gcc/4.9.0/tf1120-lowprec" - deepmd_path: "/data2/publicsoft/deepmd-kit/0.12.4-s/" machine: - machine_type: slurm + batch: slurm hostname: 115.27.161.2 port: 22 username: anguse @@ -60,7 +60,7 @@ train: _comment: that's all model_devi: - machine: - machine_type: slurm + batch: slurm hostname: localhost port: 22 username: '1600017784' @@ -79,7 +79,7 @@ model_devi: command: lmp_serial group_size: 10 - machine: - machine_type: slurm + batch: slurm hostname: localhost port: 22 username: '1600017784' @@ -98,7 +98,7 @@ model_devi: command: lmp_serial group_size: 10 - machine: - machine_type: slurm + batch: slurm hostname: 115.27.161.2 port: 22 username: anguse @@ -121,7 +121,7 @@ model_devi: group_size: 20 fp: - machine: - machine_type: slurm + batch: slurm hostname: localhost port: 22 username: '1600017784' @@ -143,7 +143,7 @@ fp: command: vasp_std group_size: 5 - machine: - machine_type: slurm + batch: slurm hostname: 162.105.133.134 port: 22 username: '1600017784' @@ -165,7 +165,7 @@ fp: command: mpirun -n 16 vasp_std group_size: 5 - machine: - machine_type: slurm + batch: slurm hostname: 162.105.133.134 port: 22 username: '1600017784' diff --git a/examples/machine/machine-slurm-vasp-single.json b/examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-single.json similarity index 95% rename from examples/machine/machine-slurm-vasp-single.json rename to examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-single.json index 199736d66..2dbdafd5e 100644 --- a/examples/machine/machine-slurm-vasp-single.json +++ b/examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-single.json @@ -2,7 +2,7 @@ "train": [ { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "localhost", "port": 22, "username": "1600017784", @@ -27,7 +27,7 @@ "model_devi": [ { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "localhost", "port": 22, "username": "1600017784", @@ -53,7 +53,7 @@ "fp": [ { "machine": { - "machine_type": "slurm", + "batch": "slurm", "hostname": "localhost", "port": 22, "username": "1600017784", diff --git a/examples/machine/machine-slurm-vasp-single.yaml b/examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-single.yaml similarity index 95% rename from examples/machine/machine-slurm-vasp-single.yaml rename to examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-single.yaml index 4162f1055..3b52e52ce 100644 --- a/examples/machine/machine-slurm-vasp-single.yaml +++ b/examples/machine/DeePMD-kit-0.12/machine-slurm-vasp-single.yaml @@ -1,7 +1,7 @@ --- train: - machine: - machine_type: slurm + batch: slurm hostname: localhost port: 22 username: '1600017784' @@ -20,7 +20,7 @@ train: deepmd_path: "/gpfs/share/software/deepmd-kit/0.12.4/gpu/gcc/4.9.0/tf1120-lowprec" model_devi: - machine: - machine_type: slurm + batch: slurm hostname: localhost port: 22 username: '1600017784' @@ -40,7 +40,7 @@ model_devi: group_size: 10 fp: - machine: - machine_type: slurm + batch: slurm hostname: localhost port: 22 username: '1600017784' diff --git a/examples/machine/DeePMD-kit-1.0/machine-local.json b/examples/machine/DeePMD-kit-1.0/machine-local.json new file mode 100644 index 000000000..d418a783e --- /dev/null +++ b/examples/machine/DeePMD-kit-1.0/machine-local.json @@ -0,0 +1,42 @@ +{ + "_comment": "training on localhost ", + "_comment" : "This is for DeePMD-kit 1.*", + "python_path": "/home/wanghan/local/deepmd/1.*/python", + "train_machine": { + "batch": "shell", + "work_path" : "/home/wanghan/tmp/subs/" + }, + "train_resources": { + "envs": { + } + }, + + + "_comment": "model_devi on localhost ", + "lmp_command": "/home/wanghan/local/bin/lmp_mpi_010", + "model_devi_group_size": 5, + "model_devi_machine": { + "batch": "shell", + "_comment" : "If lazy_local is true, calculations are done directly in current folders.", + "lazy_local" : true + }, + "model_devi_resources": { + }, + + "_comment": "fp on localhost ", + "fp_command": "/home/wanghan/local/bin/vasp_std", + "fp_group_size": 2, + "fp_machine": { + "batch": "local", + "work_path" : "/home/wanghan/tmp/subs/", + "_comment" : "that's all" + }, + "fp_resources": { + "module_list": ["mpi"], + "task_per_node":4, + "with_mpi": true, + "_comment": "that's all" + }, + + "_comment": " that's all " +} diff --git a/examples/machine/DeePMD-kit-1.0/machine-pbs-gaussian.json b/examples/machine/DeePMD-kit-1.0/machine-pbs-gaussian.json new file mode 100644 index 000000000..c08363148 --- /dev/null +++ b/examples/machine/DeePMD-kit-1.0/machine-pbs-gaussian.json @@ -0,0 +1,79 @@ +{ + "_comment": "training on localhost ", + "python_path": "/gpfs/home/tzhu/anaconda3/envs/python3.6/bin/python", + "train_machine": { + "machine_type": "lsf", + "hostname" : "59.78.197.77", + "port" : 22, + "username": "tzhu", + "work_path" : "/gpfs/home/tzhu/jzzeng/dpgen_workdir", + "_comment" : "that's all" + }, + "train_resources": { + "source_list": [ "activate deepmd" ], + "envs": { + "KMP_BLOCKTIME": 0, + "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0" + }, + "numb_gpu": 1, + "numb_node": 1, + "node_cpu": 0, + "partition": "newgpu", + "job_name": "dpgen_jzzeng", + "with_mpi": false, + "time_limit": false, + "_comment": "that's all" + }, + + + "_comment": "model_devi on localhost ", + "lmp_command": "/gpfs/home/tzhu/lammps-stable_5Jun2019/src/lmp_intel_cpu_intelmpi -pk intel 0 omp 2", + "model_devi_group_size": 1, + "model_devi_machine": { + "machine_type": "lsf", + "hostname" : "59.78.197.77", + "port" : 22, + "username": "tzhu", + "work_path" : "/gpfs/home/tzhu/jzzeng/dpgen_workdir", + "_comment" : "that's all" + }, + "model_devi_resources": { + "envs": { + "KMP_BLOCKTIME": 0 + }, + "source_list": [ "activate deepmd" ], + "numb_gpu": 1, + "numb_node": 1, + "node_cpu": 0, + "time_limit": false, + "partition": "newgpu", + "job_name": "dpgen_jzzeng", + "with_mpi": true, + "task_per_node": 1, + "_comment": "that's all" + }, + + "_comment": "fp on lsf //localhost ", + "fp_command": "/public/home/tzhu/g16/g16 < input", + "fp_group_size": 1, + "fp_machine": { + "machine_type": "pbs", + "hostname" : "59.78.189.132", + "port" : 2323, + "username": "tzhu", + "work_path" : "/public/home/tzhu/jzzeng/dpgen_workdir", + "_comment" : "that's all" + }, + "fp_resources": { + "node_cpu":28, + "numb_node": 1, + "job_name": "dpgen_jzzeng", + "task_per_node": 28, + "with_mpi": false, + "time_limit": "10:00:00", + "allow_failure": true, + "partition": "small", + "_comment": "that's all" + }, + "_comment": " that's all " +} diff --git a/examples/machine/machine-hnu.json b/examples/machine/bk/machine-hnu.json similarity index 100% rename from examples/machine/machine-hnu.json rename to examples/machine/bk/machine-hnu.json diff --git a/examples/machine/machine-tiger-pwscf-della.json b/examples/machine/bk/machine-tiger-pwscf-della.json similarity index 100% rename from examples/machine/machine-tiger-pwscf-della.json rename to examples/machine/bk/machine-tiger-pwscf-della.json diff --git a/examples/machine/machine-tiger-vasp-della.json b/examples/machine/bk/machine-tiger-vasp-della.json similarity index 100% rename from examples/machine/machine-tiger-vasp-della.json rename to examples/machine/bk/machine-tiger-vasp-della.json diff --git a/examples/machine/machine-tiger.json b/examples/machine/bk/machine-tiger.json similarity index 100% rename from examples/machine/machine-tiger.json rename to examples/machine/bk/machine-tiger.json diff --git a/examples/machine/machine-ucloud.json b/examples/machine/bk/machine-ucloud.json similarity index 100% rename from examples/machine/machine-ucloud.json rename to examples/machine/bk/machine-ucloud.json diff --git a/examples/run/dp-lammps-vasp/Al/param_al_all_gpu-deepmd-kit-1.1.0.json b/examples/run/dp-lammps-vasp/Al/param_al_all_gpu-deepmd-kit-1.1.0.json new file mode 100644 index 000000000..cfe0b855d --- /dev/null +++ b/examples/run/dp-lammps-vasp/Al/param_al_all_gpu-deepmd-kit-1.1.0.json @@ -0,0 +1,379 @@ +{ + "type_map": ["Al"], + "mass_map": [27], + + "init_data_prefix": "/data1/yfb222333/2_dpgen_gpu_multi/init/", + + "init_data_sys": [ + "al.fcc.02x02x02/02.md/sys-0032/deepmd", + "al.hcp.02x02x02/02.md/sys-0016/deepmd", + "al.bcc.02x02x02/02.md/sys-0016/deepmd" + ], + "init_batch_size": [ + 1, + 2, + 2 + ], + "sys_configs": [ + + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00000[0-4]/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00000[5-9]/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00001*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00002*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00003*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00004*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00005*/POSCAR", + "/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00006*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00007*/POSCAR", + "/data1/yfb222333/2_dpgen_gpu_multi/init/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.000/00008*/POSCAR"], + + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00000[0-4]/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00000[5-9]/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00001*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00002*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00003*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00004*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00005*/POSCAR", + "/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00006*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00007*/POSCAR", + "/data1/yfb222333/2_dpgen_gpu_multi/init/al.hcp.02x02x02/01.scale_pert/sys-0016/scale-1.000/00008*/POSCAR"], + + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00000[0-4]/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00000[5-9]/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00001*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00002*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00003*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00004*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00005*/POSCAR", + "/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00006*/POSCAR"], + ["/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00007*/POSCAR", + "/data1/yfb222333/2_dpgen_gpu_multi/init/al.bcc.02x02x02/01.scale_pert/sys-0016/scale-1.000/00008*/POSCAR"] + ], + "_comment": " 00.train ", + "numb_models": 4, + "default_training_param" : { + "model":{ + "_comment": " model parameters", + "type_map":["Al"], + "descriptor":{ + "type": "se_a", + "sel": [300], + "rcut_smth": 2.00, + "rcut": 8.00, + "neuron": [240, 240, 240], + "resnet_dt": true, + "axis_neuron": 12, + "seed": 1 +}, + "fitting_net":{ + "neuron": [25, 50, 100], + "resnet_dt": false, + "sedd": 1 +}}, + "learning_rate":{ + "type": "exp", + "start_lr": 0.001, + "decay_steps": 2000, + "decay_rate": 0.95 +}, + "loss":{ + "start_pref_e": 0.02, + "limit_pref_e": 2, + "start_pref_f": 1000, + "limit_pref_f": 1, + "start_pref_v": 0.0, + "limit_pref_v": 0.0 +}, + "training":{ + "coord_norm": true, + "type_fitting_net": false, + "_comment": " traing controls", + "systems": [], + "set_prefix": "set", + "stop_batch": 20000, + "batch_size": 1, + "seed": 0, + "_comment": " display and restart", + "_comment": " frequencies counted in batch", + "disp_file": "lcurve.out", + "disp_freq": 2000, + "numb_test": 4, + "save_freq": 2000, + "save_ckpt": "model.ckpt", + "load_ckpt": "model.ckpt", + "disp_training": true, + "time_training": true, + "profiling": false, + "profiling_file": "timeline.json", + "_comment": "that's all"} + }, + + "_comment": " 01.model_devi ", + "_comment": "model_devi_skip: the first x of the recorded frames", + "model_devi_dt": 0.002, + "model_devi_skip": 0, + "model_devi_f_trust_lo": 0.05, + "model_devi_f_trust_hi": 0.20, + "model_devi_e_trust_lo": 1e10, + "model_devi_e_trust_hi": 1e10, + "model_devi_clean_traj": false, + "model_devi_jobs": +[ + { + "_idx": 0, + "ensemble": "npt", + "nsteps": 1000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 0, + 8, + 16 + ], + "temps": [ + 50, + 132.0, + 198.0, + 264.0 + ], + "trj_freq": 10 + }, + { + "_idx": 1, + "ensemble": "npt", + "nsteps": 1000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 1, + 9, + 17 + ], + "temps": [ + 50, + 132.0, + 198.0, + 264.0 + ], + "trj_freq": 10 + }, + { + "_idx": 2, + "ensemble": "npt", + "nsteps": 3000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 2, + 10, + 18 + ], + "temps": [ + 50, + 132.0, + 198.0, + 264.0 + ], + "trj_freq": 10 + }, + { + "_idx": 3, + "ensemble": "npt", + "nsteps": 3000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 3, + 11, + 19 + ], + "temps": [ + 50, + 132.0, + 198.0, + 264.0 + ], + "trj_freq": 10 + }, + { + "_idx": 4, + "ensemble": "npt", + "nsteps": 3000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 4, + 12, + 20 + ], + "temps": [ + 50, + 132.0, + 198.0, + 264.0 + ], + "trj_freq": 10 + }, + { + "_idx": 5, + "ensemble": "npt", + "nsteps": 3000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 5, + 13, + 21 + ], + "temps": [ + 50, + 132.0, + 198.0, + 264.0 + ], + "trj_freq": 10 + }, + { + "_idx": 6, + "ensemble": "npt", + "nsteps": 3000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 6, + 14, + 22 + ], + "temps": [ + 50, + 132.0, + 198.0, + 264.0 + ], + "trj_freq": 10 + }, + { + "_idx": 7, + "ensemble": "npt", + "nsteps": 3000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 7, + 15, + 23 + ], + "temps": [ + 50, + 132.0, + 198.0, + 264.0 + ], + "trj_freq": 10 + }, + { + "_idx": 8, + "ensemble": "npt", + "nsteps": 1000, + "press": [ + 1.0, + 10.0, + 100.0, + 1000.0, + 5000.0, + 10000.0, + 20000.0, + 50000.0 + ], + "sys_idx": [ + 0, + 8, + 16 + ], + "temps": [ + 330.0, + 396.0, + 462.0, + 528.0, + 594.0 + ], + "trj_freq": 10 + } +], + + + "_comment": " 02.fp ", + "fp_style": "vasp", + "shuffle_poscar": false, + "fp_task_max": 300, + "fp_task_min": 5, + "fp_pp_path": "/data1/yfb222333/2_dpgen_gpu_multi/POTCAR-Al", + "fp_pp_files": ["POTCAR"], + "fp_incar": "/data1/yfb222333/2_dpgen_gpu_multi/INCAR_metal_scf_gpu", + "_comment": " that's all " +} + diff --git a/examples/run/dp-lammps-vasp/CH4/param_CH4_deepmd-kit-1.1.0.json b/examples/run/dp-lammps-vasp/CH4/param_CH4_deepmd-kit-1.1.0.json new file mode 100644 index 000000000..b2bba7be7 --- /dev/null +++ b/examples/run/dp-lammps-vasp/CH4/param_CH4_deepmd-kit-1.1.0.json @@ -0,0 +1,138 @@ +{ + "type_map": [ + "H", + "C" + ], + "mass_map": [ + 1, + 12 + ], + "init_data_prefix": "/data1/yfb222333/2_dpgen_gpu_multi", + "init_data_sys": [ + "CH4.POSCAR.01x01x01/02.md/sys-0004-0001/deepmd" + ], + "sys_configs_prefix": "/data1/yfb222333/2_dpgen_gpu_multi", + "sys_configs": [ + [ + "CH4.POSCAR.01x01x01/01.scale_pert/sys-0004-0001/scale*/00000*/POSCAR" + ], + [ + "CH4.POSCAR.01x01x01/01.scale_pert/sys-0004-0001/scale*/00001*/POSCAR" + ] + ], + "_comment": " that's all ", + "numb_models": 4, + "default_training_param": { + "model": { + "type_map": [ + "H", + "C" + ], + "descriptor": { + "type": "se_a", + "sel": [ + 16, + 4 + ], + "rcut_smth": 0.5, + "rcut": 5, + "neuron": [ + 120, + 120, + 120 + ], + "resnet_dt": true, + "axis_neuron": 12, + "seed": 1 + }, + "fitting_net": { + "neuron": [ + 25, + 50, + 100 + ], + "resnet_dt": false, + "seed": 1 + } + }, + "learning_rate": { + "type": "exp", + "start_lr": 0.001, + "decay_steps": 100, + "decay_rate": 0.95 + }, + "loss": { + "start_pref_e": 0.02, + "limit_pref_e": 2, + "start_pref_f": 1000, + "limit_pref_f": 1, + "start_pref_v": 0.0, + "limit_pref_v": 0.0 + }, + "training": { + "set_prefix": "set", + "stop_batch": 2000, + "batch_size": 1, + "disp_file": "lcurve.out", + "disp_freq": 1000, + "numb_test": 4, + "save_freq": 1000, + "save_ckpt": "model.ckpt", + "load_ckpt": "model.ckpt", + "disp_training": true, + "time_training": true, + "profiling": false, + "profiling_file": "timeline.json", + "_comment": "that's all" + } + }, + "model_devi_dt": 0.002, + "model_devi_skip": 0, + "model_devi_f_trust_lo": 0.05, + "model_devi_f_trust_hi": 0.15, + "model_devi_e_trust_lo": 10000000000.0, + "model_devi_e_trust_hi": 10000000000.0, + "model_devi_clean_traj": true, + "model_devi_jobs": [ + { + "sys_idx": [ + 0 + ], + "temps": [ + 100 + ], + "press": [ + 1.0 + ], + "trj_freq": 10, + "nsteps": 300, + "ensemble": "nvt", + "_idx": "00" + }, + { + "sys_idx": [ + 1 + ], + "temps": [ + 100 + ], + "press": [ + 1.0 + ], + "trj_freq": 10, + "nsteps": 3000, + "ensemble": "nvt", + "_idx": "01" + } + ], + "fp_style": "vasp", + "shuffle_poscar": false, + "fp_task_max": 20, + "fp_task_min": 5, + "fp_pp_path": "/data1/yfb222333/2_dpgen_gpu_multi", + "fp_pp_files": [ + "H/POTCAR", + "C/POTCAR" + ], + "fp_incar": "/data1/yfb222333/2_dpgen_gpu_multi/INCAR_methane" +} diff --git a/examples/simplify/qm7.json b/examples/simplify/qm7.json new file mode 100644 index 000000000..648c589e7 --- /dev/null +++ b/examples/simplify/qm7.json @@ -0,0 +1,113 @@ +{ + "type_map": [ + "C", + "H", + "N", + "O", + "S" + ], + "mass_map": [ + 12.011, + 1.008, + 14.007, + 15.999, + 32.065 + ], + "pick_data": "/scratch/jz748/simplify/qm7", + "init_data_prefix": "", + "init_data_sys": [], + "sys_batch_size": [ + "auto" + ], + "numb_models": 4, + "train_param": "input.json", + "default_training_param": { + "model": { + "type_map": [ + "C", + "H", + "N", + "O", + "S" + ], + "descriptor": { + "type": "se_a", + "sel": [ + 7, + 16, + 3, + 3, + 1 + ], + "rcut_smth": 1.00, + "rcut": 6.00, + "neuron": [ + 25, + 50, + 100 + ], + "resnet_dt": false, + "axis_neuron": 12 + }, + "fitting_net": { + "neuron": [ + 240, + 240, + 240 + ], + "resnet_dt": true + } + }, + "learning_rate": { + "type": "exp", + "start_lr": 0.001, + "decay_steps": 10, + "decay_rate": 0.99 + }, + "loss": { + "start_pref_e": 0.02, + "limit_pref_e": 1, + "start_pref_f": 1000, + "limit_pref_f": 1, + "start_pref_v": 0, + "limit_pref_v": 0, + "start_pref_pf": 0, + "limit_pref_pf": 0 + }, + "training": { + "set_prefix": "set", + "stop_batch": 10000, + "disp_file": "lcurve.out", + "disp_freq": 1000, + "numb_test": 1, + "save_freq": 1000, + "save_ckpt": "model.ckpt", + "load_ckpt": "model.ckpt", + "disp_training": true, + "time_training": true, + "profiling": false, + "profiling_file": "timeline.json" + }, + "_comment": "that's all" + }, + "use_clusters": true, + "fp_style": "gaussian", + "shuffle_poscar": false, + "fp_task_max": 1000, + "fp_task_min": 10, + "fp_pp_path": "/home/jzzeng/", + "fp_pp_files": [], + "fp_params": { + "keywords": "mn15/6-31g** force nosymm scf(maxcyc=512)", + "nproc": 28, + "multiplicity": 1, + "_comment": " that's all " + }, + "init_pick_number":100, + "iter_pick_number":100, + "e_trust_lo":1e10, + "e_trust_hi":1e10, + "f_trust_lo":0.25, + "f_trust_hi":0.45, + "_comment": " that's all " +} diff --git a/examples/test/deepmd_param.json b/examples/test/deepmd_param.json index 294455623..98e79afdb 100644 --- a/examples/test/deepmd_param.json +++ b/examples/test/deepmd_param.json @@ -1,60 +1,64 @@ { "_comment": "models", - "potcar_map" : { - "Al" : "/somewhere/example/POTCAR" + "potcar_map": { + "Al": "/somewhere/example/POTCAR" }, - "conf_dir":"confs/Al/std-fcc", - "key_id":"key id of Material project", - "task_type":"deepmd", - "task":"all", - - "vasp_params": { - "ecut": 650, - "ediff": 1e-6, - "kspacing": 0.1, - "kgamma": false, - "npar": 1, - "kpar": 1, - "_comment": " that's all " + "conf_dir": "confs/Al/std-fcc", + "key_id": "key id of Material project", + "task_type": "deepmd", + "task": "all", + + "vasp_params": { + "ecut": 650, + "ediff": 1e-6, + "kspacing": 0.1, + "kgamma": false, + "npar": 1, + "kpar": 1, + "_comment": " that's all " }, - "lammps_params": { - "model_dir":"somewhere/example/Al_model", - "type_map":["Al"], - "model_name":false, - "model_param_type":false + "lammps_params": { + "model_dir": "somewhere/example/Al_model", + "type_map": [ + "Al" + ], + "model_name": false, + "model_param_type": false }, - - "_comment":"00.equi", - "store_stable":true, - + "_comment": "00.equi", + "alloy_shift": false, "_comment": "01.eos", - "vol_start": 12, - "vol_end": 22, - "vol_step": 0.5, - + "vol_start": 12, + "vol_end": 22, + "vol_step": 0.5, "_comment": "02.elastic", - "norm_deform": 2e-2, - "shear_deform": 5e-2, - - "_comment":"03.vacancy", - "supercell":[3,3,3], - - "_comment":"04.interstitial", - "insert_ele":["Al"], - "reprod-opt":false, - + "norm_deform": 2e-2, + "shear_deform": 5e-2, + "_comment": "03.vacancy", + "supercell": [ + 3, + 3, + 3 + ], + "_comment": "04.interstitial", + "insert_ele": [ + "Al" + ], + "reprod-opt": false, "_comment": "05.surface", - "min_slab_size": 10, - "min_vacuum_size": 11, + "min_slab_size": 10, + "min_vacuum_size": 11, "_comment": "pert xz to work around vasp bug...", - "pert_xz": 0.01, + "pert_xz": 0.01, "max_miller": 2, - "static-opt":false, - "relax_box":false, - - "_comment":"06.phonon", - "supercell_matrix":[2,2,2], - "band":"0 1 0 0.5 1 0.5 0.375 0.75 0.375 0 0 0 0.5 0.5 0.5", - - "_comment": "that's all" + "static-opt": false, + "relax_box": false, + "_comment": "06.phonon", + "supercell_matrix": [ + 2, + 2, + 2 + ], + "band": "0 1 0 0.5 1 0.5 0.375 0.75 0.375 0 0 0 0.5 0.5 0.5", + "_comment": "that's all" } diff --git a/examples/test/meam_param.json b/examples/test/meam_param.json index 5beff5324..f89b432e2 100644 --- a/examples/test/meam_param.json +++ b/examples/test/meam_param.json @@ -1,60 +1,77 @@ { "_comment": "models", - "potcar_map" : { - "Al" : "/somewhere/example/POTCAR" + "potcar_map": { + "Al": "/somewhere/example/POTCAR" }, - "conf_dir":"confs/Al/std-fcc", - "key_id":"key id of Material project", - "task_type":"meam", - "task":"all", - - "vasp_params": { - "ecut": 650, - "ediff": 1e-6, - "kspacing": 0.1, - "kgamma": false, - "npar": 1, - "kpar": 1, - "_comment": " that's all " + "conf_dir": "confs/Al/std-fcc", + "key_id": "key id of Material project", + "task_type": "meam", + "task": "all", + + "vasp_params": { + "ecut": 650, + "ediff": 1e-6, + "kspacing": 0.1, + "kgamma": false, + "npar": 1, + "kpar": 1, + "_comment": " that's all " }, - "lammps_params": { - "model_dir":"somewhere/example/meam", - "type_map":["Al","Si","Mg","Cu","Fe"], - "model_name":["meam.AlSiMgCuFe","meam.library"], - "model_param_type":["AlS", "SiS", "MgS", "CuS", "FeS"] + "lammps_params": { + "model_dir": "somewhere/example/meam", + "type_map": [ + "Al", + "Si", + "Mg", + "Cu", + "Fe" + ], + "model_name": [ + "meam.AlSiMgCuFe", + "meam.library" + ], + "model_param_type": [ + "AlS", + "SiS", + "MgS", + "CuS", + "FeS" + ] }, - - "_comment":"00.equi", - "store_stable":true, - + "_comment": "00.equi", + "alloy_shift": false, "_comment": "01.eos", - "vol_start": 12, - "vol_end": 22, - "vol_step": 0.5, - + "vol_start": 12, + "vol_end": 22, + "vol_step": 0.5, "_comment": "02.elastic", - "norm_deform": 2e-2, - "shear_deform": 5e-2, - - "_comment":"03.vacancy", - "supercell":[3,3,3], - - "_comment":"04.interstitial", - "insert_ele":["Al"], - "reprod-opt":false, - + "norm_deform": 2e-2, + "shear_deform": 5e-2, + "_comment": "03.vacancy", + "supercell": [ + 3, + 3, + 3 + ], + "_comment": "04.interstitial", + "insert_ele": [ + "Al" + ], + "reprod-opt": false, "_comment": "05.surface", - "min_slab_size": 10, - "min_vacuum_size": 11, + "min_slab_size": 10, + "min_vacuum_size": 11, "_comment": "pert xz to work around vasp bug...", - "pert_xz": 0.01, + "pert_xz": 0.01, "max_miller": 2, - "static-opt":false, - "relax_box":false, - - "_comment":"06.phonon", - "supercell_matrix":[2,2,2], - "band":"0 1 0 0.5 1 0.5 0.375 0.75 0.375 0 0 0 0.5 0.5 0.5", - - "_comment": "that's all" + "static-opt": false, + "relax_box": false, + "_comment": "06.phonon", + "supercell_matrix": [ + 2, + 2, + 2 + ], + "band": "0 1 0 0.5 1 0.5 0.375 0.75 0.375 0 0 0 0.5 0.5 0.5", + "_comment": "that's all" } diff --git a/examples/test/vasp_param.json b/examples/test/vasp_param.json index 2ec2ed9df..4fab168ab 100644 --- a/examples/test/vasp_param.json +++ b/examples/test/vasp_param.json @@ -1,54 +1,56 @@ { - "_comment": "models", - "potcar_map" : { - "Al" : "/somewhere/example/POTCAR" - }, - "conf_dir":"confs/Al/std-fcc", - "key_id":"key id of Material project", - "task_type":"vasp", - "task":"all", - - "vasp_params": { - "ecut": 650, - "ediff": 1e-6, - "kspacing": 0.1, - "kgamma": false, - "npar": 1, - "kpar": 1, - "_comment": " that's all " - }, - - "_comment":"00.equi", - "store_stable":true, - - "_comment": "01.eos", - "vol_start": 12, - "vol_end": 22, - "vol_step": 0.5, - - "_comment": "02.elastic", - "norm_deform": 2e-2, - "shear_deform": 5e-2, - - "_comment":"03.vacancy", - "supercell":[3,3,3], - - "_comment":"04.interstitial", - "insert_ele":["Al"], - "reprod-opt":false, - - "_comment": "05.surface", - "min_slab_size": 10, - "min_vacuum_size": 11, - "_comment": "pert xz to work around vasp bug...", - "pert_xz": 0.01, - "max_miller": 2, - "static-opt":false, - "relax_box":false, - - "_comment":"06.phonon", - "supercell_matrix":[2,2,2], - "band":"0 1 0 0.5 1 0.5 0.375 0.75 0.375 0 0 0 0.5 0.5 0.5", - - "_comment": "that's all" + "_comment": "models", + "potcar_map": { + "Al": "/somewhere/example/POTCAR" + }, + "conf_dir": "confs/Al/std-fcc", + "key_id": "key id of Material project", + "task_type": "vasp", + "task": "all", + + "vasp_params": { + "ecut": 650, + "ediff": 1e-6, + "kspacing": 0.1, + "kgamma": false, + "npar": 1, + "kpar": 1, + "_comment": " that's all " + }, + "_comment": "00.equi", + "alloy_shift": false, + "_comment": "01.eos", + "vol_start": 12, + "vol_end": 22, + "vol_step": 0.5, + "_comment": "02.elastic", + "norm_deform": 2e-2, + "shear_deform": 5e-2, + "_comment": "03.vacancy", + "supercell": [ + 3, + 3, + 3 + ], + "_comment": "04.interstitial", + "insert_ele": [ + "Al" + ], + "reprod-opt": false, + "_comment": "05.surface", + "min_slab_size": 10, + "min_vacuum_size": 11, + "_comment": "pert xz to work around vasp bug...", + "pert_xz": 0.01, + "max_miller": 2, + "static-opt": false, + "relax_box": false, + "_comment": "06.phonon", + "supercell_matrix": [ + 2, + 2, + 2 + ], + "band": "0 1 0 0.5 1 0.5 0.375 0.75 0.375 0 0 0 0.5 0.5 0.5", + "_comment": "that's all" } diff --git a/examples/test/vasp_param_from_incar.json b/examples/test/vasp_param_from_incar.json new file mode 100644 index 000000000..d89230d65 --- /dev/null +++ b/examples/test/vasp_param_from_incar.json @@ -0,0 +1,50 @@ +{ + "_comment": "models", + "potcar_map": { + "Al": "/somewhere/example/POTCAR" + }, + "conf_dir": "confs/Al/std-fcc", + "key_id": "key id of Material project", + "task_type": "vasp", + "task": "all", + + "relax_incar": "somewhere/relax_incar", + "scf_incar": "somewhere/scf_incar", + + "_comment": "00.equi", + "alloy_shift": false, + "_comment": "01.eos", + "vol_start": 12, + "vol_end": 22, + "vol_step": 0.5, + "_comment": "02.elastic", + "norm_deform": 2e-2, + "shear_deform": 5e-2, + "_comment": "03.vacancy", + "supercell": [ + 3, + 3, + 3 + ], + "_comment": "04.interstitial", + "insert_ele": [ + "Al" + ], + "reprod-opt": false, + "_comment": "05.surface", + "min_slab_size": 10, + "min_vacuum_size": 11, + "_comment": "pert xz to work around vasp bug...", + "pert_xz": 0.01, + "max_miller": 2, + "static-opt": false, + "relax_box": false, + "_comment": "06.phonon", + "supercell_matrix": [ + 2, + 2, + 2 + ], + "band": "0 1 0 0.5 1 0.5 0.375 0.75 0.375 0 0 0 0.5 0.5 0.5", + "_comment": "that's all" +} diff --git a/examples/test/vasp_poscar_param.json b/examples/test/vasp_poscar_param.json deleted file mode 100644 index eeca53d7d..000000000 --- a/examples/test/vasp_poscar_param.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "_comment": "models", - "potcar_map" : { - "Al" : "/somewhere/example/POTCAR" - }, - "conf_dir":"confs/Al/std-fcc", - "key_id":"key id of Material project", - "task_type":"vasp", - "task":"all", - - "relax_incar":"somewhere/relax_incar", - "scf_incar":"somewhere/scf_incar", - - "_comment":"00.equi", - "store_stable":true, - - "_comment": "01.eos", - "vol_start": 12, - "vol_end": 22, - "vol_step": 0.5, - - "_comment": "02.elastic", - "norm_deform": 2e-2, - "shear_deform": 5e-2, - - "_comment":"03.vacancy", - "supercell":[3,3,3], - - "_comment":"04.interstitial", - "insert_ele":["Al"], - "reprod-opt":false, - - "_comment": "05.surface", - "min_slab_size": 10, - "min_vacuum_size": 11, - "_comment": "pert xz to work around vasp bug...", - "pert_xz": 0.01, - "max_miller": 2, - "static-opt":false, - "relax_box":false, - - "_comment":"06.phonon", - "supercell_matrix":[2,2,2], - "band":"0 1 0 0.5 1 0.5 0.375 0.75 0.375 0 0 0 0.5 0.5 0.5", - - "_comment": "that's all" -} diff --git a/setup.py b/setup.py index c2451552d..71286b666 100755 --- a/setup.py +++ b/setup.py @@ -17,7 +17,7 @@ with open(path.join('dpgen', '_date.py'), 'w') as fp : fp.write('date = \'%s\'' % today) -install_requires=['numpy>=1.14.3', 'dpdata>=0.1.10', 'pymatgen>=2017.9.1', 'ase', 'monty>2.0.0', 'paramiko', 'custodian'] +install_requires=['numpy>=1.14.3', 'dpdata>=0.1.12', 'pymatgen>=2019.1.13', 'ase', 'monty>2.0.0', 'paramiko', 'custodian'] setuptools.setup( name=NAME, @@ -40,9 +40,10 @@ 'dpgen/remote', 'dpgen/dispatcher', 'dpgen/database', - 'dpgen/tools' + 'dpgen/tools', + 'dpgen/simplify', ], - data_files = [('dpgen/tools/', ['dpgen/tools/update_time.sh', ])], + # data_files = [('dpgen/tools/', ['dpgen/tools/update_time.sh', ])], # package_data={'example':['*.json']}, classifiers=[ "Programming Language :: Python :: 3.6", diff --git a/tests/data/POSCAR b/tests/data/POSCAR new file mode 100644 index 000000000..c8163bf00 --- /dev/null +++ b/tests/data/POSCAR @@ -0,0 +1,12 @@ +FCC : a = 4.100000 +4.0999999999999996 +1.0000000000000000 0.0000000000000000 0.0000000000000000 +0.0000000000000000 1.0000000000000000 0.0000000000000000 +0.0000000000000000 0.0000000000000000 1.0000000000000000 +Al +4 +Direct +0.0000000000000000 0.0000000000000000 0.0000000000000000 +0.5000000000000000 0.5000000000000000 0.0000000000000000 +0.5000000000000000 0.0000000000000000 0.5000000000000000 +0.0000000000000000 0.5000000000000000 0.5000000000000000 diff --git a/tests/data/context_surf_poscar.py b/tests/data/context_surf_poscar.py new file mode 100644 index 000000000..89bc492cc --- /dev/null +++ b/tests/data/context_surf_poscar.py @@ -0,0 +1,6 @@ +import sys,os +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))) +from dpgen.data.surf import * + +param_file = 'surf_poscar.json' + diff --git a/tests/data/surf_poscar.json b/tests/data/surf_poscar.json new file mode 100644 index 000000000..b5727ff83 --- /dev/null +++ b/tests/data/surf_poscar.json @@ -0,0 +1,46 @@ +{ + "stages": [ + 1, + 2 + ], + "cell_type": "fcc", + "from_poscar": true, + "from_poscar_path": "POSCAR", + "super_cell": [ + 1, + 1, + 1 + ], + "layer_numb": 3, + "vacuum_max": 9, + "vacuum_resol": [ + 0.5, + 1 + ], + "mid_point": 4.0, + "head_ratio": 0.6, + "vacuum_numb": 20, + "millers": [ + [ + 1, + 0, + 0 + ] + ], + "elements": [ + "Al" + ], + "potcars": [ + "./POTCAR" + ], + "relax_incar" : "INCAR_metal_rlx_low", + "scale": [ + 1.0 + ], + "skip_relax": true, + "pert_numb": 5, + "pert_box": 0.03, + "pert_atom": 0.01, + "coll_ndata": 5000, + "_comment": "that's all" +} diff --git a/tests/data/test_gen_surf_poscar.py b/tests/data/test_gen_surf_poscar.py new file mode 100644 index 000000000..5c49b0578 --- /dev/null +++ b/tests/data/test_gen_surf_poscar.py @@ -0,0 +1,48 @@ +import os,sys,json,glob,shutil +import unittest +from pymatgen import Structure + +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) +__package__ = 'data' +from .context import setUpModule +from .context_surf_poscar import * + +class TestGenSurfPOSCAR(unittest.TestCase): + def setUp(self): + self.surfs=["surf-100"] + self.elongs=["elong-0.500", "elong-1.000", "elong-1.500", "elong-2.000", "elong-2.500",\ + "elong-3.000", "elong-3.500", "elong-4.000", "elong-5.000", "elong-6.000",\ + "elong-7.000", "elong-8.000" ] + with open (param_file, 'r') as fp : + jdata = json.load (fp) + out_dir = out_dir_name(jdata) + jdata['out_dir'] = out_dir + self.root_dir= out_dir + create_path(out_dir) + make_super_cell_pymatgen(jdata) + place_element(jdata) + make_vasp_relax(jdata) + make_scale(jdata) + pert_scaled(jdata) + + def tearDown(self): + shutil.rmtree(self.root_dir) + + def test(self): + surfs=glob.glob("POSCAR.01x01x01/01.scale_pert/surf*") + surfs=[ii.split('/')[-1] for ii in surfs] + surfs.sort() + self.assertEqual(surfs,self.surfs) + poscars=glob.glob("POSCAR.01x01x01/00.place_ele/surf*/sys*/POSCAR") + for poscar in poscars: + surf=poscar.split('/')[-3] + st1=Structure.from_file(surf+'.POSCAR') + st2=Structure.from_file(poscar) + self.assertEqual(st1,st2) + + for surf in self.surfs: + elongs=glob.glob("POSCAR.01x01x01/01.scale_pert/"+surf+"/sys-*/scale-1.000/el*") + elongs=[ii.split('/')[-1] for ii in elongs] + elongs.sort() + self.assertEqual(elongs,self.elongs) + diff --git a/tests/dispatcher/context.py b/tests/dispatcher/context.py index 4519ca9e6..1ab29dc9a 100644 --- a/tests/dispatcher/context.py +++ b/tests/dispatcher/context.py @@ -7,7 +7,7 @@ from dpgen.dispatcher.LazyLocalContext import LazyLocalContext from dpgen.dispatcher.SSHContext import SSHSession from dpgen.dispatcher.SSHContext import SSHContext -from dpgen.dispatcher.Dispatcher import FinRecord +# from dpgen.dispatcher.Dispatcher import FinRecord from dpgen.dispatcher.Dispatcher import _split_tasks from dpgen.dispatcher.LocalContext import _identical_files diff --git a/tests/dispatcher/shell/test_shell_local.py b/tests/dispatcher/shell/test_shell_local.py index 36a1eb119..b4d02ba67 100644 --- a/tests/dispatcher/shell/test_shell_local.py +++ b/tests/dispatcher/shell/test_shell_local.py @@ -33,13 +33,20 @@ def tearDown(self): if os.path.exists('run.sub.1'): os.remove('run.sub.1') - def test_manual_gpu(self): + def test_manual_cuda_devices(self): job_dirs = ['task0', 'task1'] res = {'manual_cuda_devices': 3} ret = self.shell.sub_script(job_dirs, ['touch test1', 'touch test2'], res = res) with open('run.sub.gpu', 'w') as fp: fp.write(ret) + def test_manual_cuda_multiplicity(self): + job_dirs = ['task0', 'task1', 'task2', 'task3'] + res = {'manual_cuda_devices': 2, 'manual_cuda_multiplicity': 2} + ret = self.shell.sub_script(job_dirs, ['touch test1', 'touch test2'], res = res) + with open('run.sub.gpu.multi', 'w') as fp: + fp.write(ret) + def test_gen_sub_script(self): job_dirs = ['task0', 'task1'] self.shell.context.upload(job_dirs, ['test0']) diff --git a/tests/dispatcher/test_dispatcher_utils.py b/tests/dispatcher/test_dispatcher_utils.py index 00b6c5fbc..01f0e0a1f 100644 --- a/tests/dispatcher/test_dispatcher_utils.py +++ b/tests/dispatcher/test_dispatcher_utils.py @@ -3,29 +3,29 @@ sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) __package__ = 'dispatcher' -from .context import FinRecord +# from .context import FinRecord from .context import _split_tasks from .context import setUpModule -class TestFinRecord(unittest.TestCase): - def setUp(self): - self.njobs = 10 - self.fr = FinRecord('.', self.njobs) - - def tearDown(self): - if os.path.isfile('fin.record'): - os.remove('fin.record') - - def test_all_false(self) : - recd = self.fr.get_record() - self.assertEqual(recd, [False]*self.njobs) - - def test_write_read(self) : - recd = self.fr.get_record() - recd[self.njobs//3] = True - self.fr.write_record(recd) - recd1 = self.fr.get_record() - self.assertEqual(recd, recd1) +# class TestFinRecord(unittest.TestCase): +# def setUp(self): +# self.njobs = 10 +# self.fr = FinRecord('.', self.njobs) + +# def tearDown(self): +# if os.path.isfile('fin.record'): +# os.remove('fin.record') + +# def test_all_false(self) : +# recd = self.fr.get_record() +# self.assertEqual(recd, [False]*self.njobs) + +# def test_write_read(self) : +# recd = self.fr.get_record() +# recd[self.njobs//3] = True +# self.fr.write_record(recd) +# recd1 = self.fr.get_record() +# self.assertEqual(recd, recd1) class TestDispatchSplit(unittest.TestCase): def test_split(self): diff --git a/tests/generator/context.py b/tests/generator/context.py index 75baecb0d..0668e7fe2 100644 --- a/tests/generator/context.py +++ b/tests/generator/context.py @@ -14,6 +14,7 @@ param_gaussian_file = 'param-pyridine-gaussian.json' param_siesta_file = 'param-pyridine-siesta.json' param_cp2k_file = 'param-pyridine-cp2k.json' +param_cp2k_file_v1 = 'param-pyridine-cp2k-choose-vdw.json' machine_file = 'machine-local.json' machine_file_v1 = 'machine-local-v1.json' param_diy_file = 'param-mg-vasp-diy.json' diff --git a/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000000/POSCAR b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000000/POSCAR new file mode 100644 index 000000000..c0d25f361 --- /dev/null +++ b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000000/POSCAR @@ -0,0 +1,40 @@ +POSCAR file written by OVITO +1.04 + 8.0852460528260064 0.0000000700556454 0.0000000165920926 + 0.0000000700459398 8.0852460812690730 -0.0000000878331783 + 0.0000000165994710 -0.0000000878293815 8.0852460822851544 + Al + 32 +Direct + 0.5000000055222689 0.0000000095334217 0.4999999965645718 + 0.5000000034529869 0.7499999969981677 0.2499999981901819 + 0.5000000083231070 -0.0000000011197416 0.0000000014389411 + -0.0000000004748536 0.0000000005310409 0.0000000063786655 + 0.7499999995103841 0.0000000012827216 0.2499999977659152 + 0.0000000000448991 0.0000000048734801 0.5000000037789456 + 0.2499999999857682 0.5000000028441646 0.7500000018075699 + 0.2500000030261135 0.7499999944555688 0.5000000026313640 + 0.2500000097291175 0.5000000129780301 0.2499999975347162 + 0.5000000035346804 0.7499999971754691 0.7500000063111742 + 0.0000000017515228 0.4999999968593779 0.5000000121259237 + 0.7499999970723630 0.7499999983979078 0.4999999968129619 + 0.7499999910768409 0.0000000040350684 0.7499999999698314 + -0.0000000040407868 0.7499999975356930 0.2499999972984601 + -0.0000000013440370 0.5000000065441171 -0.0000000011351674 + 0.5000000069922398 0.2500000092556521 0.7499999925172053 + 0.2500000074218809 -0.0000000006822529 0.2500000011219505 + 0.7499999954414514 0.2500000084865056 -0.0000000006412232 + 0.7499999967294295 0.4999999938307768 0.7500000077731870 + 0.5000000030383994 0.5000000063795264 -0.0000000001750512 + 0.2500000064022705 0.7499999919020213 0.0000000019913002 + 0.0000000043319078 0.7499999907276940 0.7500000023288846 + 0.2500000021034525 0.0000000026582450 0.7499999979035360 + 0.7500000002384054 0.5000000058460430 0.2499999970004410 + 0.2500000006602736 0.2500000097862060 -0.0000000026740404 + -0.0000000102302524 0.2500000011729165 0.2500000007828680 + -0.0000000041345229 0.2500000119032792 0.7499999956587747 + 0.7499999985377536 0.7499999945443449 0.0000000074372283 + 0.5000000022954397 0.4999999989012168 0.5000000041503138 + 0.2500000115215270 0.2500000006209973 0.5000000113478330 + 0.7500000009516035 0.2500000011452493 0.5000000093672294 + 0.5000000100332991 0.2500000001020252 0.2500000061404412 diff --git a/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000001/POSCAR b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000001/POSCAR new file mode 100644 index 000000000..822bfddde --- /dev/null +++ b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000001/POSCAR @@ -0,0 +1,40 @@ +Al + 1.0000000000000000 + 8.2797705159728547 0.0000000000000000 0.0000000000000000 + -0.1719719010892178 8.6486456546964234 0.0000000000000000 + -0.0010128078172081 0.1054655382171854 8.6490647580104696 + Al + 32 +Cartesian + 4.1405303605609065 -0.0002311813140235 -0.0001044821347404 + 6.2101270205420587 0.0274025201585026 2.1631618555812433 + 1.9860812664680023 4.3565843685598704 2.1650983474684780 + 4.0078083948451368 6.5615160044379568 6.4876604020180846 + 2.0278390331727807 2.2135513637631408 4.3236944572396503 + 4.0953786649781048 2.2492100294422253 6.4810339435242588 + -0.0875414497269289 4.3743834013208236 4.3201317850229435 + 2.0736042567106190 0.0228671778411013 2.1619287357074879 + 6.1696411956777881 2.2150078661563208 4.3251923877351022 + 2.0231935553242515 2.1675246049264332 -0.0015871536767673 + 2.0697228575465982 0.0796308216693133 6.4881338826802768 + 6.0718728633823593 6.5345829673724909 4.3249524423528563 + 6.1260924754969297 4.3544178531848186 2.1700539999418975 + 6.0794537263845294 6.4852313292416728 0.0015781459271990 + -0.0432175780609881 2.1967100107205435 2.1657427325361875 + -0.1253958469750332 6.5189656427965783 2.1567228526414568 + 0.0011475057041571 0.0057830702773709 0.0037842622675432 + 1.9347714457652205 6.5376281930141227 4.3309171757107432 + -0.1318239919085722 6.5598304611092644 6.4942196366725833 + 6.2126013464109491 0.0748052719901856 6.4830859016113873 + 4.0566814116069088 4.3260615903766206 0.0036768623969541 + 4.0552518198425744 4.3755027511287912 4.3233549827874995 + 4.0931949504606155 2.1918523869765418 2.1567781105403996 + -0.0859138586713617 4.3244750685396607 -0.0001642409340360 + 4.1410642367747599 0.0526367919882268 4.3269464189048445 + 1.9824002179387932 4.4020860732001070 6.4886621332933405 + 6.1694313242655578 2.1596032692396117 0.0024578050259201 + -0.0382210609417227 2.2399140383166798 6.4831900551393771 + 4.0090325775580240 6.5139439803656938 2.1610083538493630 + 1.9422012459502989 6.4887816039836386 -0.0014546717614078 + -0.0024327533265666 0.0493459591650801 4.3216382226840757 + 6.1257798713385236 4.4062854392471484 6.4890048938443261 diff --git a/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000002/POSCAR b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000002/POSCAR new file mode 100644 index 000000000..a32dbcdcf --- /dev/null +++ b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.040/000002/POSCAR @@ -0,0 +1,40 @@ +Al + 1.0000000000000000 + 8.2095456552916222 0.0000000000000000 0.0000000000000000 + -0.2286402743603651 8.6043644363763701 0.0000000000000000 + -0.0854312756696513 0.2303641815368059 8.2540862211675634 + Al + 32 +Cartesian + 4.0306771832041433 2.2066722958454528 2.0659499762400642 + 3.9151098746436204 6.5167215563456802 2.0596964734510785 + 4.1077432130635714 0.0034764787551480 -0.0006236073814073 + 6.1379489191490046 0.0621351322778998 2.0578263563441910 + 3.9920482810774476 4.3011538909470639 0.0006335990343001 + -0.1949340111748481 6.5111390966709761 2.0643396931358104 + -0.0463856323727473 0.1142994955204610 4.1244311196037833 + 6.0952948094811621 0.1729888932314524 6.1926808591164919 + 4.0664344218920485 0.1100139469701655 4.1206812470148089 + 6.0998444929030891 2.1505475346234428 -0.0008449794097100 + 3.9863296520655518 2.3222885078447568 6.1918144614588400 + -0.1209016406533330 2.3228404619168743 6.1906560340571639 + 1.8807466235222732 6.4532438855725953 -0.0001516652152993 + -0.2395163861307614 6.6235798336430829 6.1872455636950026 + 0.0025147768770161 -0.0036452495960412 0.0013932280132470 + -0.0774171953107122 2.2090747731610469 2.0631980492036508 + 1.9926291269205347 0.1692401152050739 6.1955859284074473 + 1.8357205748765402 6.5677154774808173 4.1263622066414323 + 6.0605653449404215 2.2579040246556001 4.1295688716429204 + -0.1145731952354822 4.3025332838938875 0.0002034325056423 + 2.0285609270479301 0.0566514378431752 2.0691391271530861 + 3.9484454855523943 4.4183019897368441 4.1259239183811047 + -0.1562941091523503 4.4148144783033310 4.1218372204552605 + 5.9888446160288860 6.4572474392376282 -0.0053712419664877 + 1.8705640110869806 4.4688581130775136 6.1928385980937639 + 6.0243391462447384 4.3596689965884412 2.0572276103221316 + 1.9548418838285899 2.2603793712918661 4.1249155308753114 + 3.8725040967945219 6.6258873546504624 6.1934468686522104 + 1.9164525698273898 4.3586937734571798 2.0631666976308018 + 5.9422421861362489 6.5697524312290252 4.1250284026673434 + 1.9946789721739846 2.1477356173779114 -0.0011905739695201 + 5.9756996701929497 4.4740331240564739 6.1925671609051642 diff --git a/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000000/POSCAR b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000000/POSCAR new file mode 100644 index 000000000..d2c2f1415 --- /dev/null +++ b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000000/POSCAR @@ -0,0 +1,40 @@ +POSCAR file written by OVITO +1.06 + 8.0852460528260064 0.0000000700556454 0.0000000165920926 + 0.0000000700459398 8.0852460812690730 -0.0000000878331783 + 0.0000000165994710 -0.0000000878293815 8.0852460822851544 + Al + 32 +Direct + 0.5000000034529869 0.7499999969981677 0.2499999981901819 + 0.5000000083231070 -0.0000000011197416 0.0000000014389411 + 0.7500000009516035 0.2500000011452493 0.5000000093672294 + 0.7499999954414514 0.2500000084865056 -0.0000000006412232 + 0.5000000069922398 0.2500000092556521 0.7499999925172053 + 0.2500000074218809 -0.0000000006822529 0.2500000011219505 + 0.0000000000448991 0.0000000048734801 0.5000000037789456 + 0.0000000043319078 0.7499999907276940 0.7500000023288846 + 0.7500000002384054 0.5000000058460430 0.2499999970004410 + 0.5000000022954397 0.4999999989012168 0.5000000041503138 + 0.2500000097291175 0.5000000129780301 0.2499999975347162 + 0.0000000017515228 0.4999999968593779 0.5000000121259237 + 0.5000000055222689 0.0000000095334217 0.4999999965645718 + 0.5000000035346804 0.7499999971754691 0.7500000063111742 + 0.2500000006602736 0.2500000097862060 -0.0000000026740404 + -0.0000000041345229 0.2500000119032792 0.7499999956587747 + 0.2499999999857682 0.5000000028441646 0.7500000018075699 + 0.7499999985377536 0.7499999945443449 0.0000000074372283 + 0.7499999995103841 0.0000000012827216 0.2499999977659152 + 0.7499999967294295 0.4999999938307768 0.7500000077731870 + 0.2500000115215270 0.2500000006209973 0.5000000113478330 + 0.7499999970723630 0.7499999983979078 0.4999999968129619 + -0.0000000004748536 0.0000000005310409 0.0000000063786655 + 0.7499999910768409 0.0000000040350684 0.7499999999698314 + -0.0000000040407868 0.7499999975356930 0.2499999972984601 + -0.0000000102302524 0.2500000011729165 0.2500000007828680 + 0.2500000030261135 0.7499999944555688 0.5000000026313640 + 0.2500000021034525 0.0000000026582450 0.7499999979035360 + 0.5000000100332991 0.2500000001020252 0.2500000061404412 + -0.0000000013440370 0.5000000065441171 -0.0000000011351674 + 0.5000000030383994 0.5000000063795264 -0.0000000001750512 + 0.2500000064022705 0.7499999919020213 0.0000000019913002 diff --git a/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000001/POSCAR b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000001/POSCAR new file mode 100644 index 000000000..36606db02 --- /dev/null +++ b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000001/POSCAR @@ -0,0 +1,40 @@ +Al + 1.0000000000000000 + 8.3598251083796260 0.0000000000000000 0.0000000000000000 + -0.1014062409602743 8.7520905745998139 0.0000000000000000 + -0.0687187698129919 0.2297954247305231 8.4431009214511974 + Al + 32 +Cartesian + 6.1683189190564951 4.5502784129692708 6.3296577854868010 + 6.1965525638957679 6.5680765848084750 0.0029863609922990 + 2.0396624544096955 0.1732344670827352 6.3322678633817251 + 6.2130901492377939 2.2989932295758888 4.2302216016831782 + -0.0520945632303641 4.3765674258355460 -0.0019779716557334 + -0.0014490531239594 0.0012499082614180 -0.0010991018816276 + -0.1314129164714025 6.7351442984719725 6.3334773536245867 + 4.0456687581733002 6.7364197513151751 6.3249997121311603 + -0.0937651071923047 6.6208677247742465 2.1123046275329069 + 4.0960677591482924 4.4926744904871390 4.2211873921508687 + 4.0880108686193317 6.6202639448043019 2.1017079440084196 + -0.0795969571128076 2.3607658696959479 6.3291855425176813 + 1.9821746056744098 6.6786660367944020 4.2232289243210728 + 2.0619415753574173 2.1905908271047374 0.0029559804043806 + -0.0423314300948392 2.2481168790599964 2.1152302614428637 + 6.2580765405823202 0.0586103911760227 2.1189007725333151 + 6.2474387239813538 2.1793900662909729 0.0007938903889081 + 2.0147829169000611 6.5645865794055860 -0.0001444989147900 + -0.0258599514919108 0.1117266738246711 4.2242506233803780 + 4.1299576049645594 2.2428788732388525 2.1156576001669882 + 4.1780305116160363 0.0007882753632893 0.0020554992942880 + 6.1980770425407243 4.4325703498605566 2.1177678801435378 + 2.0761042989529921 0.0635168993108198 2.1141164816869815 + 4.1245594764626183 4.3737564400720563 0.0016920716977401 + 4.1049053635275321 2.3584914444124645 6.3317740374727416 + 2.0300731360268154 2.3110137563067226 4.2250342291910101 + -0.0825921356405149 4.5001269110724493 4.2197502279248438 + 6.2200264777784939 0.1690022116720367 6.3297823964633926 + 4.1402157372025750 0.1130411806163994 4.2157293332569230 + 1.9873675974746883 4.5475018134559129 6.3330281685682630 + 2.0218653030525817 4.4335739372995064 2.1107199808299701 + 6.1513189901181322 6.6792290755422474 4.2210350859448864 diff --git a/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000002/POSCAR b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000002/POSCAR new file mode 100644 index 000000000..ab31e5a1a --- /dev/null +++ b/tests/generator/data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale-1.060/000002/POSCAR @@ -0,0 +1,40 @@ +Al + 1.0000000000000000 + 8.5898428958538098 0.0000000000000000 0.0000000000000000 + 0.0309441405711094 8.4546949858918694 0.0000000000000000 + -0.0097566977499483 0.1352429661398475 8.6069624601986590 + Al + 32 +Cartesian + 6.4505233521585650 2.1132720745167752 -0.0002917768226800 + 4.2856914127143968 0.0029095529351454 0.0017848218181846 + 4.3222301043099822 6.3756488928484174 2.1447310524399770 + 2.1717132096933316 6.3408284501098091 0.0012979398509617 + 6.4660412818589155 6.3408284140534965 0.0008257072136749 + 6.4415828204454799 0.0373236972123070 2.1567463274061964 + 4.2931753316114944 0.0669786487775630 4.3027987284844418 + 0.0194443480527756 6.4431701023804635 6.4497730612072282 + 4.3060466719098018 4.2307551072148986 -0.0076141629479681 + 6.4647457206166568 6.4159376018330612 4.3043627733846455 + 6.4543672215286696 4.2616842707592282 2.1501340036161087 + 0.0178633739715773 6.3773805485049309 2.1510337600328335 + 0.0074211862640189 2.1388573002970750 2.1499882226977185 + -0.0090807869046197 0.0663696800456818 4.3063098880206878 + 2.1561472240230288 2.1159006184690967 0.0003640614342978 + 2.1666028329523659 6.4100635002351751 4.3039636538495722 + 4.2938374404072981 2.2154389707323623 6.4518622611638312 + 2.1607276454484490 4.2586788732587717 2.1431275751005945 + 2.1545652702527165 2.1818588369923066 4.2971275692596187 + 0.0154806579451338 4.2247284677183679 0.0023145646149488 + 6.4488405336196610 2.1799745727408162 4.3004273550219212 + -0.0067627517102337 2.2110390808792646 6.4508074158614228 + 2.1450165614049719 0.0337577346617454 2.1516105872138498 + 4.3028135306784758 2.1478273727691706 2.1476429789225437 + -0.0016945267055740 0.0002113575755896 0.0011305540017342 + 2.1361451381142724 0.0977829525904214 6.4504510973337910 + 6.4334091010848260 0.1075410768182249 6.4607377349253454 + 6.4486872853119488 4.3313683286982663 6.4579060261439194 + 4.3048400221154921 4.2973032817718257 4.3031856299503035 + 0.0120088951200463 4.2991197661156040 4.3076853311025882 + 4.3123633082405872 6.4471478938493689 6.4530621833311201 + 2.1555961162651558 4.3296495732327607 6.4545934390565591 diff --git a/tests/generator/lmp/input.lammps b/tests/generator/lmp/input.lammps new file mode 100644 index 000000000..946ad05c5 --- /dev/null +++ b/tests/generator/lmp/input.lammps @@ -0,0 +1,35 @@ +variable NSTEPS equal V_NSTEPS +variable THERMO_FREQ equal 10 +variable DUMP_FREQ equal 10 +variable TEMP equal V_TEMP +variable PRES equal V_PRES +variable TAU_T equal 0.100000 +variable TAU_P equal 0.500000 + +units metal +boundary p p p +atom_style atomic + +neighbor 1.0 bin + +box tilt large +read_data conf.lmp +change_box all triclinic +mass 1 27.000000 +mass 2 24.000000 + +pair_style deepmd ../graph.003.pb ../graph.001.pb ../graph.002.pb ../graph.000.pb out_freq ${THERMO_FREQ} out_file model_devi.out +pair_coeff + +fix dpgen_plm + +thermo_style custom step temp pe ke etotal press vol lx ly lz xy xz yz +thermo ${THERMO_FREQ} + +dump dpgen_dump + +velocity all create ${TEMP} 826513 +fix 1 all npt temp ${TEMP} ${TEMP} ${TAU_T} iso ${PRES} ${PRES} ${TAU_P} + +timestep 0.002000 +run ${NSTEPS} diff --git a/tests/generator/lmp/input.plumed b/tests/generator/lmp/input.plumed new file mode 100644 index 000000000..598b924b1 --- /dev/null +++ b/tests/generator/lmp/input.plumed @@ -0,0 +1,4 @@ +DISTANCE ATOMS=3,5 LABEL=d1 +DISTANCE ATOMS=2,4 LABEL=d2 +RESTRAINT ARG=d1,d2 AT=V_DIST0,V_DIST1 KAPPA=150.0,150.0 LABEL=restraint +PRINT ARG=restraint.bias diff --git a/tests/generator/param-pyridine-cp2k.json b/tests/generator/param-pyridine-cp2k.json index 09ccc4d3a..d1e27aafc 100644 --- a/tests/generator/param-pyridine-cp2k.json +++ b/tests/generator/param-pyridine-cp2k.json @@ -101,16 +101,16 @@ "fp_pp_path": ".", "fp_pp_files": [], "user_fp_params": { - "cutoff": "400", - "rel_cutoff": "50", - "functional": "PBE", - "pair_potential_path": "./cp2k_basis_pp_file/dftd3.dat", - "pair_ref_functional": "PBE", - "basis_path": "./cp2k_basis_pp_file/BASIS_MOLOPT", - "pp_path": "./cp2k_basis_pp_file/GTH_POTENTIALS", - "element_list": ["H", "C", "N"], - "basis_list": ["DZVP-MOLOPT-GTH", "DZVP-MOLOPT-GTH", "DZVP-MOLOPT-GTH"], - "pp_list": ["GTH-PBE-q1", "GTH-PBE-q4", "GTH-PBE-q5"] + "FORCE_EVAL":{ + "SUBSYS":{ + "KIND":{ + "_": ["N","C","H"], + "POTENTIAL": ["GTH-PBE-q5","GTH-PBE-q4", "GTH-PBE-q1"], + "BASIS_SET": ["DZVP-MOLOPT-GTH","DZVP-MOLOPT-GTH","DZVP-MOLOPT-GTH"] + } + } + } + }, "_comment": " that's all " } diff --git a/tests/generator/test_make_fp.py b/tests/generator/test_make_fp.py index 20fdc51dc..e77e94d36 100644 --- a/tests/generator/test_make_fp.py +++ b/tests/generator/test_make_fp.py @@ -15,6 +15,7 @@ from .context import param_siesta_file from .context import param_gaussian_file from .context import param_cp2k_file +from .context import param_cp2k_file_v1 from .context import machine_file from .context import param_diy_file from .context import make_kspacing_kpoints @@ -159,14 +160,6 @@ &XC\n\ &XC_FUNCTIONAL PBE\n\ &END XC_FUNCTIONAL\n\ -&VDW_POTENTIAL\n\ -DISPERSION_FUNCTIONAL PAIR_POTENTIAL\n\ -&PAIR_POTENTIAL\n\ -TYPE DFTD3\n\ -PARAMETER_FILE_NAME ./cp2k_basis_pp_file/dftd3.dat\n\ -REFERENCE_FUNCTIONAL PBE\n\ -&END PAIR_POTENTIAL\n\ -&END VDW_POTENTIAL\n\ &END XC\n\ &END DFT\n\ &SUBSYS\n\ @@ -195,6 +188,8 @@ &END FORCE_EVAL\n" + + def _box2lmpbox(orig, box) : lohi = np.zeros([3,2]) for dd in range(3) : @@ -384,7 +379,7 @@ def _check_incar(testCase, idx): fp_path = os.path.join('iter.%06d' % idx, '02.fp') tasks = glob.glob(os.path.join(fp_path, 'task.*')) cwd = os.getcwd() - for ii in tasks : + for ii in tasks : os.chdir(ii) with open('INCAR') as fp: incar = fp.read() @@ -395,7 +390,7 @@ def _check_incar_ele_temp(testCase, idx, ele_temp): fp_path = os.path.join('iter.%06d' % idx, '02.fp') tasks = glob.glob(os.path.join(fp_path, 'task.*')) cwd = os.getcwd() - for ii in tasks : + for ii in tasks : os.chdir(ii) bname = os.path.basename(ii) sidx = int(bname.split('.')[1]) @@ -473,6 +468,7 @@ def _check_cp2k_input_head(testCase, idx) : testCase.assertEqual(('\n'.join(lines_check)).strip(), cp2k_input_ref.strip()) + class TestMakeFPPwscf(unittest.TestCase): def test_make_fp_pwscf(self): if os.path.isdir('iter.000000') : diff --git a/tests/generator/test_make_md.py b/tests/generator/test_make_md.py index 6c35b4b30..3ee930344 100644 --- a/tests/generator/test_make_md.py +++ b/tests/generator/test_make_md.py @@ -1,4 +1,4 @@ -import os,sys,json,glob,shutil +import os,sys,json,glob,shutil,copy import dpdata import numpy as np import unittest @@ -7,10 +7,16 @@ __package__ = 'generator' from .context import make_model_devi from .context import parse_cur_job +from .context import parse_cur_job_revmat from .context import param_file from .context import machine_file from .context import my_file_cmp from .context import setUpModule +from .context import find_only_one_key +from .context import revise_lmp_input_model +from .context import revise_lmp_input_dump +from .context import revise_lmp_input_plm +from .context import revise_by_keys from .comp_sys import test_atom_names from .comp_sys import test_atom_types from .comp_sys import test_coord @@ -53,7 +59,7 @@ def _check_confs(testCase, idx, jdata) : l_conf_file = os.path.basename(os.readlink(conf_file)) poscar_file = poscars[int(l_conf_file.split('.')[0])][int(l_conf_file.split('.')[1])] sys_0 = dpdata.System(conf_file, type_map = jdata['type_map']) - sys_1 = dpdata.System(poscar_file) + sys_1 = dpdata.System(poscar_file, type_map = jdata['type_map']) test_atom_names(testCase, sys_0, sys_1) test_atom_types(testCase, sys_0, sys_1) test_cell(testCase, sys_0, sys_1) @@ -123,6 +129,10 @@ def _check_pt(testCase, idx, jdata) : class TestMakeModelDevi(unittest.TestCase): + def tearDown(self): + if os.path.isdir('iter.000000') : + shutil.rmtree('iter.000000') + def test_make_model_devi (self) : if os.path.isdir('iter.000000') : shutil.rmtree('iter.000000') @@ -137,7 +147,253 @@ def test_make_model_devi (self) : _check_traj_dir(self, 0) _check_pt(self, 0, jdata) shutil.rmtree('iter.000000') + + + +class TestMakeModelDeviRevMat(unittest.TestCase): + def tearDown(self): + if os.path.isdir('iter.000000') : + shutil.rmtree('iter.000000') + + def test_make_model_devi (self) : + if os.path.isdir('iter.000000') : + shutil.rmtree('iter.000000') + jdata = { + "type_map": ["Mg", "Al"], + "mass_map": [24, 27], + "init_data_prefix": "data", + "init_data_sys": ["deepmd"], + "init_batch_size": [16], + "sys_configs_prefix": os.getcwd(), + "sys_configs": [ + ["data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale*/000001/POSCAR"], + ["data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale*/000000/POSCAR"] + ], + "numb_models": 4, + "shuffle_poscar": False, + "model_devi_f_trust_lo": 0.050, + "model_devi_f_trust_hi": 0.150, + "model_devi_e_trust_lo": 1e10, + "model_devi_e_trust_hi": 1e10, + "model_devi_plumed": True, + "model_devi_jobs": [ + {"sys_idx": [0, 1], 'traj_freq': 10, "template":{"lmp": "lmp/input.lammps", "plm": "lmp/input.plumed"}, + "rev_mat":{ + "lmp": {"V_NSTEPS": [1000], "V_TEMP": [50, 100], "V_PRES": [1, 10]}, "plm": {"V_DIST0": [3,4], "V_DIST1": [5, 6]} + }} + ] + } + mdata = {'deepmd_version': '1'} + _make_fake_models(0, jdata['numb_models']) + make_model_devi(0, jdata, mdata) + _check_pb(self, 0) + _check_confs(self, 0, jdata) + _check_traj_dir(self, 0) + # check the first task + md_dir = os.path.join('iter.%06d' % 0, '01.model_devi') + tasks = glob.glob(os.path.join(md_dir, 'task.*')) + tasks.sort() + # 4 accounts for 2 systems each with 2 frames + self.assertEqual(len(tasks), (len(jdata['model_devi_jobs'][0]['rev_mat']['lmp']['V_NSTEPS']) * + len(jdata['model_devi_jobs'][0]['rev_mat']['lmp']['V_TEMP']) * + len(jdata['model_devi_jobs'][0]['rev_mat']['lmp']['V_PRES']) * + len(jdata['model_devi_jobs'][0]['rev_mat']['plm']['V_DIST0']) * + len(jdata['model_devi_jobs'][0]['rev_mat']['plm']['V_DIST1']) * + 4)) + + cur_job = jdata['model_devi_jobs'][0] + rev_keys = ['V_NSTEPS', 'V_TEMP', 'V_PRES', 'V_DIST0', 'V_DIST1'] + rev_matrix = [] + for i0 in cur_job['rev_mat']['lmp']['V_NSTEPS']: + for i1 in cur_job['rev_mat']['lmp']['V_TEMP']: + for i2 in cur_job['rev_mat']['lmp']['V_PRES']: + for i3 in cur_job['rev_mat']['plm']['V_DIST0']: + for i4 in cur_job['rev_mat']['plm']['V_DIST1']: + rev_matrix.append([i0, i1, i2, i3, i4]) + numb_rev = len(rev_matrix) + for ii in range(len(tasks)): + with open(os.path.join(tasks[ii], 'job.json')) as fp: + rev_values = rev_matrix[ii % numb_rev] + job_recd = json.load(fp) + for kk in job_recd.keys(): + kidx = rev_keys.index(kk) + self.assertEqual(rev_values[kidx], job_recd[kk]) + + cwd_ = os.getcwd() + os.chdir(tasks[0]) + with open('input.lammps') as fp: + lines = fp.readlines() + for ii in lines: + if 'variable' in ii and 'TEMP' in ii: + self.assertEqual('variable TEMP equal 50', + ' '.join(ii.split())) + if 'variable' in ii and 'PRES' in ii: + self.assertEqual('variable PRES equal 1', + ' '.join(ii.split())) + if 'variable' in ii and 'NSTEPS' in ii: + self.assertEqual('variable NSTEPS equal 1000', + ' '.join(ii.split())) + with open('input.plumed') as fp: + lines = fp.readlines() + for ii in lines: + if 'RESTRAINT' in ii: + self.assertEqual('RESTRAINT ARG=d1,d2 AT=3,5 KAPPA=150.0,150.0 LABEL=restraint', + ' '.join(ii.split())) + os.chdir(cwd_) + + + def test_make_model_devi_null (self) : + if os.path.isdir('iter.000000') : + shutil.rmtree('iter.000000') + jdata = { + "type_map": ["Mg", "Al"], + "mass_map": [24, 27], + "init_data_prefix": "data", + "init_data_sys": ["deepmd"], + "init_batch_size": [16], + "sys_configs_prefix": os.getcwd(), + "sys_configs": [ + ["data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale*/000001/POSCAR"], + ["data/al.fcc.02x02x02/01.scale_pert/sys-0032/scale*/000000/POSCAR"] + ], + "numb_models": 4, + "shuffle_poscar": False, + "model_devi_f_trust_lo": 0.050, + "model_devi_f_trust_hi": 0.150, + "model_devi_e_trust_lo": 1e10, + "model_devi_e_trust_hi": 1e10, + "model_devi_plumed": True, + "model_devi_jobs": [ + {"sys_idx": [0, 1], 'traj_freq': 10, "template":{"lmp": "lmp/input.lammps", "plm": "lmp/input.plumed"}, + } + ] + } + mdata = {'deepmd_version': '1'} + _make_fake_models(0, jdata['numb_models']) + make_model_devi(0, jdata, mdata) + _check_pb(self, 0) + _check_confs(self, 0, jdata) + _check_traj_dir(self, 0) + # check the first task + md_dir = os.path.join('iter.%06d' % 0, '01.model_devi') + tasks = glob.glob(os.path.join(md_dir, 'task.*')) + # 4 accounts for 2 systems each with 2 frames + self.assertEqual(len(tasks), (4)) + tasks.sort() + cwd_ = os.getcwd() + os.chdir(tasks[0]) + with open('input.lammps') as fp: + lines = fp.readlines() + for ii in lines: + if 'variable' in ii and 'TEMP' in ii: + self.assertEqual('variable TEMP equal V_TEMP', + ' '.join(ii.split())) + if 'variable' in ii and 'PRES' in ii: + self.assertEqual('variable PRES equal V_PRES', + ' '.join(ii.split())) + if 'variable' in ii and 'NSTEPS' in ii: + self.assertEqual('variable NSTEPS equal V_NSTEPS', + ' '.join(ii.split())) + with open('input.plumed') as fp: + lines = fp.readlines() + for ii in lines: + if 'RESTRAINT' in ii: + self.assertEqual('RESTRAINT ARG=d1,d2 AT=V_DIST0,V_DIST1 KAPPA=150.0,150.0 LABEL=restraint', + ' '.join(ii.split())) + os.chdir(cwd_) + + + + +class TestParseCurJobRevMat(unittest.TestCase): + def setUp(self): + self.cur_job = { + "sys_idx": [0, 1], + "template":{"lmp": "lmp/input.lammps", "plm": "lmp/input.plumed"}, + "rev_mat":{ + "lmp": {"V_NSTEPS": [1000], "V_TEMP": [50, 100], "V_PRES": [1, 10]}, "plm": {"V_DIST0": [3,4], "V_DIST1": [5, 6]} + } + } + self.ref_matrix = [] + for i0 in self.cur_job['rev_mat']['lmp']['V_NSTEPS']: + for i1 in self.cur_job['rev_mat']['lmp']['V_TEMP']: + for i2 in self.cur_job['rev_mat']['lmp']['V_PRES']: + for i3 in self.cur_job['rev_mat']['plm']['V_DIST0']: + for i4 in self.cur_job['rev_mat']['plm']['V_DIST1']: + self.ref_matrix.append([i0, i1, i2, i3, i4]) + self.ref_keys = ['V_NSTEPS', 'V_TEMP', 'V_PRES', 'V_DIST0', 'V_DIST1'] + self.ref_nlmp = 3 + def test_parse_cur_job(self): + rk, rm, nl = parse_cur_job_revmat(self.cur_job, use_plm = True) + self.assertEqual(rk, self.ref_keys) + self.assertEqual(nl, self.ref_nlmp) + self.assertEqual(rm, self.ref_matrix) + + +class MakeModelDeviByReviseMatrix(unittest.TestCase): + def test_find_only_one_key_1(self): + lines = ['aaa bbb ccc\n', 'bbb ccc\n', 'ccc bbb ccc\n'] + idx = find_only_one_key(lines, ['bbb', 'ccc']) + self.assertEqual(idx, 1) + + def test_find_only_one_key_0(self): + lines = ['aaa bbb\n', 'bbb aaa\n', 'ccc ddd\n'] + with self.assertRaises(RuntimeError): + idx = find_only_one_key(lines, ['ccc','eee']) + + def test_find_only_one_key_2(self): + lines = ['aaa bbb\n', 'bbb ccc\n', 'bbb ccc\n', 'fff eee\n'] + with self.assertRaises(RuntimeError): + idx = find_only_one_key(lines, ['bbb','ccc']) + + def test_revise_lmp_input_model_0(self): + lines = ['foo\n', 'pair_style deepmd aaa ccc fff\n', 'bar\n', '\n'] + ref_lines = copy.deepcopy(lines) + lines = revise_lmp_input_model(lines, ['model0', 'model1'], 10, '0.1') + for ii in [0, 2, 3] : + self.assertEqual(lines[ii], ref_lines[ii]) + tmp = " ".join(lines[1].split()) + self.assertEqual(tmp, "pair_style deepmd model0 model1 10 model_devi.out") + + def test_revise_lmp_input_model_1(self): + lines = ['foo\n', 'pair_style deepmd aaa ccc fff\n', 'bar\n', '\n'] + ref_lines = copy.deepcopy(lines) + lines = revise_lmp_input_model(lines, ['model0', 'model1'], 10, '1') + for ii in [0, 2, 3] : + self.assertEqual(lines[ii], ref_lines[ii]) + tmp = " ".join(lines[1].split()) + self.assertEqual(tmp, "pair_style deepmd model0 model1 out_freq 10 out_file model_devi.out") + + def test_revise_lmp_input_dump(self): + lines = ['foo\n', 'dump dpgen_dump ccc fff\n', 'bar\n', '\n'] + ref_lines = copy.deepcopy(lines) + lines = revise_lmp_input_dump(lines, 10) + for ii in [0, 2, 3] : + self.assertEqual(lines[ii], ref_lines[ii]) + tmp = " ".join(lines[1].split()) + self.assertEqual(tmp, "dump dpgen_dump all custom 10 traj/*.lammpstrj id type x y z") + + def test_revise_lmp_input_plm(self): + lines = ['foo\n', 'fix dpgen_plm ccc fff\n', 'bar\n', '\n'] + ref_lines = copy.deepcopy(lines) + lines = revise_lmp_input_plm(lines, 'input.plumed') + for ii in [0, 2, 3] : + self.assertEqual(lines[ii], ref_lines[ii]) + tmp = " ".join(lines[1].split()) + self.assertEqual(tmp, "fix dpgen_plm all plumed plumedfile input.plumed outfile output.plumed") + + def test_revise_by_key(self): + lines = ['foo\n', 'aaa\n', 'bar\n', 'bbb\n', '\n'] + ref_lines = copy.deepcopy(lines) + lines = revise_by_keys(lines, ['aaa', 'bbb'], ['ccc','ddd']) + for ii in [0, 2, 4] : + self.assertEqual(lines[ii], ref_lines[ii]) + tmp = " ".join(lines[1].split()) + self.assertEqual(tmp, "ccc") + tmp = " ".join(lines[3].split()) + self.assertEqual(tmp, "ddd") + if __name__ == '__main__': unittest.main() diff --git a/tests/generator/test_nbands_esti.py b/tests/generator/test_nbands_esti.py index 3e1c39756..b5edf447b 100644 --- a/tests/generator/test_nbands_esti.py +++ b/tests/generator/test_nbands_esti.py @@ -71,5 +71,9 @@ def test_get_res(self): 'ele_temp': 20000.0, 'nbands': 81 } - self.assertEqual(res, ref) - + self.assertEqual(res['natoms'], ref['natoms']) + self.assertAlmostEqual(res['vol'], ref['vol']) + self.assertAlmostEqual(res['nvalence'][0], ref['nvalence'][0]) + self.assertEqual(len(res['nvalence']), len(ref['nvalence'])) + self.assertAlmostEqual(res['ele_temp'], ref['ele_temp'], places = 1) + self.assertEqual(res['nbands'], ref['nbands'])