Upgrade to be compatible with pyfirecreset v2.6.0

.firecrest-demo-config.json

@@ -1,8 +1,11 @@
-{
-    "url": "http://localhost:8000/",
-    "token_uri": "http://localhost:8080/auth/realms/kcrealm/protocol/openid-connect/token",
-    "client_id": "firecrest-sample",
-    "client_secret": "b391e177-fa50-4987-beaf-e6d33ca93571",
-    "machine": "cluster",
-    "scratch_path": "/home/service-account-firecrest-sample"
+ {
+    "url": "",
+    "token_uri": "",
+    "client_id": "",
+    "client_secret": "",
+    "compute_resource": "",
+    "temp_directory": "",
+    "small_file_size_mb": 5.0,
+    "workdir": "",
+    "api_version": "1.16.0"
 }

.github/workflows/tests.yml

@@ -49,13 +49,14 @@ jobs:
         python -m pip install --upgrade pip
         pip install -e .[dev]
     - name: Test with pytest
-      run: pytest -vv --firecrest-requests --cov=aiida_firecrest --cov-report=xml --cov-report=term
-
-    - name: Upload coverage reports to Codecov
-      uses: codecov/codecov-action@v3
-      with:
-        name: aiida-firecrest-pytests
-        flags: pytests
-        file: ./coverage.xml
-        fail_ci_if_error: true
-        token: ${{ secrets.CODECOV_TOKEN }}
+      run: pytest -vv  --cov=aiida_firecrest --cov-report=xml --cov-report=term
+
+    # Codecov failing, we need to fix token: https://github.com/aiidateam/aiida-firecrest/issues/38
+    # - name: Upload coverage reports to Codecov
+    #   uses: codecov/codecov-action@v3
+    #   with:
+    #     name: aiida-firecrest-pytests
+    #     flags: pytests
+    #     file: ./coverage.xml
+    #     fail_ci_if_error: true
+    #     token: ${{ secrets.CODECOV_TOKEN }}

.gitignore

@@ -132,3 +132,6 @@ dmypy.json
 .vscode/
 .demo-server/
 _archive/
+
+
+.firecrest-config.json

.pre-commit-config.yaml

@@ -44,5 +44,5 @@ repos:
       additional_dependencies:
       - "types-PyYAML"
       - "types-requests"
-      - "pyfirecrest~=1.4.0"
-      - "aiida-core~=2.4.0"
+      - "pyfirecrest>=2.6.0"
+      - "aiida-core>=2.6.0"  # please change to 2.6.2 when released Issue #45

CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+## v0.2.0 - (not released yet)
+
+### Transport plugin
+- Refactor `put` & `get` & `copy` now they mimic behavior `aiida-ssh` transport plugin.
+- `put` & `get` & `copy` now support glob patterns.
+- Added `dereference` option wherever relevant
+- Added `recursive` functionality for `listdir`
+- Added `_create_secret_file` to store user secret locally in `~/.firecrest/`
+- Added `_validate_temp_directory` to allocate a temporary directory useful for `extract` and `compress` methods on FirecREST server.
+- Added `_dynamic_info_direct_size` this is able to get info of direct transfer from the server rather than asking from users. Raise if fails to make a connection.
+- Added `_validate_checksum` to check integrity of downloaded/uploaded files.
+- Added `_gettreetar` & `_puttreetar` to transfer directories as tar files internally.
+- Added `payoff` function to calculate when is gainful to transfer as zip, and when to transfer individually.
+
+### Scheduler plugin
+- `get_job` now supports for pagination for retrieving active jobs
+- `get_job` is parsing more data than before: `submission_time`, `wallclock_time_seconds`, `start_time`, `time_left`, `nodelist`. see open issues [39](https://github.com/aiidateam/aiida-firecrest/issues/39) & [40](https://github.com/aiidateam/aiida-firecrest/issues/40)
+- bug fix: `get_job` won't raise if the job cannot be find (completed/error/etc.)
+- `_convert_time` and `_parse_time_string` copied over from `slurm-plugin` see [open issue](https://github.com/aiidateam/aiida-firecrest/issues/42)
+
+### Tests
+
+- The testing utils responsible for mocking the FirecREST server (specifically FirecrestMockServer) have been have been replaced with utils monkeypatching pyfirecrest. The FirecREST mocking utils introduced a maintenance overhead that is not in the responsibility of this repository. We still continue to support running with a real server and plan to continue running the tests with the [demo docker image](https://github.com/eth-cscs/firecrest/tree/master/deploy/demo) offered by CSCS.
+The downside was debugging wasn't easy at all. Not always obvious which of the three is causing a bug.
+Because of this, a new set of tests only verify the functionality of `aiida-firecrest` by directly mocking PyFirecREST. Maintaining this set in `tests/` is simpler because we just need to monitor the return values of PyFirecREST​. While maintaining the former is more difficult as you have to keep up with both FirecREST and PyFirecREST.
+
+
+### Miscellaneous
+
+- class `FcPath` is removed from interface here, as it has [merged](https://github.com/eth-cscs/pyfirecrest/pull/43) into `pyfirecrest`
+
 ## v0.1.0 (December 2021)
 
 Initial release.

README.md

@@ -5,9 +5,8 @@
 
 AiiDA Transport/Scheduler plugins for interfacing with [FirecREST](https://products.cscs.ch/firecrest/), via [pyfirecrest](https://github.com/eth-cscs/pyfirecrest).
 
-It is currently tested against [FirecREST v1.13.0](https://github.com/eth-cscs/firecrest/releases/tag/v1.13.0).
+It is currently tested against [FirecREST v2.6.0](https://github.com/eth-cscs/pyfirecrest/tree/v2.6.0).
 
-**NOTE:** This plugin is currently dependent on a fork of `aiida-core` from [PR #6043](https://github.com/aiidateam/aiida-core/pull/6043)
 
 ## Usage
 
@@ -65,11 +64,12 @@ $ verdi -p quicksetup computer configure firecrest firecrest-client
 Report: enter ? for help.
 Report: enter ! to ignore the default and set no value.
 Server URL: https://firecrest.cscs.ch
-Token URI: https://auth.cscs.ch/auth/realms/cscs/protocol/openid-connect/token
+Token URI: https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token
 Client ID: username-client
 Client Secret: xyz
 Client Machine: daint
 Maximum file size for direct transfer (MB) [5.0]:
+Temp directory on server: /scratch/something/ # "A temp directory on user's space on the server for creating temporary files (compression, extraction, etc.)"
 Report: Configuring computer firecrest-client for user [email protected].
 Success: firecrest-client successfully configured for [email protected]
 ```
@@ -100,15 +100,9 @@ See [tests/test_calculation.py](tests/test_calculation.py) for a working example
 
 ### Current Issues
 
-Simple calculations are now running successfully [in the tests](tests/test_calculation.py), however, there are still some critical issues, before this could be production ready:
+Calculations are now running successfully, however, there are still issues regarding efficiency, Could be improved:
 
-1. Currently uploading via firecrest changes `_aiidasubmit.sh` to `aiidasubmit.sh` 😱 ([see #191](https://github.com/eth-cscs/firecrest/issues/191)), so `metadata.options.submit_script_filename` should be set to this.
-
-2. Handling of large (>5Mb) file uploads/downloads needs to be improved
-
-3. Handling of the client secret, which should likely not be stored in the database
-
-4. Monitoring / management of API request rates could to be improved
+1. Monitoring / management of API request rates could to be improved. Currently this is left up to PyFirecREST.
 
 ## Development
 
@@ -128,6 +122,16 @@ pre-commit run --all-files
 
 ### Testing
 
+There are two types of tests: mocking the PyFirecREST or the FirecREST server.
+While the latter is a good practice to ensure that all three (`aiida-firecrest`, FirecREST, and PyFirecREST) work flawlessly, debugging may not always be easy because it may not always be obvious which of the three is causing a bug.
+Because of this, we have another set of tests that only verify the functionality of `aiida-firecrest` by directly mocking PyFirecREST. Maintaining the second set in `tests/tests_mocking_pyfirecrest/` is simpler because we just need to monitor the return values of PyFirecREST​. While maintaining the former is more difficult as you have to keep up with both FirecREST and PyFirecREST.
+
+
+#### Mocking FirecREST server
+
+These tests were successful against [FirecREST v1.13.0](https://github.com/eth-cscs/firecrest/releases/tag/v1.13.0).
+For newer version please refer to tests Mocking PyFirecREST
+
 It is recommended to run the tests via [tox](https://tox.readthedocs.io/en/latest/).
 
 ```bash
@@ -174,7 +178,7 @@ you can use the `--firecrest-requests` option:
 tox -- --firecrest-requests
 ```
 
-### Notes on using the demo server on MacOS
+##### Notes on using the demo server on MacOS
 
 A few issues have been noted when using the demo server on MacOS (non-Mx):
 
@@ -195,3 +199,18 @@ although it is of note that you can find these files directly where you your `fi
 [codecov-link]: https://codecov.io/gh/aiidateam/aiida-firecrest
 [black-badge]: https://img.shields.io/badge/code%20style-black-000000.svg
 [black-link]: https://github.com/ambv/black
+
+
+
+#### Mocking PyFirecREST
+
+These set of test do not gurantee that the firecrest protocol is working, but it's very useful to quickly check if `aiida-firecrest` is behaving as it's expected to do. To run just simply use `pytest`.
+
+
+If these tests, pass and still you have trouble in real deployment, that means your installed version of pyfirecrest is behaving differently from what `aiida-firecrest` expects in `MockFirecrest` in `tests/tests_mocking_pyfirecrest/conftest.py`.
+If there is no version of `aiida-firecrest` available that supports your `pyfirecrest` version and if down/upgrading your `pyfirecrest` to a supported version is not an option, you might try the following:
+- open an issue on the `aiida-firecrest` repository on GitHub to request supporting your version of pyfirecrest
+- if you feel up to finding the discrepancy and fixing it within `aiida-firecrest`, open a PR instead
+- if you think the problem is a bug in `pyfirecrest`, open an issue there
+
+Either way, make sure to report which version of `aiida-firecrest` and `pyfirecrest` you are using.