Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Organising your data (aka. groups) #364

Merged
Merged
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 30 additions & 57 deletions docs/sections/data/groups.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,14 @@
# Organising your data

In this section of the tutorial we will focus on how to organise and explore the data in an AiiDA database.
As in (**TODO FIX LINK**), we will be using the previously created database entries for this tutorial.
To follow the tutorial then, you can use the profile in to which you have previously imported this data, or you may wish to create a fresh profile and import the archive into that:
You will be using the previously created database entries for this tutorial.
CasperWA marked this conversation as resolved.
Show resolved Hide resolved
To follow the tutorial, you can use the profile into which you have previously imported this data, or you may wish to create a fresh profile and import the archive into that.
CasperWA marked this conversation as resolved.
Show resolved Hide resolved
To do so, you can copy-paste the following code into a terminal after activating your AiiDA virtual environment:

```{code-block} console

$ verdi quicksetup --profile data
$ verdi profile setdefault data
$ verdi import https://object.cscs.ch/v1/AUTH_b1d80408b3d340db9f03d373bbde5c1e/marvel-vms/tutorials/aiida_tutorial_2020_07_perovskites_v0.9.aiida

verdi quicksetup --profile data
verdi profile setdefault data
verdi import https://object.cscs.ch/v1/AUTH_b1d80408b3d340db9f03d373bbde5c1e/marvel-vms/tutorials/aiida_tutorial_2020_07_perovskites_v0.9.aiida
CasperWA marked this conversation as resolved.
Show resolved Hide resolved
```

## How to group nodes
Expand All @@ -26,7 +25,6 @@ A typical use case is to store all nodes that share a common property in a singl
Lets explore the groups already present in the imported archive:
CasperWA marked this conversation as resolved.
Show resolved Hide resolved

```{code-block} console

$ verdi group list -a -A
PK Label Type string User
---- --------------- ------------- ---------------
Expand All @@ -37,28 +35,26 @@ PK Label Type string User
5 GBRV_pbesol core.upf aiida@localhost
6 GBRV_lda core.upf aiida@localhost
7 20200705-071658 core.import aiida@localhost

```

The default table shows us four pieces of information:

* **PK**: The Primary Key of the group
* **Label**: The label by which the group can be identified
* **Type string**: This tells us what "sub-class" of group this is.
* **PK**: The Primary Key of the group.
* **Label**: The label by which the group can be identified.
* **Type string**: This tells us what type of group this is.
Type strings can be used to class certain types of data, for example here we have general groups (`core`), groups containing pseudopotentials (`core.upf`), and an auto-generated group containing the nodes we imported from the archive (`core.import`).
mbercx marked this conversation as resolved.
Show resolved Hide resolved
For advanced use, you can create your own group subclass plugins, with specialised methods.
For advanced use, you can create your own group type plugins, with specialised methods by sub-classing the general `Group` class.
* **User**: The email of the user that created this group.

:::{tip}

The `-a` and `-A` flags we used above ensure that groups for all type strings and users are shown respectively.
The `-a` and `-A` flags used above ensure that groups for *all* type strings and users are shown, respectively.

:::

We can then inspect the content of a group by its label (if it is unique) or the PK:
You can then inspect the content of a group by its label (if it is unique) or the PK:

```{code-block} console

$ verdi group show tutorial_pbesol
----------------- ----------------
Group label tutorial_pbesol
Expand All @@ -71,54 +67,44 @@ PK Type Created
380 CalcJobNode 2078D:17h:46m ago
1273 CalcJobNode 2078D:18h:03m ago
...

```

Conversely, if you want to see all the groups a node belongs to, you can run:
Conversely, if you want to see all the groups a node belongs to, you can use its PK and run:

```{code-block} console

$ verdi group list -a -A --node 380
CasperWA marked this conversation as resolved.
Show resolved Hide resolved
PK Label Type string User
---- --------------- ------------- ---------------
1 tutorial_pbesol core aiida@localhost
7 20200705-071658 core.import aiida@localhost

```

### Creating and manipulating groups

Lets make a new group:
CasperWA marked this conversation as resolved.
Show resolved Hide resolved

```{code-block} console

$ verdi group create a_group
Success: Group created with PK = 8 and name 'a_group'

```

If we want to change the name of the group at any time:
If you want to change the name of the group at any time try:
CasperWA marked this conversation as resolved.
Show resolved Hide resolved

```{code-block} console

$ verdi group relabel a_group my_group
Success: Label changed to my_group

```

Now we can add one or more nodes to it:
Now you can add one or more nodes to it by listing any number of node PKs:
CasperWA marked this conversation as resolved.
Show resolved Hide resolved

```{code-block} console

$ verdi group add-nodes -G my_group 380 1273
Do you really want to add 2 nodes to Group<my_group>? [y/N]: y

```

We can also copy the nodes from an existing group to another group:
You can also copy the nodes from an existing group to another group:
CasperWA marked this conversation as resolved.
Show resolved Hide resolved

```{code-block} console

$ verdi group copy tutorial_pbesol my_group
Warning: Destination group<my_group> already exists and is not empty.
Do you wish to continue anyway? [y/N]: y
Expand All @@ -135,31 +121,26 @@ PK Type Created
74 CalcJobNode 2078D:17h:51m ago
76 CalcJobNode 2078D:17h:57m ago
...

```

To remove nodes from the group run:
To remove nodes from the group is similar to adding them, try:
CasperWA marked this conversation as resolved.
Show resolved Hide resolved

```{code-block} console

$ verdi group remove-nodes -G my_group 74
Do you really want to remove 1 nodes from Group<my_group>? [y/N]: y

```

and finally to remove the group entirely:

```{code-block} console

$ verdi group delete --clear my_group
Are you sure to delete Group<my_group>? [y/N]: y
Success: Group<my_group> deleted.

```

:::{important}

Any deletion operation related to groups won't affect the nodes themselves.
Any deletion operation related to groups will not affect the nodes themselves.
For example if you delete a group, the nodes that belonged to the group will remain in the database.
The same happens if you remove nodes from the group -- they will remain in the database but won't belong to the group any more.

Expand All @@ -168,65 +149,57 @@ The same happens if you remove nodes from the group -- they will remain in the d
## Organising groups in hierarchies

Earlier we mentioned that groups are like files in folders on your filesystem.
As with folders and sub-folders then, as the amount of groups we have grows, we may also wish to structure our groups in a hierarchy.
Groups in AiiDA are inherently "flat", in that groups may only contain nodes and not other groups.
However it is possible to construct *virtual* group hierarchies based on delimited group labels, using the `grouppath` utility.
Then as with folders and sub-folders, as the amount of groups you have grows, you may also wish to structure your groups in a hierarchy.
CasperWA marked this conversation as resolved.
Show resolved Hide resolved
Groups in AiiDA are inherently "flat", meaning groups may only contain nodes and not other groups.
However, it is possible to construct *virtual* group hierarchies based on delimited group labels, using the `grouppath` utility.

Like folder paths grouppath requires delimitation by `/` (forward slash) characters.
Like folder paths on Unix systems `grouppath` requires delimitation by forward slash (`/`) characters.
CasperWA marked this conversation as resolved.
Show resolved Hide resolved
Lets copy and rename the three tutorial groups:
CasperWA marked this conversation as resolved.
Show resolved Hide resolved

```{code-block} console

$ verdi group copy tutorial_lda tutorial/lda/basic
$ verdi group copy tutorial_pbe tutorial/gga/pbe
$ verdi group copy tutorial_pbesol tutorial/gga/pbesol

verdi group copy tutorial_lda tutorial/lda/basic
verdi group copy tutorial_pbe tutorial/gga/pbe
verdi group copy tutorial_pbesol tutorial/gga/pbesol
```

We can now list the groups in a new way:
You can now list the groups in a new way:

```{code-block} console

$ verdi group path ls -l
Path Sub-Groups
--------------- ------------
tutorial 3
tutorial_lda 0
tutorial_pbe 0
tutorial_pbesol 0

```

:::{note}

In the terminal, paths that contain nodes are listed in bold
In the terminal, paths that contain nodes are listed in bold.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true? I don't seem to notice this. 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, i'll double check it during my pass, thanks!


:::

You can see that the actual groups that we create do not show, only the initial part of the "path", and how many sub-groups that path contains.
We can then step into a path:
You can see that the actual groups that are created do not appear, only the initial part of the "path", and how many sub-groups that path contains.
CasperWA marked this conversation as resolved.
Show resolved Hide resolved
You can then step into a path:

```{code-block} console

$ verdi group path ls -l tutorial
Path Sub-Groups
------------ ------------
tutorial/gga 2
tutorial/lda 1

```

This feature is also particularly useful in the verdi shell:
This feature is also particularly useful in the `verdi shell`:

```{code-block} ipython

In [1]: from aiida.tools.groups import GroupPath
In [2]: for subpath in GroupPath("tutorial/gga").walk(return_virtual=False):
...: print(subpath.get_group())
...:
"tutorial/gga/pbesol" [type core], of user aiida@localhost
"tutorial/gga/pbe" [type core], of user aiida@localhost

```

:::{seealso}
Expand Down