Merge pull request #22 from opensciencegrid/master

merging from above

projects/CaseWestern_Zhang.yaml

@@ -0,0 +1,9 @@
+Description: A project involving neuroimaging and genetics data for neurodegenerative disease. 
+Department: Dept. of Population & Quantitative Health Sciences (PQHS)
+FieldOfScience: Biological and Biomedical Sciences
+Organization: Case Western Reserve University
+PIName: Lijun Zhang
+
+Sponsor:
+  CampusGrid:
+    Name: OSG Connect

projects/FSU_RCC.yaml

@@ -0,0 +1,11 @@
+Description: The Research Computing Center at Florida State University
+  enables research and education by maintaining a diverse campus
+  cyberinfrastructure 
+Department: Research Computing Center
+FieldOfScience: Research Computing
+Organization: Florida State University
+PIName: Paul van der Mark
+
+Sponsor:
+  CampusGrid:
+    Name: OSG Connect

projects/MontgomeryCollege_Dillman.yaml

@@ -0,0 +1,12 @@
+Description: >-
+  workforce training, 2-5 day workshops every couple of months to get
+  bench biologists comfortable with various bioinformatics pipelines
+  on the command line.
+Department: Workforce Development & Continuing Education
+FieldOfScience: Biological and Biomedical Sciences
+Organization: Montgomery College
+PIName: Allissa Dillman
+
+Sponsor:
+  CampusGrid:
+    Name: OSG Connect

projects/PSU_Anandakrishnan.yaml

@@ -0,0 +1,12 @@
+Description: >-
+  Penn State Ice and Climate Exploration is an interdisciplinary group
+  of researchers from across the university dedicated to a better
+  understanding of the cryosphere.
+Department: Geosciences
+FieldOfScience: Geological and Earth Sciences
+Organization: Pennsylvania State University
+PIName: Sridhar Anandakrishnan
+
+Sponsor:
+  CampusGrid:
+    Name: OSG Connect

projects/UCBerkeley_Zaletel.yaml

@@ -0,0 +1,14 @@
+Description: >-
+  Tensor networks provide an efficient approximation to quantum
+  many-body wavefunctions and a controllable method to simulate
+  quantum computing on classical hardware. We apply these techniques
+  to problems in condensed matter physics and quantum error
+  correction.
+Department: Physics
+FieldOfScience: Condensed Matter Physics
+Organization: University of California, Berkeley
+PIName: Mike Zaletel
+
+Sponsor:
+  CampusGrid:
+    Name: OSG Connect

projects/USD_PHYS733.yaml

@@ -0,0 +1,9 @@
+Description: A course on Elementary Particle and Nuclear Physics at the University of South Dakota
+Department: Physics 
+FieldOfScience: Elementary Particles
+Organization: University of South Dakota
+PIName: Jing Liu
+
+Sponsor:
+  CampusGrid:
+    Name: OSG Connect

projects/UTK_Luettgau.yaml

@@ -0,0 +1,11 @@
+Description: >-
+  Piloting the National Science Data Fabric, A Platform Agnostic
+  Testbed for Democratizing Data Delivery
+Department: Electrical Engineering & Computer Science
+FieldOfScience: Computer and Information Science and Engineering
+Organization: University of Tennessee, Knoxville
+PIName: Jakob Luettgau
+
+Sponsor:
+  CampusGrid:
+    Name: OSG Connect

src/README.md

@@ -279,9 +279,11 @@ There are three kinds of authorization types:
 
       - SciTokens:
           Issuer: https://chtc.cs.wisc.edu
-          Base Path: /chtc
-          Restricted Path: /PROTECTED/matyas,/PROTECTED/bbockelm
-          Map Subject: True
+          BasePath: /chtc
+          RestrictedPath: /PROTECTED/matyas,/PROTECTED/bbockelm
+          MapSubject: True
+
+  (for backwards compat, `Base Path`, `Restricted Path`, and `Map Subject` are also accepted)
 
   This results in an issuer block that looks like
 
@@ -293,8 +295,8 @@ There are three kinds of authorization types:
 
   See [the XrdSciTokens readme](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens#readme) for a reference of what these mean.
 
-  `Restricted Path` is optional (and rarely set); it is omitted if not specified. 
-  `Map Subject` is optional and defaults to `false` if not specified.
+  `RestrictedPath` is optional (and rarely set); it is omitted if not specified. 
+  `MapSubject` is optional and defaults to `false` if not specified.
   It is only used in scitokens.cfg for the origin.
 
 ```yaml
@@ -333,6 +335,21 @@ DirList: https://<HOST>:<PORT>
 DirList is the HTTPS URL of an XRootD service that can be used to get a directory listing.
 DirList is optional.
 
+```yaml
+CredentialGeneration:
+  Strategy: "Vault" or "OAuth2"
+  Issuer: "<ISSUER URL>"
+  MaxScopeDepth: <INTEGER>
+  VaultServer: "<HOST>:<PORT>"
+```
+CredentialGeneration is an optional block of information about how clients can obtain credentials for the namespace.
+If specified:
+- Strategy must be `OAuth2` or `Vault`, depending on whether OAuth2 or a Hashicorp Vault server is being used
+- Issuer is a token issuer URL
+- MaxScopeDepth (optional) is the maximum number of path components a token's scope field may have;
+  note that scopes are relative to the BasePath.
+  If missing, assumed to be 0, i.e. the scope is always `/`.
+- VaultServer is the endpoint for the Hashicorp Vault server used with the Vault strategy 
 
 ### Contents of a cache or origin in resource data
 
@@ -505,6 +522,14 @@ The JSON also contains an attribute `namespaces` that is a list of namespaces wi
 - `usetokenonread` is `true` if the namespace has a SciTokens entry in its Authorizations list and `false` otherwise
 - `caches` is a list of caches that support the namespace;
   each cache in the list contains the `endpoint`, `auth_endpoint`, and `resource` attributes as in the `caches` list above
+- `credential_generation` is information about how to generate credentials that can access the namespace.
+  If not null, it has:
+  - `strategy`: either `OAuth2` or `Vault`
+  - `issuer`: the token issuer for the credentials
+  - `max_scope_depth`: integer; the max number of levels you can get a credential to be scoped for;
+    "0" means that the scope will always be `/`.
+    Note that scopes are usually relative to the namespace path.
+  - `vault_server`: the Vault server for the `Vault` strategy or null
 
 The final result looks like
 ```json
@@ -530,6 +555,7 @@ The final result looks like
           "resource": "RDS_AUTH_OSDF_CACHE"
         }
       ],
+      "credential_generation": null,
       "dirlisthost": null,
       "path": "/xenon/PROTECTED",
       "readhttps": true,
@@ -540,6 +566,11 @@ The final result looks like
       "caches": [
         (a whole bunch)
       ],
+      "credential_generation": {
+        "issuer": "https://osg-htc.org/ospool",
+        "max_scope_depth": 4,
+        "strategy": "OAuth2"
+      },
       "dirlisthost": "https://origin-auth2001.chtc.wisc.edu:1095",
       "path": "/ospool/PROTECTED",
       "readhttps": true,

src/schema/rgsummary.xsd

@@ -151,6 +151,7 @@
                                 <xsd:element name="StorageCapacityMax" type="xsd:string"/>
                                 <xsd:element name="HEPSPEC" type="xsd:string" />
                                 <xsd:element minOccurs="0" name="APELNormalFactor" type="xsd:string" />
+                                <xsd:element minOccurs="0" name="HEPScore23Percentage" type="xsd:string" />
                                 <xsd:element name="TapeCapacity" type="xsd:string"/>
                               </xsd:sequence>
                             </xsd:complexType>

src/stashcache.py

@@ -1,18 +1,19 @@
 from collections import defaultdict
 from typing import Dict, List, Optional
 
-from webapp.common import is_null, readfile, PreJSON, XROOTD_CACHE_SERVER, XROOTD_ORIGIN_SERVER
+from webapp.common import is_null, PreJSON, XROOTD_CACHE_SERVER, XROOTD_ORIGIN_SERVER
 from webapp.exceptions import DataError, ResourceNotRegistered, ResourceMissingService
-from webapp.ldap_data import get_ligo_ldap_dn_list
 from webapp.models import GlobalData
 from webapp.topology import Resource, ResourceGroup, Topology
-from webapp.vos_data import AuthMethod, DNAuth, SciTokenAuth, Namespace, \
-    parse_authz, ANY, ANY_PUBLIC, VOsData
+from webapp.vos_data import VOsData
+from webapp.data_federation import AuthMethod, DNAuth, SciTokenAuth, Namespace, parse_authz
 
 import logging
 
 log = logging.getLogger(__name__)
 
+ANY = "ANY"
+ANY_PUBLIC = "ANY_PUBLIC"
 
 def _log_or_raise(suppress_errors: bool, an_exception: BaseException, logmethod=log.debug):
     if suppress_errors:
@@ -512,6 +513,19 @@ def generate_origin_scitokens(global_data: GlobalData, fqdn: str, suppress_error
     return template.format(**locals()).rstrip() + "\n"
 
 
+def get_credential_generation_dict_for_namespace(ns: Namespace) -> Optional[Dict]:
+    if not ns.credential_generation:
+        return None
+    cg = ns.credential_generation
+    info = {
+        "strategy": cg.strategy,
+        "issuer": cg.issuer,
+        "max_scope_depth": cg.max_scope_depth or 0,
+        "vault_server": cg.vault_server or None
+    }
+    return info
+
+
 def get_namespaces_info(global_data: GlobalData) -> PreJSON:
     """Return data for the /stashcache/namespaces JSON endpoint.
 
@@ -540,6 +554,7 @@ def _namespace_dict(ns: Namespace):
             "writebackhost": ns.writeback,
             "dirlisthost": ns.dirlist,
             "caches": [],
+            "credential_generation": get_credential_generation_dict_for_namespace(ns),
         }
 
         for cache_name, cache_resource_obj in cache_resource_objs.items():

src/tests/test_api.py

@@ -1,6 +1,7 @@
 import re
 import flask
 import pytest
+import urllib.parse
 from pytest_mock import MockerFixture
 
 # Rewrites the path so the app can be imported like it normally is
@@ -12,6 +13,7 @@
 
 from app import app, global_data
 from webapp.topology import Facility, Site, Resource, ResourceGroup
+from webapp.data_federation import CredentialGeneration
 
 HOST_PORT_RE = re.compile(r"[a-zA-Z0-9.-]{3,63}:[0-9]{2,5}")
 PROTOCOL_HOST_PORT_RE = re.compile(r"[a-z]+://" + HOST_PORT_RE.pattern)
@@ -195,6 +197,15 @@ def validate_namespace_schema(ns):
             assert isinstance(ns["usetokenonread"], bool)
             assert ns["dirlisthost"] is None or PROTOCOL_HOST_PORT_RE.match(ns["dirlisthost"])
             assert ns["writebackhost"] is None or PROTOCOL_HOST_PORT_RE.match(ns["writebackhost"])
+            credgen = ns["credential_generation"]
+            if credgen is not None:
+                assert isinstance(credgen["max_scope_depth"], int) and credgen["max_scope_depth"] > -1
+                assert credgen["strategy"] in CredentialGeneration.STRATEGIES
+                assert credgen["issuer"]
+                parsed_issuer = urllib.parse.urlparse(credgen["issuer"])
+                assert parsed_issuer.netloc and parsed_issuer.scheme == "https"
+                if credgen["vault_server"]:
+                    assert isinstance(credgen["vault_server"], str)
 
         response = client.get('/stashcache/namespaces')
         assert response.status_code == 200
@@ -212,11 +223,15 @@ def validate_namespace_schema(ns):
         # Have a reasonable number of namespaces
         assert len(namespaces) > 15
 
+        found_credgen = False
         for namespace in namespaces:
+            if namespace["credential_generation"] is not None:
+                found_credgen = True
             validate_namespace_schema(namespace)
             if namespace["caches"]:
                 for cache in namespace["caches"]:
                     validate_cache_schema(cache)
+        assert found_credgen, "At least one namespace with credential_generation"
 
     def test_origin_grid_mapfile(self, client: flask.Flask):
         TEST_ORIGIN = "origin-auth2001.chtc.wisc.edu"  # This origin serves protected data

src/webapp/common.py

@@ -70,7 +70,7 @@ def is_null(x, *keys) -> bool:
                      ])
 
 
-def ensure_list(x: Union[None, T, List[T]]) -> List[T]:
+def ensure_list(x: Union[None, List[T], T]) -> List[T]:
     if isinstance(x, list):
         return x
     elif x is None: