Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Function.get_jobs() #1406

Merged
merged 38 commits into from
Jul 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
70f9548
add Function.get_jobs()
akihikokuroda Jul 11, 2024
ea484fc
review comments
akihikokuroda Jul 17, 2024
dbb3103
add tests
akihikokuroda Jul 22, 2024
0d1393f
add tests
akihikokuroda Jul 22, 2024
74f2206
add tests
akihikokuroda Jul 22, 2024
954180f
add test
akihikokuroda Jul 22, 2024
c5a811a
add tests
akihikokuroda Jul 22, 2024
3ee36b1
add tests
akihikokuroda Jul 22, 2024
86bd2bb
add tests
akihikokuroda Jul 22, 2024
21ef6fb
add tests
akihikokuroda Jul 22, 2024
2ba45cb
add tests
akihikokuroda Jul 22, 2024
5dcca98
add tests
akihikokuroda Jul 22, 2024
2483d8c
add tests
akihikokuroda Jul 22, 2024
82ca936
add tests
akihikokuroda Jul 22, 2024
bd1bf0d
add tests
akihikokuroda Jul 22, 2024
3e58162
add tests
akihikokuroda Jul 22, 2024
e968082
add tests
akihikokuroda Jul 22, 2024
4ff9f33
add tests
akihikokuroda Jul 22, 2024
c1b9a47
add tests
akihikokuroda Jul 22, 2024
adf2025
add tests
akihikokuroda Jul 22, 2024
7c28375
review comments
akihikokuroda Jul 22, 2024
d2bf6a6
lint
akihikokuroda Jul 22, 2024
fee3673
review comments
akihikokuroda Jul 22, 2024
d49f9af
review comments
akihikokuroda Jul 22, 2024
0414be4
review comments
akihikokuroda Jul 22, 2024
c617fbe
review comments
akihikokuroda Jul 22, 2024
882c205
review comments
akihikokuroda Jul 22, 2024
70bed15
review comments
akihikokuroda Jul 23, 2024
bbf8446
review comments
akihikokuroda Jul 23, 2024
746d622
review comments
akihikokuroda Jul 23, 2024
6d247e6
review comments
akihikokuroda Jul 23, 2024
c7828bb
review comments
akihikokuroda Jul 23, 2024
6b39e30
review comments
akihikokuroda Jul 23, 2024
0a91733
review comments
akihikokuroda Jul 23, 2024
1b75de1
review comments
akihikokuroda Jul 23, 2024
6364ddb
review comments
akihikokuroda Jul 23, 2024
311bc42
review comments
akihikokuroda Jul 23, 2024
cd2d16c
Merge branch 'main' into functionjobs
akihikokuroda Jul 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/docker-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ jobs:
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- name: Build the function
run: docker build -t test_function:latest --build-arg TARGETARCH="amd64" -f ./tests/basic/function/Sample-Docker ./tests/basic
- name: Build the containers
run: docker compose -f docker-compose-dev.yaml build
- name: Run the jupyter profile
Expand Down
8 changes: 3 additions & 5 deletions .github/workflows/kubernetes-deploy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,6 @@ jobs:
docker build -t ray:test --build-arg TARGETARCH="amd64" -f ./Dockerfile-ray-node .
kind load docker-image ray:test
docker image rm ray:test
- name: Build and load proxy
run: |
docker build -t proxy:test --build-arg TARGETARCH="amd64" -f ./proxy/Dockerfile .
kind load docker-image proxy:test
docker image rm proxy:test
- name: Install helm chart
run: |
cd charts/qiskit-serverless
Expand All @@ -52,6 +47,8 @@ jobs:
--set gateway.application.ray.proxyImage=proxy:test \
--set gateway.application.ray.cpu=1 \
--set gateway.application.limits.keepClusterOnComplete=false \
--set gateway.application.authMockproviderRegistry=test \
--set gateway.application.proxy.enabled=false \
.
GATEWAY=$(kubectl get pod -l app.kubernetes.io/name=gateway -o name)
kubectl wait --for=condition=Ready "$GATEWAY" --timeout 5m
Expand Down Expand Up @@ -81,6 +78,7 @@ jobs:
echo $GATEWAY_HOST
# basic tests
cd /home/runner/work/qiskit-serverless/qiskit-serverless/tests/basic
rm 06_function.py
Tansito marked this conversation as resolved.
Show resolved Hide resolved
for f in *.py; do echo "TEST: $f" && python "$f"; done
# experimental tests
cd /home/runner/work/qiskit-serverless/qiskit-serverless/tests/experimental
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/notebook-local-verify.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ jobs:
run: |
for f in tests/basic/*.py; do sed -i "s/import ServerlessClient/import LocalClient/;s/= ServerlessClient(/= LocalClient(/;/token=os\.environ\.get/d;/host=os\.environ\.get/d" "$f"; done
for f in tests/experimental/*.py; do sed -i "s/import ServerlessClient/import LocalClient/;s/= ServerlessClient(/= LocalClient(/;/token=os\.environ\.get/d;/host=os\.environ\.get/d" "$f"; done
rm tests/basic/06_function.py
- name: install dependencies
shell: bash
run: pip install client/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,8 @@ spec:
value: {{ .Values.application.auth.token.verificationUrl | quote }}
- name: SETTINGS_TOKEN_AUTH_VERIFICATION_FIELD
value: {{ .Values.application.auth.token.verificationField | quote }}
- name: SETTINGS_AUTH_MOCKPROVIDER_REGISTRY
value: {{ .Values.application.authMockproviderRegistry }}
Tansito marked this conversation as resolved.
Show resolved Hide resolved
- name: RAY_CLUSTER_WORKER_REPLICAS
value: {{ .Values.application.ray.replicas | quote }}
- name: RAY_CLUSTER_WORKER_MIN_REPLICAS
Expand Down
3 changes: 2 additions & 1 deletion charts/qiskit-serverless/charts/gateway/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ application:
mechanism: mock_token
token:
mock: awesome_token
authMockproviderRegistry: "icr.io"
superuser:
enable: true
ray:
Expand All @@ -27,7 +28,7 @@ application:
maxReplicas: 4
opensslImage: registry.access.redhat.com/ubi8/openssl:8.8-9
kubectlImage: alpine/k8s:1.29.2@sha256:a51aa37f0a34ff827c7f2f9cb7f6fbb8f0e290fa625341be14c2fcc4b1880f60
proxyImage: "icr.io/quantum-public/qiskit-serverless/proxy:0.9.0"
proxyImage: "icr.io/quantum-public/qiskit-serverless/proxy:0.14.0"
scrapeWithPrometheus: true
openTelemetry: false
openTelemetryCollector:
Expand Down
25 changes: 25 additions & 0 deletions client/qiskit_serverless/core/function.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,31 @@ def run(self, **kwargs):
config=config,
)

def get_jobs(self):
"""Run function

Raises:
QiskitServerlessException: validation exception

Returns:
Job ids : job executed this function
"""
if self.job_client is None:
raise ValueError("No clients specified for a function.")

if self.validate:
is_valid, validation_errors = self._validate_function()
if not is_valid:
error_string = "\n".join(validation_errors)
raise ValueError(
f"Function validation failed. Validation errors:\n {error_string}",
)

return self.job_client.get_jobs(
title=self.title,
provider=self.provider,
)

def _validate_function(self) -> Tuple[bool, List[str]]:
"""Validate function arguments using schema provided.

Expand Down
32 changes: 32 additions & 0 deletions client/qiskit_serverless/core/job.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,10 @@ def get_program(
"""Returns program based on parameters."""
raise NotImplementedError

def get_jobs(self, title: str, provider: Optional[str] = None):
"""Returns job ids of executed program based on parameters."""
raise NotImplementedError


class RayJobClient(BaseJobClient):
"""RayJobClient."""
Expand Down Expand Up @@ -610,6 +614,34 @@ def get_program(
job_client=self,
)

def get_jobs(self, title: str, provider: Optional[str] = None):
"""Returns job ids executed the program based on parameters."""
provider, title = format_provider_name_and_title(
request_provider=provider, title=title
)

tracer = trace.get_tracer("client.tracer")
with tracer.start_as_current_span("program.get_by_title"):
response_data = safe_json_request(
request=lambda: requests.get(
f"{self.host}/api/{self.version}/programs/get_by_title/{title}",
headers={"Authorization": f"Bearer {self._token}"},
params={"provider": provider},
timeout=REQUESTS_TIMEOUT,
)
)
program_id = response_data.get("id", None)
Tansito marked this conversation as resolved.
Show resolved Hide resolved
if not program_id:
return None
response_data = safe_json_request(
request=lambda: requests.get(
f"{self.host}/api/{self.version}/programs/{program_id}/get_jobs/",
headers={"Authorization": f"Bearer {self._token}"},
timeout=REQUESTS_TIMEOUT,
)
)
return response_data


class Job:
"""Job."""
Expand Down
1 change: 1 addition & 0 deletions docker-compose-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ services:
- [email protected]
- SITE_HOST=http://gateway:8000
- SETTINGS_AUTH_MECHANISM=mock_token
- SETTINGS_AUTH_MOCKPROVIDER_REGISTRY=test
Tansito marked this conversation as resolved.
Show resolved Hide resolved
- DATABASE_HOST=postgres
- DATABASE_PORT=5432
- DATABASE_NAME=serverlessdb
Expand Down
1 change: 1 addition & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ services:
- [email protected]
- SITE_HOST=http://gateway:8000
- SETTINGS_AUTH_MECHANISM=mock_token
- SETTINGS_AUTH_MOCKPROVIDER_REGISTRY=test
- DATABASE_HOST=postgres
- DATABASE_PORT=5432
- DATABASE_NAME=serverlessdb
Expand Down
6 changes: 5 additions & 1 deletion gateway/api/authentication.py
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,11 @@ def authenticate(self, request):
group.permissions.add(run_program)
group.user_set.add(user)
logger.info("New group created")
Provider.objects.create(name="mockprovider", admin_group=group)
Provider.objects.create(
name="mockprovider",
admin_group=group,
registry=settings.SETTINGS_AUTH_MOCKPROVIDER_REGISTRY,
)
logger.info("New provider created")

return user, CustomToken(token.encode()) if token else None
1 change: 1 addition & 0 deletions gateway/api/v1/serializers.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ class ProgramSerializer(serializers.ProgramSerializer):

class Meta(serializers.ProgramSerializer.Meta):
fields = [
"id",
"title",
"entrypoint",
"artifact",
Expand Down
29 changes: 29 additions & 0 deletions gateway/api/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,35 @@ def _get_program_queryset_for_title_and_provider(

return result_queryset

@action(methods=["GET"], detail=True)
def get_jobs(
self, request, pk=None
): # pylint: disable=invalid-name,unused-argument
"""Returns jobs of the program."""
tracer = trace.get_tracer("gateway.tracer")
ctx = TraceContextTextMapPropagator().extract(carrier=request.headers)
with tracer.start_as_current_span("gateway.program.get_jobs", context=ctx):
program = Program.objects.filter(id=pk).first()
Tansito marked this conversation as resolved.
Show resolved Hide resolved
Tansito marked this conversation as resolved.
Show resolved Hide resolved
if not program:
return Response(
{"message": f"program [{pk}] was not found."},
status=status.HTTP_404_NOT_FOUND,
)
if (
program.provider
and program.provider.admin_group in request.user.groups.all()
):
jobs = Job.objects.filter(program=program)
else:
jobs = Job.objects.filter(program=program, author=request.user)
return Response(
list(
jobs.values(
"status", "result", "id", "created", "version", "arguments"
)
)
)


class JobViewSet(viewsets.GenericViewSet):
"""
Expand Down
3 changes: 3 additions & 0 deletions gateway/main/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,9 @@
)
# mock token value
SETTINGS_AUTH_MOCK_TOKEN = os.environ.get("SETTINGS_AUTH_MOCK_TOKEN", "awesome_token")
SETTINGS_AUTH_MOCKPROVIDER_REGISTRY = os.environ.get(
"SETTINGS_AUTH_MOCKPROVIDER_REGISTRY", "icr.io"
)
# =============

REST_FRAMEWORK = {
Expand Down
33 changes: 33 additions & 0 deletions gateway/tests/api/test_v1_program.py
Original file line number Diff line number Diff line change
Expand Up @@ -414,3 +414,36 @@ def test_get_by_title(self):
format="json",
)
self.assertEqual(programs_response_do_not_have_access.status_code, 404)

def test_get_jobs(self):
"""Tests run existing authorized."""

user = models.User.objects.get(username="test_user_2")
self.client.force_authenticate(user=user)

# program w/o provider
response = self.client.get(
"/api/v1/programs/1a7947f9-6ae8-4e3d-ac1e-e7d608deec82/get_jobs/",
format="json",
)
self.assertEqual(len(response.data), 1)
self.assertEqual(response.status_code, status.HTTP_200_OK)

# program w/ provider by not author
response = self.client.get(
"/api/v1/programs/6160a2ff-e482-443d-af23-15110b646ae2/get_jobs/",
format="json",
)
self.assertEqual(len(response.data), 2)
self.assertEqual(response.status_code, status.HTTP_200_OK)

# program w/ provider by author
user = models.User.objects.get(username="test_user")
self.client.force_authenticate(user=user)

response = self.client.get(
"/api/v1/programs/6160a2ff-e482-443d-af23-15110b646ae2/get_jobs/",
format="json",
)
self.assertEqual(len(response.data), 1)
self.assertEqual(response.status_code, status.HTTP_200_OK)
13 changes: 12 additions & 1 deletion gateway/tests/fixtures/fixtures.json
Original file line number Diff line number Diff line change
Expand Up @@ -109,10 +109,21 @@
"created": "2023-02-01T15:30:43.281796Z",
"result": "{\"somekey\":1}",
"status": "QUEUED",
"author": 1,
"author": 2,
"logs": "log entry 1"
}
},
{
"model": "api.job",
"pk": "1a7947f9-6ae8-4e3d-ac1e-e7d608deec86",
"fields": {
"program": "6160a2ff-e482-443d-af23-15110b646ae2",
"created": "2023-02-01T15:30:43.281796Z",
"result": "{\"somekey\":1}",
"status": "QUEUED",
"author": 1
}
},
{
"model": "api.runtimejob",
"pk": "runtime_job_1",
Expand Down
39 changes: 39 additions & 0 deletions tests/basic/06_function.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
import os
Tansito marked this conversation as resolved.
Show resolved Hide resolved
from qiskit_serverless import QiskitFunction, ServerlessClient

serverless = ServerlessClient(
token=os.environ.get("GATEWAY_TOKEN", "awesome_token"),
host=os.environ.get("GATEWAY_HOST", "http://localhost:8000"),
)

help = """
title: custom-image-function
description: sample function implemented in a custom image
arguments:
service: service created with the accunt information
circuit: circuit
observable: observable
"""

function_with_custom_image = QiskitFunction(
title="custom-image-function",
image="test_function:latest",
provider=os.environ.get("PROVIDER_ID", "mockprovider"),
description=help
)
serverless.upload(function_with_custom_image)

my_functions = serverless.list()
for function in my_functions:
print("Name: " + function.title)
print(function.description)
print()

my_function = serverless.get("custom-image-function")
job = my_function.run(message="Argument for the custum function")

print(job.result())
print(job.logs())

jobs = my_function.get_jobs()
print(jobs)
15 changes: 15 additions & 0 deletions tests/basic/function/Sample-Docker
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
FROM icr.io/quantum-public/qiskit-serverless/ray-node:0.14.0-py310
Tansito marked this conversation as resolved.
Show resolved Hide resolved

# install all necessary dependencies for your custom image

# copy our function implementation in `/runner.py` of the docker image
USER 0
RUN mkdir /runner
WORKDIR /runner
COPY function/runner.py .
WORKDIR /

USER $RAY_UID



25 changes: 25 additions & 0 deletions tests/basic/function/runner.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
from qiskit import QuantumCircuit
from qiskit.primitives import StatevectorSampler as Sampler

def custom_function(arguments):
# all print statement will be available in job logs
print("Running function...")
message = arguments.get("message")
print(message)

# creating circuit
circuit = QuantumCircuit(2)
circuit.h(0)
circuit.cx(0, 1)
circuit.measure_all()

# running Sampler primitive
sampler = Sampler()
quasi_dists = sampler.run([(circuit)]).result()[0].data.meas.get_counts()

print("Completed running pattern.")
return quasi_dists

class Runner:
def run(self, arguments: dict) -> dict:
return custom_function(arguments)
Loading