Skip to content

Commit

Permalink
Adding better readme to example
Browse files Browse the repository at this point in the history
  • Loading branch information
AndrewFarley committed Jan 26, 2023
1 parent 7155d17 commit a1ef345
Showing 1 changed file with 17 additions and 5 deletions.
22 changes: 17 additions & 5 deletions examples/kubernetes.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@
- Official kubernetes-client python library: ``pip install kubernetes``
- See: https://github.com/kubernetes-client/python/
- A kubectl configuration and that is active and functional
- Can be your kubectl config setup for EKS with `aws eks update-kubeconfig --name clustername`
- Run Within' an Kubernetes cluster with an ClusterRole granting it access to Kubernetes APIs
- OR A working kubectl configuration and that is active and functional
- This can be your kubectl config setup for EKS with `aws eks update-kubeconfig --name clustername`
- Or access to any other hosted/managed/self-setup Kubernetes cluster
- For devs can a local minikube custer up and running: http://kubernetes.io/docs/getting-started-guides/minikube/
- Or for devs Docker Desktop has support for minikube: https://www.docker.com/products/docker-desktop
Expand All @@ -35,9 +36,19 @@
$ python -m luigi --module examples.kubernetes PerlPi --local-scheduler
Running this code will create a pi-luigi-uuid kubernetes job within the cluster
pointed to by the default context in "~/.kube/config". It will also auto-detect if this
of whatever your current context is for kubectl. The login herein will auto-detect if this
is running in an Kubernetes Cluster and has functional access to the local cluster's APIs
via an ClusterRole.
via an ClusterRole, and if not it will then try to use the kubeconfig and its various environment
variables to support configuring and tweaking kubectl.
DEPRECATION NOTE: The previous version of the kubernetes library had you configure your kubectl configuration file
as an property in the config file. This was removed in favor of using the kubectl standards. To specify a config use
the env variable KUBECONFIG. For example:
.. code:: console
$ KUBECONFIG=~/.kube/my-custom-config PYTHONPATH=. luigi --module examples.kubernetes_job PerlPi --local-scheduler
For more see: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/
"""

# import os
Expand All @@ -49,7 +60,8 @@ class PerlPi(KubernetesJobTask):
name = "pi" # The name (prefix) of the job that will be created for identification purposes
kubernetes_namespace = "default" # This is the kubernetes namespace you wish to run, if not specified it uses "default"
labels = {"job_name": "pi"} # This is to add labels in Kubernetes to help identify it, this is on top of the internal luigi labels
backoff_limit = 2 # This is the number of retries incase there is a pod/node/code failure, default 6
backoff_limit = 0 # This is the number of retries incase there is a pod/node/code failure, default 6
active_deadline_seconds = None # This is a "timeout" in seconds, how long to wait for the pods to schedule and execute before failing, default None
poll_interval = 1 # To poll more regularly reduce this number, default 5
print_pod_logs_on_exit = False # Set this to True if you wish to see the logs of this run after completion, False by default
print_pod_logs_during_run = True # Set this to True if you wish to see the logs of this run while it is running, False by default
Expand Down

0 comments on commit a1ef345

Please sign in to comment.