RFD - Proposal for more explicit nebari-config.yaml

[Adam-D-Lewis]

Status	Open for comments 💬
Author(s)	@Adam-D-Lewis
Date Created	04-11-2024
Date Last updated	04-11-2024
Decision deadline	TBD

Proposal for more explicit nebari-config.yaml

Summary

The average user doesn't know what options they have available to them to adjust from nebari-config.yaml. Those who deploy fresh deployments may run into the situation where they need to adjust these immediately (see here, and it's not apparent how to do so and not straightforward at the moment.

Additionally, extensions InputSchema are not currently written to nebari-config.yaml, meaning users have to copy those in manually to manage them from nebari-config.yaml.

User benefit

Users will better understand what they can configure from the nebari-config.yaml file. They will also see what they've set to custom values easily since those will be the only values not commented out.

Design Proposal

The following features are a part of this design proposal.

• Add sections to the nebari-config.yaml
• attributes that are set to their default values are commented out so it's easy for users to see what they've set explicitly
• ability for the extension to exclude certain fields from being written to nebari-config.yaml.
• add ability to display helpful descriptions for each field in nebari-config.yaml
• if default values change in Nebari, then running nebari upgrade should notice the updated defaults and update the corresponding commented out default values in nebari-config.yaml

A sample nebari config.yaml might look something like below under this design proposal (inspired from grafana's defaults.ini)

##################### Main #####################

# Whether to prevent deploying after upgrades due to potentially problematic changes
prevent_deploy: false
# Current nebari version used for deployment
nebari_version: 2024.3.3.dev138+g924659d5
# whether to use a cloud provider, local, or existing k8s cluster
provider: local
# k8s namespace to deploy into
namespace: dev
# project name for the deployment
project_name: scratch-2efc

##################### Bootstrap Stage #####################
# whether to use CI/CD for deployments
# ci_cd:
#   type: none
#   branch: main
#   commit_render: true
#   before_script: []
#   after_script: []

##################### Terraform State Stage #####################
# where to store the terraform state
terraform_state:
  type: remote
  backend:
  config: {}

##################### Infrastructure Stage #####################
# local deployment settings
local:
  kube_context:
  node_selectors:
    general:
      key: kubernetes.io/os
      value: linux
    user:
      key: kubernetes.io/os
      value: linux
    worker:
      key: kubernetes.io/os
      value: linux

##################### Kubernetes Initialize Stage #####################
# configuration for external container registry
external_container_reg:
  enabled: false
  access_key_id:
  secret_access_key:
  extcr_account:
  extcr_region:

##################### Kubernetes Ingress Stage #####################
# dns configuration
dns:
  provider:
  auto_provision: false

# ingress
ingress:
  terraform_overrides: {}

# certificate settings
certificate: self-signed
self_signed_certificate:

certificate:
  type: self-signed
domain: github-actions.nebari.dev

##################### Kubernetes Keycloak Stage #####################
security:
  authentication:
    type: password
  shared_users_group: true
  keycloak:
    initial_root_password: 9u9le3v5f3nb2kf98rh77xthjsl00v80
    overrides: {}
    realm_display_name: Nebari

##################### Kubernetes Services Stage #####################
# jhub apps settings
jhub_apps:
  enabled: false

# jupyterlab settings
jupyterlab:
  default_settings: {}
  idle_culler:
    terminal_cull_inactive_timeout: 15
    terminal_cull_interval: 5
    kernel_cull_idle_timeout: 15
    kernel_cull_interval: 5
    kernel_cull_connected: true
    kernel_cull_busy: false
    server_shutdown_no_activity_timeout: 15
  initial_repositories: []
  preferred_dir:

# jupyterhub settings
jupyterhub:
  overrides: {}

# jupyterlab telemetry settings
telemetry:
  jupyterlab_pioneer:
    enabled: false
    log_format:

# monitoring settings
monitoring:
  enabled: true
  overrides:
    loki: {}
    promtail: {}
    minio: {}
  minio_enabled: true

# argo workflows settings
argo_workflows:
  enabled: true
  overrides: {}
  nebari_workflow_controller:
    enabled: true
    image_tag: 2024.3.3

# conda_store settings
conda_store:
  extra_settings: {}
  extra_config: ''
  image: quansight/conda-store-server
  image_tag: 2024.3.1
  default_namespace: nebari-git
  object_storage: 200Gi

# default conda environments
environments:
  environment-dask.yaml:
    name: dask
    channels:
    - conda-forge
    dependencies:
    - python==3.11.6
    - ipykernel==6.26.0
    - ipywidgets==8.1.1
    - nebari-dask==2024.3.3
    - python-graphviz==0.20.1
    - pyarrow==14.0.1
    - s3fs==2023.10.0
    - gcsfs==2023.10.0
    - numpy=1.26.0
    - numba=0.58.1
    - pandas=2.1.3
    - xarray==2023.10.1
  environment-dashboard.yaml:
    name: dashboard
    channels:
    - conda-forge
    dependencies:
    - python==3.11.6
    - cufflinks-py==0.17.3
    - dash==2.14.1
    - geopandas==0.14.1
    - geopy==2.4.0
    - geoviews==1.11.0
    - gunicorn==21.2.0
    - holoviews==1.18.1
    - ipykernel==6.26.0
    - ipywidgets==8.1.1
    - jupyter==1.0.0
    - jupyter_bokeh==3.0.7
    - matplotlib==3.8.1
    - nebari-dask==2024.3.3
    - nodejs=20.8.1
    - numpy==1.26.0
    - openpyxl==3.1.2
    - pandas==2.1.3
    - panel==1.3.1
    - param==2.0.1
    - plotly==5.18.0
    - python-graphviz==0.20.1
    - rich==13.6.0
    - streamlit==1.28.1
    - sympy==1.12
    - voila==0.5.5
    - xarray==2023.10.1
    - pip==23.3.1
    - pip:
      - streamlit-image-comparison==0.0.4
      - noaa-coops==0.1.9
      - dash_core_components==2.0.0
      - dash_html_components==2.0.0

# jupyterlab profiles for users / dask workers
profiles:
  jupyterlab:
  - access: all
    display_name: Small Instance
    description: Stable environment with 2 cpu / 8 GB ram
    default: true
    users:
    groups:
    kubespawner_override:
      cpu_limit: 2.0
      cpu_guarantee: 1.5
      mem_limit: 8G
      mem_guarantee: 5G
  - access: all
    display_name: Medium Instance
    description: Stable environment with 4 cpu / 16 GB ram
    default: false
    users:
    groups:
    kubespawner_override:
      cpu_limit: 4.0
      cpu_guarantee: 3.0
      mem_limit: 16G
      mem_guarantee: 10G
  dask_worker:
    Small Worker:
      worker_cores_limit: 2.0
      worker_cores: 1.5
      worker_memory_limit: 8G
      worker_memory: 5G
      worker_threads: 2
    Medium Worker:
      worker_cores_limit: 4.0
      worker_cores: 3.0
      worker_memory_limit: 16G
      worker_memory: 10G
      worker_threads: 4

# Nebari theme settings
theme:
  jupyterhub:
    hub_title: Nebari - scratch-2efc
    hub_subtitle: Your open source data science platform, hosted
    welcome: Welcome! Learn about Nebari's features and configurations in <a href="https://www.nebari.dev/docs/welcome">the
      documentation</a>. If you have any questions or feedback, reach the team on
      <a href="https://www.nebari.dev/docs/community#getting-support">Nebari's support
      forums</a>.
    logo: 
      https://raw.githubusercontent.com/nebari-dev/nebari-design/main/logo-mark/horizontal/Nebari-Logo-Horizontal-Lockup-White-text.svg
    favicon: 
      https://raw.githubusercontent.com/nebari-dev/nebari-design/main/symbol/favicon.ico
    primary_color: '#4f4173'
    primary_color_dark: '#4f4173'
    secondary_color: '#957da6'
    secondary_color_dark: '#957da6'
    accent_color: '#32C574'
    accent_color_dark: '#32C574'
    text_color: '#111111'
    h1_color: '#652e8e'
    h2_color: '#652e8e'
    version: v2024.3.3.dev138+g924659d5
    navbar_color: '#1c1d26'
    navbar_text_color: '#f1f1f6'
    navbar_hover_color: '#db96f3'
    display_version: 'True'

# conda_store settings
storage:
  conda_store: 200Gi
  shared_filesystem: 200Gi

# docker images to use for deployment
default_images:
  jupyterhub: quay.io/nebari/nebari-jupyterhub:2024.3.3
  jupyterlab: quay.io/nebari/nebari-jupyterlab:2024.3.3
  dask_worker: quay.io/nebari/nebari-dask-worker:2024.3.3

##################### Nebari Terraform/Helm Extensions Stage #####################
# extensions to install
tf_extensions: []
helm_extensions: []

Not all Nebari config will be shown. Instead, just the config relevant to what they've chosen in the nebari init command will be shown.
E.g. Only the e.g. aws provider will be shown instead of all the cloud provider config sections.

Alternatives or approaches considered (if any)

We could allow more flexibility by allowing each stage to have it's own write_config method which returns yaml, but I'm not sure we have a use case for that currently and it's nice to enforce some stylistic uniformity between the sections.

Best practices

User impact

Unresolved questions

[Adam-D-Lewis]

Ruamel does not seem flexible enough to write the config as I've described it. I think we'll need to write our own yaml writer if we want to accomplish default values being commented out.

I'm also appreciating more fully that it may be difficult to change the commented out default values when running nebari upgrade. It'd be easier to just have a cli command to regenerate the config file preserving the values that have been set explicitly.

[viniciusdc]

@Adam-D-Lewis, this looks great!! Based on what was discussed during our last meeting, having all the values in the yaml by default would be great. My overall perspective on this would be:

• docs should have the help/description of the fields
• While the yaml should have all the default fields present as the default behavior, I would reason that users can only look for things in the docs if they exist.

On a side note, regarding dynamically updating the docs on conda-forge, for example (this doc for the conda-forge.yml schema):

• We have the pydantic models here;
• Then, we use this typescript component on docssauros to generate the docs reference page based on the models' JSON schema.

[Adam-D-Lewis]

On a side note, regarding dynamically updating the docs on conda-forge, for example (this doc for the conda-forge.yml schema):

We have the pydantic models here;
Then, we use this typescript component on docssauros to generate the docs reference page based on the models' JSON schema.

Thanks @viniciusdc, that sounds very useful and I'll work on adding something similar for Nebari.

[viniciusdc]

No worries. I think this would be a separate issue, though, a the docs repo.

For this RFD, I suggest:

• using your current Pydantic v2 PR as the solution for showing user the default config value, only including an extra flag to the CLI where the user can opt for a more minimalist config file (defaults to all fields).
• And add another task where we make sure to update our models with descriptions and dynamic generating our docs

[dcmcand]

I think @viniciusdc is dead on. There is room for further refinement later, but this seems to be the most direct line to delivering value.

I think additional related tickets would be:

1. auto generate docs with descriptions on each merge to main. If there is a difference in the docs, auto open a docs pr.
2. Add example configs for commonly requested configurations (for example: gpu instances). If we add a ci step that runs nebari validate against all of the example configs, in theory that should help keep them up to date.

[Adam-D-Lewis]

As far as generating a nebari config with commented out sections, the tomlkit library looked promising. https://tomlkit.readthedocs.io/en/latest/quickstart/#writing

[Adam-D-Lewis]

then, we use this typescript component on docssauros to generate the docs reference page based on the models' JSON schema.

I like this idea @viniciusdc of autogenerating some docs from openapi.json, but it seems like we'd need some typescript expertise to create this which I don't think any contributors have at the moment. Is that your understanding as well?

[Adam-D-Lewis]

Talking with other members of nebari core team, we propose that we add this as a non default cli flag that can be enabled for the time being e.g. nebari init --verbose .... Eventually, it may become the default. This allows us to add it soon for users who want it, and continue to improve it based on user feedback to the point where it may become the default one day. It may eventually look like what's above, but for now it'll be less polished and more for advanced users only.

PR here: nebari-dev/nebari#2294

[Adam-D-Lewis]

We merged a PR adding nebari init ... --explicit here so I'll close this issue