refactor(dask): remove the dask backend

BREAKING CHANGE: The `dask` backend is removed. Please use one of the other backends that Ibis supports.
ibis-project · Aug 24, 2024 · 1db5d0e · 1db5d0e
1 parent 949fbea
commit 1db5d0e
Show file tree

Hide file tree

Showing 64 changed files with 124 additions and 5,407 deletions.
diff --git a/.github/renovate.json b/.github/renovate.json
@@ -29,10 +29,6 @@
       "matchPackagePrefixes": ["clickhouse"],
       "addLabels": ["clickhouse"]
     },
-    {
-      "matchPackagePatterns": ["dask"],
-      "addLabels": ["dask"]
-    },
     {
       "matchPackagePatterns": ["datafusion"],
       "addLabels": ["datafusion"]

diff --git a/.github/workflows/ibis-backends.yml b/.github/workflows/ibis-backends.yml
@@ -233,13 +233,6 @@ jobs:
             services:
               - flink
         include:
-          - os: ubuntu-latest
-            python-version: "3.11.8"
-            backend:
-              name: dask
-              title: Dask
-              extras:
-                - dask
           - os: ubuntu-latest
             python-version: "3.11"
             backend:
@@ -531,17 +524,6 @@ jobs:
           - "3.10"
           - "3.12"
         backend:
-          - name: dask
-            title: Dask
-            deps:
-              required:
-                - "[email protected]"
-                - "[email protected]"
-              optional:
-                - "dask[array,dataframe]@2022.9.1"
-                - "[email protected]"
-            extras:
-              - dask
           - name: postgres
             title: PostgreSQL
             deps:
@@ -592,19 +574,6 @@ jobs:
               extras:
                 - postgres
                 - geospatial
-          - python-version: "3.12"
-            backend:
-              name: dask
-              title: Dask
-              deps:
-                required:
-                  - "[email protected]"
-                  - "[email protected]"
-                optional:
-                  - "dask[array,dataframe]@2022.9.1"
-                  - "[email protected]"
-              extras:
-                - dask
     steps:
       - name: checkout
         uses: actions/checkout@v4

diff --git a/README.md b/README.md
@@ -144,7 +144,6 @@ Ibis supports 20+ backends:
 - [Apache PySpark](https://ibis-project.org/backends/pyspark/)
 - [BigQuery](https://ibis-project.org/backends/bigquery/)
 - [ClickHouse](https://ibis-project.org/backends/clickhouse/)
-- [Dask](https://ibis-project.org/backends/dask/)
 - [DuckDB](https://ibis-project.org/backends/duckdb/)
 - [Exasol](https://ibis-project.org/backends/exasol)
 - [MySQL](https://ibis-project.org/backends/mysql/)

diff --git a/conda/environment-arm64-flink.yml b/conda/environment-arm64-flink.yml
@@ -7,7 +7,6 @@ dependencies:
   - atpublic >=2.3
   - black >=22.1.0,<25
   - clickhouse-connect >=0.5.23
-  - dask >=2022.9.1
   - datafusion >=0.6
   - db-dtypes >=0.3.0,<2
   - deltalake

diff --git a/conda/environment-arm64.yml b/conda/environment-arm64.yml
@@ -7,7 +7,6 @@ dependencies:
   - atpublic >=2.3
   - black >=22.1.0,<25
   - clickhouse-connect >=0.5.23
-  - dask >=2022.9.1
   - datafusion >=0.6
   - db-dtypes >=0.3.0,<2
   - deltalake

diff --git a/conda/environment.yml b/conda/environment.yml
@@ -7,7 +7,6 @@ dependencies:
   - atpublic >=2.3
   - black >=22.1.0,<25
   - clickhouse-connect >=0.5.23
-  - dask >=2022.9.1
   - datafusion >=0.6
   - db-dtypes >=0.3.0,<2
   - deltalake

diff --git a/docs/backend_table_hiearchy.qmd b/docs/backend_table_hiearchy.qmd
@@ -20,7 +20,6 @@ use the terms `catalog` and `database` and map them onto the appropriate fields.
 |------------|----------------|------------|
 | bigquery   | project        | database   |
 | clickhouse |                | database   |
-| dask       |                | NA         |
 | datafusion | catalog        | schema     |
 | druid      | dataSourceType | dataSource |
 | duckdb     | database       | schema     |

diff --git a/docs/backends/dask.qmd b/docs/backends/dask.qmd
@@ -1,81 +1,7 @@
 # Dask
 
-[https://www.dask.org](https://www.dask.org)
-
-![](https://img.shields.io/badge/memtables-native-green?style=flat-square) ![](https://img.shields.io/badge/inputs-CSV | Parquet-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-CSV | pandas | Parquet | PyArrow-orange?style=flat-square)
-
-::: {.callout-warning}
-## The Dask backend is slated for removal in Ibis 10.0
-We recommend using one of our other backends.
-
-Many workloads work well on the DuckDB and Polars backends, for example.
-:::
-
-## Install
-
-Install Ibis and dependencies for the Dask backend:
-
-::: {.panel-tabset}
-
-## `pip`
-
-Install with the `dask` extra:
-
-```{.bash}
-pip install 'ibis-framework[dask]'
-```
-
-And connect:
-
-```{.python}
-import ibis
-
-con = ibis.dask.connect()  # <1>
-```
-
-1. Adjust connection parameters as needed.
-
-## `conda`
-
-Install for Dask:
-
-```{.bash}
-conda install -c conda-forge ibis-dask
-```
-
-And connect:
-
-```{.python}
-import ibis
-
-con = ibis.dask.connect()  # <1>
-```
-
-1. Adjust connection parameters as needed.
-
-## `mamba`
-
-Install for Dask:
-
-```{.bash}
-mamba install -c conda-forge ibis-dask
-```
-
-And connect:
-
-```{.python}
-import ibis
-
-con = ibis.dask.connect()  # <1>
-```
-
-1. Adjust connection parameters as needed.
+::: {.callout-note}
+## The Dask backend was removed in Ibis version 10.0
 
+See [our blog post](../posts/farewell-pandas/index.qmd) on the topic for more information.
 :::
-
-```{python}
-#| echo: false
-BACKEND = "Dask"
-```
-
-{{< include ./_templates/api.qmd >}}
diff --git a/docs/why.qmd b/docs/why.qmd
@@ -324,7 +324,6 @@ use Ibis with other tools over time.
 Ibis already works with other Python dataframes like:
 
 - [pandas](https://github.com/pandas-dev/pandas)
-- [Dask](https://github.com/dask/dask)
 - [Polars](https://github.com/pola-rs/polars)
 
 Ibis already works well with visualization libraries like:

diff --git a/flake.nix b/flake.nix
@@ -134,9 +134,7 @@
         ibis311 = mkDevShell pkgs.ibisDevEnv311;
         ibis312 = mkDevShell pkgs.ibisDevEnv312;
 
-        # move back to 3.12 when dask-expr is supported or the dask backend is
-        # removed
-        default = ibis310;
+        default = ibis312;
 
         preCommit = pkgs.mkShell {
           name = "preCommit";

diff --git a/ibis/backends/conftest.py b/ibis/backends/conftest.py
@@ -462,7 +462,7 @@ def _setup_backend(request, data_dir, tmp_path_factory, worker_id):
 
 
 @pytest.fixture(
-    params=_get_backends_to_test(discard=("dask", "pandas")),
+    params=_get_backends_to_test(discard=("pandas",)),
     scope="session",
 )
 def ddl_backend(request, data_dir, tmp_path_factory, worker_id):
@@ -477,7 +477,7 @@ def ddl_con(ddl_backend):
 
 
 @pytest.fixture(
-    params=_get_backends_to_test(keep=("dask", "pandas", "pyspark")),
+    params=_get_backends_to_test(keep=("pandas", "pyspark")),
     scope="session",
 )
 def udf_backend(request, data_dir, tmp_path_factory, worker_id):