From 6b446dbde18ab9775c7cf7888c5144471dbf24d7 Mon Sep 17 00:00:00 2001 From: Alexey Stukalov Date: Mon, 15 Jan 2024 15:21:26 -0800 Subject: [PATCH] docs: refactor clustering validation --- docs/source/index.md | 8 +- docs/source/validate.md | 212 +++++++++++++++++++------------------- src/clustering_quality.jl | 48 +++++---- src/confusion.jl | 5 + src/counts.jl | 5 + 5 files changed, 149 insertions(+), 129 deletions(-) diff --git a/docs/source/index.md b/docs/source/index.md index b0c25ed1..21c4b949 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -3,7 +3,7 @@ *Clustering.jl* is a Julia package for data clustering. It covers the two aspects of data clustering: - - [Clustering Algorithms](@ref clu_algo_basics), *e.g.* K-means, K-medoids, Affinity - propagation, and DBSCAN, etc. - - [Clustering Evaluation](@ref clu_validate), *e.g.* Silhouettes and variational - information. + - [Clustering Algorithms](@ref clu_algo_basics): K-means, K-medoids, Affinity + propagation, DBSCAN etc. + - [Clustering Comparison & Evaluation](@ref clu_validate): cross-tabulation, variational + and mutual information, intrinsic clustering quality indices, such as *silhouettes*, etc. diff --git a/docs/source/validate.md b/docs/source/validate.md index 8b1fdbe6..5b3640be 100644 --- a/docs/source/validate.md +++ b/docs/source/validate.md @@ -1,10 +1,13 @@ # [Evaluation & Validation](@id clu_validate) -*Clustering.jl* package provides a number of methods to evaluate the results of -a clustering algorithm and/or to validate its correctness. +*Clustering.jl* package provides a number of methods to compare different clusterings, +evaluate clustering quality or validate its correctness. +## Clustering comparison -## Cross tabulation +Methods to compare two clusterings and measure their similarity. + +### Cross tabulation [Cross tabulation](https://en.wikipedia.org/wiki/Contingency_table), or *contingency matrix*, is a basis for many clustering quality measures. @@ -13,10 +16,20 @@ It shows how similar are the two clusterings on a cluster level. *Clustering.jl* extends `StatsBase.counts()` with methods that accept [`ClusteringResult`](@ref) arguments: ```@docs -counts(a::ClusteringResult, b::ClusteringResult) +counts(::ClusteringResult, ::ClusteringResult) +``` + +### Confusion matrix + +[Confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) +for the two clusterings is a 2×2 contingency table that counts +how frequently the pair of data points are in the same or different clusters. + +```@docs +confusion ``` -## Rand index +### Rand index [Rand index](http://en.wikipedia.org/wiki/Rand_index) is a measure of the similarity between the two data clusterings. From a mathematical @@ -27,51 +40,68 @@ even when the original class labels are not used. randindex ``` -## Silhouettes +### Variation of Information -[Silhouettes](http://en.wikipedia.org/wiki/Silhouette_(clustering)) is -a method for evaluating the quality of clustering. Particularly, it provides a -quantitative way to measure how well each point lies within its cluster in -comparison to the other clusters. +[Variation of information](http://en.wikipedia.org/wiki/Variation_of_information) +(also known as *shared information distance*) is a measure of the +distance between the two clusterings. It is devised from the *mutual +information*, but it is a true metric, *i.e.* it is symmetric and satisfies +the triangle inequality. -The *Silhouette* value for the ``i``-th data point is: -```math -s_i = \frac{b_i - a_i}{\max(a_i, b_i)}, \ \text{where} +```@docs +Clustering.varinfo ``` - - ``a_i`` is the average distance from the ``i``-th point to the other points in - the same cluster ``z_i``, - - ``b_i ≝ \min_{k \ne z_i} b_{ik}``, where ``b_{ik}`` is the average distance - from the ``i``-th point to the points in the ``k``-th cluster. +### V-measure -Note that ``s_i \le 1``, and that ``s_i`` is close to ``1`` when the ``i``-th -point lies well within its own cluster. This property allows using average silhouette value -`mean(silhouettes(assignments, counts, X))` as a measure of clustering quality; it is also available using `clustering_quality(...; quality_index = :silhouettes)` method. -Higher values indicate better separation of clusters w.r.t. point distances. +*V*-measure can be used to compare the clustering results with the +existing class labels of data points or with the alternative clustering. +It is defined as the harmonic mean of homogeneity (``h``) and completeness +(``c``) of the clustering: +```math +V_{\beta} = (1+\beta)\frac{h \cdot c}{\beta \cdot h + c}. +``` +Both ``h`` and ``c`` can be expressed in terms of the mutual information and +entropy measures from the information theory. Homogeneity (``h``) is maximized +when each cluster contains elements of as few different classes as possible. +Completeness (``c``) aims to put all elements of each class in single clusters. +The ``\beta`` parameter (``\beta > 0``) could used to control the weights of +``h`` and ``c`` in the final measure. If ``\beta > 1``, *completeness* has more +weight, and when ``\beta < 1`` it's *homogeneity*. ```@docs -silhouettes +vmeasure ``` -## Clustering quality indices +### Mutual information -A group of clustering evaluation metrics which are intrinsic, i.e. depend only on the clustering itself. They can be used to compare different clustering algorithms or choose the optimal number of clusters. +[Mutual information](https://en.wikipedia.org/wiki/Mutual_information) +quantifies the "amount of information" obtained about one random variable +through observing the other random variable. It is used in determining +the similarity of two different clusterings of a dataset. +```@docs +mutualinfo +``` +## Clustering quality indices -| **index name** | **quality_index** | **type** | **direction** | **cluster centers** | -|:-----------------:|:--------------------:|:----------:|:-------------:|:-------------------:| -| Calinski-Harabasz | `:calinsky_harabasz` | hard/fuzzy | up | required | -| Xie-Beni | `:xie_beni` | hard/fuzzy | down | required | -| Davis-Bouldin | `:davis_bouldin` | hard | down | required | -| Dunn | `:dunn` | hard | up | not required | -| silhouettes | `:silhouettes` | hard | up | not required | +[`clustering_quality()`][@ref clustering_quality] methods allow computing *intrinsic* clustering quality indices, +i.e. the metrics that depend only on the clustering itself and do not use the external knowledge. +These metrics can be used to compare different clustering algorithms or choose the optimal number of clusters. +| **quality index** | **`quality_index` option** | **clustering type** | **better quality** | **cluster centers** | +|:-------------------------------------------:|:--------------------:|:----------:|:-------------:|:-------------------:| +| [Calinski-Harabasz](@ref calinsky_harabasz) | `:calinsky_harabasz` | hard/fuzzy | *higher* values | required | +| [Xie-Beni](@ref xie_beni) | `:xie_beni` | hard/fuzzy | *lower* values | required | +| [Davis-Bouldin](@ref davis_bouldin) | `:davis_bouldin` | hard | *lower* values | required | +| [Dunn](@ref dunn) | `:dunn` | hard | *higher* values | not required | +| [silhouettes](@ref silhouettes_index) | `:silhouettes` | hard | *higher* values | not required | ```@docs -Clustering.clustering_quality +clustering_quality ``` -Notation for the index definitions below: +The clustering quality index definitions use the following notation: - ``x_1, x_2, \ldots, x_n``: data points, - ``C_1, C_2, \ldots, C_k``: clusters, - ``c_j`` and ``c``: cluster centers and global dataset center, @@ -79,30 +109,32 @@ Notation for the index definitions below: - ``w_{ij}``: weights measuring membership of a point ``x_i`` to a cluster ``C_j``, - ``\alpha``: a fuzziness parameter. -### Calinski-Harabasz index - -Option `:calinski_harabasz`. Higher values indicate better quality. Measures corrected ratio between global inertia of the cluster centers and the summed internal inertias of clusters. For hard and fuzzy (soft) clustering it is defined as +### [Calinski-Harabasz index](@id calinsky_harabasz) +[*Calinski-Harabasz* index](https://en.wikipedia.org/wiki/Calinski%E2%80%93Harabasz_index) (option `:calinski_harabasz`) +measures corrected ratio between global inertia of the cluster centers and the summed internal inertias of clusters: ```math - \frac{n-k}{k-1}\frac{\sum_{C_j}|C_j|d(c_j,c)}{\sum\limits_{C_j}\sum\limits_{x_i\in C_j} d(x_i,c_j)} \quad \text{and}\quad \frac{n-k}{k-1} \frac{\sum\limits_{C_j}\left(\sum\limits_{x_i}w_{ij}^\alpha\right) d(c_j,c)}{\sum_{C_j} \sum_{x_i} w_{ij}^\alpha d(x_i,c_j)} ``` -respectively. +for hard and fuzzy (soft) clusterings, respectively. +*Higher* values indicate better quality. +### [Xie-Beni index](@id xie_beni) -### Xie-Beni index -Option `:xie_beni`. Lower values indicate better quality. Measures ratio between summed inertia of clusters and minimum distance between cluster centres. For hard clustering and fuzzy (soft) clustering. It is defined as +*Xie-Beni* index (option `:xie_beni`) measures ratio between summed inertia of clusters +and the minimum distance between cluster centres: ```math \frac{\sum_{C_j}\sum_{x_i\in C_j}d(x_i,c_j)}{n\min\limits_{c_{j_1}\neq c_{j_2}} d(c_{j_1},c_{j_2}) } \quad \text{and}\quad \frac{\sum_{C_j}\sum_{x_i} w_{ij}^\alpha d(x_i,c_j)}{n\min\limits_{c_{j_1}\neq c_{j_2}} d(c_{j_1},c_{j_2}) } ``` -respectively. - -### [Davis-Bouldin index](https://en.wikipedia.org/wiki/Davies%E2%80%93Bouldin_index) -Option `:davis_bouldin`. Lower values indicate better quality. It measures average cohesion based on the cluster diameters and distances between cluster centers. It is defined as +for hard and fuzzy (soft) clusterings, respectively. +*Lower* values indicate better quality. +### [Davis-Bouldin index](@id davis_bouldin) +[*Davis-Bouldin* index](https://en.wikipedia.org/wiki/Davies%E2%80%93Bouldin_index) +(option `:davis_bouldin`) measures average cohesion based on the cluster diameters and distances between cluster centers: ```math \frac{1}{k}\sum_{C_{j_1}}\max_{c_{j_2}\neq c_{j_1}}\frac{S(C_{j_1})+S(C_{j_2})}{d(c_{j_1},c_{j_2})} ``` @@ -110,8 +142,11 @@ where ```math S(C_j) = \frac{1}{|C_j|}\sum_{x_i\in C_j}d(x_i,c_j). ``` -### [Dunn index](https://en.wikipedia.org/wiki/Dunn_index) -Option `:dunn`. Higher values indicate better quality. More computationally demanding index which can be used when the centres are not known. It measures ratio between the nearest neighbour distance divided by the maximum cluster diameter. It is defined as +*Lower* values indicate better quality. + +### [Dunn index](@id dunn) +[*Dunn* index](https://en.wikipedia.org/wiki/Dunn_index) (option `:dunn`) +measures the ratio between the nearest neighbour distance divided by the maximum cluster diameter: ```math \frac{\min\limits_{ C_{j_1}\neq C_{j_2}} \mathrm{dist}(C_{j_1},C_{j_2})}{\max\limits_{C_j}\mathrm{diam}(C_j)} ``` @@ -119,13 +154,36 @@ where ```math \mathrm{dist}(C_{j_1},C_{j_2}) = \min\limits_{x_{i_1}\in C_{j_1},x_{i_2}\in C_{j_2}} d(x_{i_1},x_{i_2}),\quad \mathrm{diam}(C_j) = \max\limits_{x_{i_1},x_{i_2}\in C_j} d(x_{i_1},x_{i_2}). ``` +It is more computationally demanding quality index, which can be used when the centres are not known. *Higher* values indicate better quality. -### Average silhouette index +### [Silhouettes](@id silhouettes_index) -Option `:silhouettes`. Higher values indicate better quality. It returns the average over silhouette values in the whole data set. See section [Silhouettes](#silhouettes) for a more detailed description of the method. +[*Silhouettes* metric](http://en.wikipedia.org/wiki/Silhouette_(clustering)) quantifies the correctness of point-to-cluster asssignment by +comparing the distance of the point to its cluster and to the other clusters. +The *Silhouette* value for the ``i``-th data point is: +```math +s_i = \frac{b_i - a_i}{\max(a_i, b_i)}, \ \text{where} +``` + - ``a_i`` is the average distance from the ``i``-th point to the other points in + the *same* cluster ``z_i``, + - ``b_i ≝ \min_{k \ne z_i} b_{ik}``, where ``b_{ik}`` is the average distance + from the ``i``-th point to the points in the ``k``-th cluster. + +Note that ``s_i \le 1``, and that ``s_i`` is close to ``1`` when the ``i``-th +point lies well within its own cluster. This property allows using average silhouette value +`mean(silhouettes(assignments, counts, X))` as a measure of clustering quality; +it is also available using [`clustering_quality(...; quality_index = :silhouettes)`](@ref clustering_quality) method. +Higher values indicate better separation of clusters w.r.t. point distances. -### References +```@docs +silhouettes +``` + +[`clustering_quality(..., quality_index=:silhouettes)`][@ref clustering_quality] +provides mean silhouette metric for the datapoints. Higher values indicate better quality. + +## References > Olatz Arbelaitz *et al.* (2013). *An extensive comparative study of cluster validity indices*. Pattern Recognition. 46 1: 243-256. [doi:10.1016/j.patcog.2012.07.021](https://doi.org/10.1016/j.patcog.2012.07.021) > Aybükë Oztürk, Stéphane Lallich, Jérôme Darmont. (2018). *A Visual Quality Index for Fuzzy C-Means*. 14th International Conference on Artificial Intelligence Applications and Innovations (AIAI 2018). 546-555. [doi:10.1007/978-3-319-92007-8_46](https://doi.org/10.1007/978-3-319-92007-8_46). @@ -195,62 +253,8 @@ plot(( ) ``` -## Variation of Information - -[Variation of information](http://en.wikipedia.org/wiki/Variation_of_information) -(also known as *shared information distance*) is a measure of the -distance between the two clusterings. It is devised from the *mutual -information*, but it is a true metric, *i.e.* it is symmetric and satisfies -the triangle inequality. - -```@docs -Clustering.varinfo -``` - - -## V-measure - -*V*-measure can be used to compare the clustering results with the -existing class labels of data points or with the alternative clustering. -It is defined as the harmonic mean of homogeneity (``h``) and completeness -(``c``) of the clustering: -```math -V_{\beta} = (1+\beta)\frac{h \cdot c}{\beta \cdot h + c}. -``` -Both ``h`` and ``c`` can be expressed in terms of the mutual information and -entropy measures from the information theory. Homogeneity (``h``) is maximized -when each cluster contains elements of as few different classes as possible. -Completeness (``c``) aims to put all elements of each class in single clusters. -The ``\beta`` parameter (``\beta > 0``) could used to control the weights of -``h`` and ``c`` in the final measure. If ``\beta > 1``, *completeness* has more -weight, and when ``\beta < 1`` it's *homogeneity*. - -```@docs -vmeasure -``` - -## Mutual information - -[Mutual information](https://en.wikipedia.org/wiki/Mutual_information) -quantifies the "amount of information" obtained about one random variable -through observing the other random variable. It is used in determining -the similarity of two different clusterings of a dataset. - -```@docs -mutualinfo -``` - -## Confusion matrix - -Pair [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) -arising from two clusterings is a 2×2 contingency table representation of -the partition co-occurrence, see [`counts`](@ref). - -```@docs -confusion -``` ## Other packages * [ClusteringBenchmarks.jl](https://github.com/HolyLab/ClusteringBenchmarks.jl) provides - benchmark datasets and implements additional methods for evaluating clustering performance. \ No newline at end of file + benchmark datasets and implements additional methods for evaluating clustering performance. diff --git a/src/clustering_quality.jl b/src/clustering_quality.jl index 8acadc19..2d21859a 100644 --- a/src/clustering_quality.jl +++ b/src/clustering_quality.jl @@ -1,27 +1,30 @@ # main method for hard clustering indices + docs """ -For hard clustering: +For "hard" clustering: clustering_quality(data, centers, assignments; quality_index, [metric]) clustering_quality(data, clustering; quality_index, [metric]) -For fuzzy clustering: +For fuzzy ("soft") clustering: clustering_quality(data, centers, weights; quality_index, fuzziness, [metric]) clustering_quality(data, clustering; quality_index, fuzziness, [metric]) -For hard clustering without cluster centers known: +For "hard" clustering without specifying cluster centers: - clustering_quality(assignments, dist_matrix; quality_index) - clustering_quality(clustering, dist_matrix; quality_index) clustering_quality(data, assignments; quality_index, [metric]) clustering_quality(data, clustering; quality_index, [metric]) -Compute the clustering quality index for a given clustering. +For "hard" clustering without specifying data points and cluster centers: -Returns a real number which is the value of the chosen quality index type of the given clustering. + clustering_quality(assignments, dist_matrix; quality_index) + clustering_quality(clustering, dist_matrix; quality_index) -# Arguments +Compute the *quality index* for a given clustering. + +Returns a quality index (real value). + +## Arguments - `data::AbstractMatrix`: ``d×n`` data matrix with each column representing one ``d``-dimensional data point - `centers::AbstractMatrix`: ``d×k`` matrix with cluster centers represented as columns - `assignments::AbstractVector{Int}`: ``n`` vector of point assignments (cluster indices) @@ -30,25 +33,28 @@ Returns a real number which is the value of the chosen quality index type of the - `quality_index::Symbol`: quality index to calculate; see below for the supported options - `dist_matrix::AbstractMatrix`: a ``n×n`` pairwise distance matrix; `dist_matrix[i,j]` is the distance between ``i``-th and ``j``-th points - # Keyword arguments - - `quality_index::Symbol`: quality index to calculate; see below for the supported options - - `fuzziness::Real`: clustering fuzziness > 1 + ## Keyword arguments + - `quality_index::Symbol`: clustering *quality index* to calculate; see below for the supported options + - `fuzziness::Real`: clustering *fuzziness* > 1 - `metric::SemiMetric=SqEuclidean()`: `SemiMetric` object that defines the metric/distance/similarity function -When calling `clustering_quality` one can give `centers`, `assignments` or `weights` arguments by hand or provide a single `clustering` argument from which the necessary data will be read automatically. +When calling `clustering_quality`, one can explicitly specify `centers`, `assignments`, and `weights`, +or provide `ClusteringResult` via `clustering`, from which the necessary data will be read automatically. -For clustering without known cluster centers the datapoints are not required, only `dist_matrix` is necessary. If given, `data` and `metric` will be used to calculate distance matrix instead. +For clustering without known cluster centers the `data` points are not required. +`dist_matrix` could be provided explicitly, otherwise it would be calculated from the `data` points +using the specified `metric`. -# Supported quality indices +## Supported quality indices -Symbols ↑/↓ are quality direction. -- `:calinski_harabasz`: hard or fuzzy Calinski-Harabsz index (↑) returns the corrected ratio of between cluster centers inertia and within-clusters inertia -- `:xie_beni`: hard or fuzzy Xie-Beni index (↓) returns ratio betwen inertia within clusters and minimal distance between cluster centers -- `:davies_bouldin`: Davies-Bouldin index (↓) returns average similarity between each cluster and its most similar one, averaged over all the clusters -- `:dunn`: Dunn index (↑) returns ratio between minimal distance between clusters and maximal cluster diameter; it does not make use of `centers` argument -- `:silhouettes`: average silhouette index (↑), for all silhouettes use [`silhouettes`](@ref) method instead; it does not make use of `centers` argument -Please refer to the [documentation](@ref clustering_quality) for the definitions and usage descriptions of the supported quality indices. +- `:calinski_harabasz`: hard or fuzzy Calinski-Harabsz index (↑), the corrected ratio of between cluster centers inertia and within-clusters inertia +- `:xie_beni`: hard or fuzzy Xie-Beni index (↓), the ratio betwen inertia within clusters and minimal distance between the cluster centers +- `:davies_bouldin`: Davies-Bouldin index (↓), the similarity between the cluster and the other most similar one, averaged over all clusters +- `:dunn`: Dunn index (↑), the ratio of the minimal distance between clusters and the maximal cluster diameter +- `:silhouettes`: the average silhouette index (↑), see [`silhouettes`](@ref) +The arrows ↑ or ↓ specify the direction of the incresing clustering quality. +Please refer to the [documentation](@ref clustering_quality) for more details on the clustering quality indices. """ function clustering_quality( data::AbstractMatrix{<:Real}, # d×n matrix diff --git a/src/confusion.jl b/src/confusion.jl index 89cc42d7..6556b814 100644 --- a/src/confusion.jl +++ b/src/confusion.jl @@ -19,6 +19,11 @@ true negatives is `C₂₂`: |:--:|:-:|:-:| |Positive|C₁₁|C₁₂| |Negative|C₂₁|C₂₂| + +## See also + +[`counts(a::ClusteringResult, a::ClusteringResult)`](@ref counts) for full *contingency matrix*. + """ function confusion(::Type{T}, a::AbstractVector{<:Integer}, b::AbstractVector{<:Integer}) where T<:Union{Integer, AbstractFloat} cc = counts(a, b) diff --git a/src/counts.jl b/src/counts.jl index 1eef56fb..7e5a303c 100644 --- a/src/counts.jl +++ b/src/counts.jl @@ -29,6 +29,11 @@ from `b`. The clusterings could be specified either as [`ClusteringResult`](@ref) instances or as vectors of data point assignments. + +## See also + +[`confusion(a::ClusteringResult, a::ClusteringResult)`](@ref confusion) for 2×2 *confusion matrix*. + """ counts(a::ClusteringResult, b::ClusteringResult) = _counts(assignments(a), assignments(b))