Unstable results with same parameters

[pooria98]

I get unstable results each time I run my code.
first time I ran it, is resulted in 19 reasonable topics.
I opened the jupyter notebook later that day and ran the code from the beginning again without chaging anything and got only 4 topics that all looked wrong and not related to my dataset.
then I cleared cell outputs, restarted the kernel and ran it again without changing any parameters. now it shows the "max_df corresponds to < documents than min_df" error message.

UMAP random_state is also set to prevent stochastic behavior. so what is wrong??? how can I get back those good topics?

this is the code block:

# Load embeddings from disk
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = np.load("embeddings.npy")
print(f"Embeddings: {embeddings.dtype}")

# UMAP model (to prevent stochastic behavior and get fixed results)
umap_model = UMAP(n_neighbors=30, n_components=5, min_dist=0.02, metric='cosine', random_state=42)

# HDBSCAN model (for custom cluster settings)
hdbscan_model = HDBSCAN(min_cluster_size=100, metric='euclidean', cluster_selection_method='eom', prediction_data=True)

# vectorizer model (removing stop-words - and decreasing noise - allow bigrams tokenization) // set "max_features" parameter later !!!
custom_stop_words = ["reddit", "post", "thread", "subreddit", "r/", "u/", "mod", "moderator", "admin", "report", "ban", "karma", "upvote", "downvote", "award", "gold", "silver", "platinum", "OP", "user", "account", "profile", "username", "comment", "reply", "edit", "delete", "save", "share", "imo", "imho", "fwiw", "ama", "tldr"]
combined_stop_words = list(ENGLISH_STOP_WORDS.union(custom_stop_words))
vectorizer_model = CountVectorizer(stop_words=combined_stop_words, min_df=5, ngram_range=(1, 2), token_pattern=r'\b\w{3,}\b', lowercase=True)

# representation model (improve keywords)
representation_model = KeyBERTInspired()

# Fit the BERTopic model
topic_model = BERTopic(verbose=True,
                calculate_probabilities=False,
                embedding_model=embedding_model,
                umap_model=umap_model,
                hdbscan_model=hdbscan_model,
                vectorizer_model=vectorizer_model,
                representation_model=representation_model)

topics, probs = topic_model.fit_transform(df.comment, embeddings=embeddings)

INFO: I'm running this on wsl on windows 11 and I'm using rapids cuml for UMAP and HDBSCAN. bertopic version is 0.16.4

[MaartenGr]

When I'm using the original UMAP and HDBSCAN models, I can easily reproduce the results. As such, it might be related to your cuML variants. Which version of cuML are you using? How did you install it? Have you tried it with the latest version?

[pooria98]

My cuML version is 24.12.00

I just followed rapids official installation guide and copy pasted the code it recommended for installing stable version in conda environment with cuda 12-12.5 and python 12

on windows 11 -> started WSL (Ubuntu 24.04.1 LTS) -> installed miniconda3 -> created a new env with python 3.12.8 and rapids packages

[pooria98]

Also another weird thing related to this, is the topic coherence results. I use "c_v" coherence metric from OCTIS.
My dataset is about AI related discussions in reddit.
when bertopic finds around 10 reasonable topics (about AI, automation, job replacements, hardware and vram, ...) that make sense, the c_v coherence score is mostly very low (around 0.3).
when bertopic becomes unstable in second run and results in only 2 big topics (one of which is the outliner topic and the other is AI), the coherence score is always very high (around 0.6).

should I ignore coherence metrics when I feel like the topics are good based on my knowledge of the dataset?

from octis.evaluation_metrics.coherence_metrics import Coherence

topicss = topic_model.get_topics()
top_words = [list(dict(topic).keys())[:10] for topic_id, topic in topicss.items() if topic_id != -1]  # Exclude -1 (outlier topic)
corpus = df.comment
vectorizer_model.fit(corpus)
tokenized_corpus = vectorizer_model.inverse_transform(vectorizer_model.transform(corpus))
tokenized_corpus = [list(doc) for doc in tokenized_corpus]

coherence_metric = Coherence(
    texts=tokenized_corpus,
    topk=10,
    measure='c_v'  # Other options: 'u_mass', 'c_v', 'c_uci', 'c_npmi'
)

model_output = {"topics": top_words}

coherence_score = coherence_metric.score(model_output)
print(f"Coherence Score: {coherence_score}")

[MaartenGr]

It might help to make sure you install the latest version of cuML. I remember they have pushed many updates (some related to fixing states).

when bertopic finds around 10 reasonable topics (about AI, automation, job replacements, hardware and vram, ...) that make sense, the c_v coherence score is mostly very low (around 0.3).
when bertopic becomes unstable in second run and results in only 2 big topics (one of which is the outliner topic and the other is AI), the coherence score is always very high (around 0.6).

should I ignore coherence metrics when I feel like the topics are good based on my knowledge of the dataset?

That's exactly why I generally do not recommend focusing on coherence metrics. If you dive a bit into the actual formulate you might notice that the fewer topics you get the higher the coherence likely will become. As such, it becomes a misleading metric (although by itself it isn't) when you compare different number of topics to one another.

With topic modeling, I will always urge you to validate the model yourself (with whatever metrics relate to the use case) and together with a domain expert.

[pooria98]

Thanks for the answer!

I ultimately decided to use (PCA + Kmeans) instead of (UMAP + HDBSCAN) as they result in more stable topics specially for my big dataset.

[MaartenGr]

Awesome, glad to hear that different models worked better. Great example of how the UMAP+HDBSCAN won't work for everyone.