Stop word removal with vectorizer_model

[zmarkofsky]

Hey so using the package I noticed there appears to be no ability to remove stop words from the topic model being trained. This leads to some issues as if the initial tuning is done with stop words and then stop words are later removed from the final BERTopic model via the .update_topics() method, the results end up being very different.

I'm unsure where the best place would be to fit this in, but if we could have some place to pass in a vectorizer_model similar to how it's done in the BERTopic Documentation, I believe it would allow for much more robust topic generation using only the most meaningful words within the docs.

[drob-xx]

@zmarkofsky Great question. So I'm happy to get into the design considerations I factored into TopicTuner but my short answer would be to just get the clustering you want, export to BERTopic and do the stop word removal there. Am I missing something?

[zmarkofsky]

Yeah that is basically what I'm currently doing, however that does lead to the topics dramatically shifting as expected. I was more wondering if having the stop words being factored in during the tuning process would lead to better tuning in the end as it would be able to better cluster the embeddings. However I'm no expert on how the underlying math of the algorithm operates so maybe it wouldn't have much of an effect on finding the ideal parameters.

[drob-xx]

I'm not sure what you mean by the topics shifting. The clustering should be the same. Could you share the code and point out what you would like to see operate differently?

[drob-xx]

Closed due to inactivity.