initialization with empty corpus results in ZeroDivisionError

[mattf]

rank-bm25==0.2.2

In [11]: rank_bm25.BM25(corpus=[])
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Cell In[11], line 1
----> 1 rank_bm25.BM25(corpus=[])

File .../lib/python3.11/site-packages/rank_bm25.py:27, in BM25.__init__(self, corpus, tokenizer)
     24 if tokenizer:
     25     corpus = self._tokenize_corpus(corpus)
---> 27 nd = self._initialize(corpus)
     28 self._calc_idf(nd)

File .../lib/python3.11/site-packages/rank_bm25.py:52, in BM25._initialize(self, corpus)
     48             nd[word] = 1
     50     self.corpus_size += 1
---> 52 self.avgdl = num_doc / self.corpus_size
     53 return nd

ZeroDivisionError: division by zero

[dorianbrown]

Why do you want to initialize with an empty corpus? An error seems like a good thing to do here in my opinion, but I'm curious to hear about your use case.

[mattf]

iirc, developing a RAG pipeline that used hybrid search. rank-bm25 was buried under a few frameworks, app -> framework+ -> rank-bm25, making the error difficult to interpret and patch.

[dorianbrown]

In this case I think it might be best for me to implement a custom exception (ie EmptyCorpusError), and then it would be easier to catch with a try/except. Then your stack can address what to do with that exception. Does that sound useful?

Within the scope of this project I don't see any functional use cases for empty corpuses currently.

[mattf]

that sounds like a great solution.

[dorianbrown]

If you want, you take a look at this PR with the change: #45

It's pretty small, but probably a good to have an extra pair of eyes take a look at it.