Skip to content

yuchensu1/Bm25Vectorizer

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 

Repository files navigation

BM25的Vectorizer

基于sciket-learn的TfidfVectorizer,继承修改得到的Bm25Vectorizer 使用方法和sciket-learn的方法完全相同

from BM25Vectorizer import Bm25Vectorizer
from sklearn.feature_extraction.text import TfidfVectorizer


if __name__ == '__main__':
    # format_weibo(word=False)
    # format_xiaohuangji_corpus(word=True)
    bm_vec = Bm25Vectorizer()
    tf_vec = TfidfVectorizer()
    # 1. 原始数据
    data = [
        'hello world',
        'oh hello there',
        'Play it',
        'Play it again Sam,24343,123',
    ]

    # 2. 原始数据向量化
    bm_vec.fit(data)
    tf_vec.fit(data)
    features_vec_bm = bm_vec.transform(data)
    features_vec_tf = tf_vec.transform(data)
    print("Bm25 result:",features_vec_bm.toarray())
    print("*"*100)
    print("Tfidf result:",features_vec_tf.toarray())
    

输出如下:

Bm25 result: [[0.         0.         0.         0.47878333 0.         0.
  0.         0.         0.         0.8779331 ]
 [0.         0.         0.         0.35073401 0.         0.66218791
  0.         0.         0.66218791 0.        ]
 [0.         0.         0.         0.         0.70710678 0.
  0.70710678 0.         0.         0.        ]
 [0.47038081 0.47038081 0.47038081 0.         0.23975776 0.
  0.23975776 0.47038081 0.         0.        ]]
*****************************************************
Tfidf result: [[0.         0.         0.         0.6191303  0.         0.
  0.         0.         0.         0.78528828]
 [0.         0.         0.         0.48693426 0.         0.61761437
  0.         0.         0.61761437 0.        ]
 [0.         0.         0.         0.         0.70710678 0.
  0.70710678 0.         0.         0.        ]
 [0.43671931 0.43671931 0.43671931 0.         0.34431452 0.
  0.34431452 0.43671931 0.         0.        ]]

Process finished with exit code 0

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%