Clustering and Recommender system of Physics literature taken from the Kaggle arXiv dataset.
GitHub Large File System: git lfs
Python 3.9+
Install required packages
pip install -r requirements
-
Download
arxiv-metadata-oai-snapshot.json
from the Kaggle arXiv dataset to the same directory. -
Run
clustering.ipynb
to create dataset and encode text using Sentence Transformers models or TfidfVectorizer.
After the dataset and models have been saved from running clustering.ipynb
, run recommend.ipynb
and in the last cell enter the link to the arXiv paper (eg http://www.arxiv.org/abs/1602.03837).
The number of similar papers to be recommended can be set using the num_papers
variable.
Output will be stored in output.txt
. A sample output file is provided.
- Host Recommender on Website
- Try Gaussian Mixture Modeling
- Use more sophisticated embeddings