Experiments with Kanseki Repository and sentencepiece

Install

pip install -r requirements.txt

bash scripts/makecorpus.sh

UPDATE: Since GitHub rejected the corpus files as too large, they are not included here. Instructions to create the corpus in scripts/mkcorpus.py

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
data		data
model		model
ngram		ngram
rep		rep
scripts		scripts
text		text
Readme.org		Readme.org
eval.txt		eval.txt
kr2titles0.txt		kr2titles0.txt
krp_ngram.py		krp_ngram.py
krp_ngram_a.ipynb		krp_ngram_a.ipynb
krp_sp.py		krp_sp.py
makecorpus.py		makecorpus.py
ngram-hist-30000.png		ngram-hist-30000.png
requirements.txt		requirements.txt
sp2.ipynb		sp2.ipynb
sp_notebook.ipynb		sp_notebook.ipynb
sp_text.ipynb		sp_text.ipynb
sp_text_np-play.ipynb		sp_text_np-play.ipynb
sp_text_np.ipynb		sp_text_np.ipynb