tf_sentencepiece is s going to be deprecated in tensorflow 2.3.0. tf_sentencepiece for tensorflow 2.2.0x is the last release of tf_sentencepiece. Use tensoflow_text to run sentencepiece on tensorflow.
Example
import tensorflow as tf
import tensorflow_text as text
model = open('test_model.model', 'rb').read()
s1 = text.SentencepieceTokenizer(model=model)
print(s1.tokenize(['hello world']))
print(s1.tokenize_with_offsets(['hello world']))
s2 = text.SentencepieceTokenizer(model=model, out_type=tf.dtypes.string)
print(s2.tokenize(['hello world']))
print(s2.tokenize_with_offsets(['hello world']))SentencePiece TensorFlow module implements the encode (text to id/piece) and decode (id/piece to text) operations which are executed lazily on top of TensorFlow's Session mechanism. This module allows to make an end-to-end training/inference computatation graph by directly feeding raw sentences with the tf.placeholder. The SentencePiece model (model proto) is passed as an attribute of the TensorFlow operation and embedded into the TensorFlow graph so the model and graph become purely self-contained.
For Linux (x64), macOS environment:
% pip install tf_sentencepiece
Use pydoc to see the usage instruction
% pydoc sentencepiece_processor_ops