removing tensorflow_text for aarch64 compatiblity #883
Conversation
rdyro
force-pushed
the
rdyro-remove-tftxt
branch
3 times, most recently
from
646f83b
to
08378df
Compare
|
I'm not sure why Unit Test / Common test (v4-8) (pull_request) is failing |
rdyro
force-pushed
the
rdyro-remove-tftxt
branch
2 times, most recently
from
afd0e23
to
1f7ce19
Compare
rdyro
force-pushed
the
rdyro-remove-tftxt
branch
from
1f7ce19
to
bb1666a
Compare
| for k in data_keys: | ||
| if isinstance(tokenizer, TikTokenTokenizer): | ||
| features[k] = tf.py_function(_process_string, [features[k]], Tout=[tf.int32])[0] | ||
| elif isinstance(tokenizer, SentencePieceTokenizer): | ||
| features[k] = tokenizer.encode(features[k]) | ||
| features[k] = tf.py_function(_process_string, [features[k]], Tout=[tf.int32])[0] |
There was a problem hiding this comment.
Do we still need an if-elif statement here?
| for k in data_keys: | ||
| if isinstance(tokenizer, TikTokenTokenizer): | ||
| features[k] = tf.py_function(_process_string, [features[k]], Tout=[tf.int32])[0] | ||
| elif isinstance(tokenizer, SentencePieceTokenizer): | ||
| features[k] = tokenizer.encode(features[k]) | ||
| features[k] = tf.py_function(_process_string, [features[k]], Tout=[tf.int32])[0] |
There was a problem hiding this comment.
Did you check the performance here? Using a py_function for TikToken affected the performance and that's why the recommendation is to use pygrain/hf with tiktoken
There was a problem hiding this comment.
Ok, the performance consideration is something I completely missed. From some of my tests it's either tf.py_function or SentencePiece from sentencepiece that acquires the GIL and significantly impacts performance.
I looked into a bit, but for me even calling an empty tf.py_function throttles the loader down a lot (4x-6x).
Do you have any ideas? @khatwanimohit
No description provided.