Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

causal_lm: migrate to string tensors #92

Merged

Conversation

Wovchena
Copy link
Collaborator

Merge after #74 and #81

@ilya-lavrenov
Copy link
Contributor

I suppose changes in thirdparty folder should also trigger SD CI https://github.com/openvinotoolkit/openvino.genai/blob/master/.github/workflows/stable_diffusion_1_5_cpp.yml#L4, because SD also leverages tokenizers and should migrate to string tensors

Instead of moving to string tensors, remove tokenization completely because I don't see a use of it. Moreover setting --with_detokenizer to False would have resulted in failure without my removal. I don't see the use for prints like `print("Load HF Tokenizer ...")` neither, but I leave it for another day, because it's not affected by string tensors.
@Wovchena Wovchena changed the base branch from master to releases/2023/3 December 21, 2023 08:32
@Wovchena Wovchena requested a review from FionaZZ92 December 21, 2023 08:33
@Wovchena Wovchena marked this pull request as ready for review December 21, 2023 08:33
@ilya-lavrenov ilya-lavrenov merged commit f06b970 into openvinotoolkit:releases/2023/3 Dec 21, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants