Skip to content

Commit

Permalink
Bump tokenizers submodule to latest master
Browse files Browse the repository at this point in the history
  • Loading branch information
@yatarkan
yatarkan committed Sep 13, 2024
1 parent 6a76c25 commit 029e018
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion thirdparty/openvino_tokenizers
Submodule openvino_tokenizers updated 6 files
+10 −9 src/CMakeLists.txt
+5 −5 src/bpe_tokenizer.cpp
+14 −0 src/patches/fast_tokenizers.patch
+1 −1 src/regex_split.hpp
+4 −4 src/utf8_validate.cpp
+12 −6 src/utils.cpp

0 comments on commit 029e018

Please sign in to comment.