Implementation of today's most used tokenizers
Screenshots
-
tokenizers-0.19.1
~amd64
test debug python_single_target_python3_10 python_single_target_python3_11 python_single_target_python3_12 debug
View 
Download 
Browse License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016
-
tokenizers-0.15.2-r1
~amd64
test debug python_single_target_python3_10 python_single_target_python3_11 python_single_target_python3_12 debug
View Download Browse License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016
-
tokenizers-0.14.1-r1
~amd64
test debug python_targets_python3_10 python_targets_python3_11 python_targets_python3_12 debug
View Download Browse License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016
ChangeLog
commit 561517911880f6e2d57705e7f9745bdeb93fa20a
Author: Alex Efros <powerman@powerman.name>
Date: Mon Aug 19 19:31:28 2024 +0300
sci-libs/tokenizers: new package, add 0.19.1
