sci-libs/tokenizers
Implementation of today's most used tokenizers
Runtime Dependencies
tokenizers-0.19.1
python_single_target_python3_10?
( dev-lang/python:3.10 )
python_single_target_python3_11?
( dev-lang/python:3.11 )
python_single_target_python3_12?
( dev-lang/python:3.12 )
tokenizers-0.15.2-r1
python_single_target_python3_10?
( dev-lang/python:3.10 )
python_single_target_python3_11?
( dev-lang/python:3.11 )
python_single_target_python3_12?
( dev-lang/python:3.12 )
tokenizers-0.14.1-r1
python_targets_python3_10?
( dev-lang/python:3.10 )
python_targets_python3_11?
( dev-lang/python:3.11 )
python_targets_python3_12?
( dev-lang/python:3.12 )