Unsupervised text tokenizer and detokenizer that supports encoding, decoding, and training of SentencePiece models. Provides a Python wrapper with features comparable to Hugging Face Tokenizers and OpenAI's tiktoken.

sentencepiece

sentencepiece-0.2.1-cp310-cp310-manylinux_2_38_aarch64.whl

sentencepiece-0.2.1-cp311-cp311-manylinux_2_38_aarch64.whl

sentencepiece-0.2.1-cp312-cp312-manylinux_2_38_aarch64.whl

sentencepiece-0.2.1-cp313-cp313-manylinux_2_38_aarch64.whl

sentencepiece-0.2.1-cp314-cp314-manylinux_2_38_aarch64.whl

sentencepiece-0.2.1-cp310-cp310-manylinux_2_38_x86_64.whl

sentencepiece-0.2.1-cp311-cp311-manylinux_2_38_x86_64.whl

sentencepiece-0.2.1-cp312-cp312-manylinux_2_38_x86_64.whl

sentencepiece-0.2.1-cp313-cp313-manylinux_2_38_x86_64.whl

sentencepiece-0.2.1-cp314-cp314-manylinux_2_38_x86_64.whl