A fast BPE tokenizer for use with OpenAI's models, allowing encoding and decoding of text to match specific model requirements like gpt-4o. It provides a simple interface to work with various encodings such as o200k_base.

tiktoken

tiktoken-0.14.0-1.1-cp310-cp310-manylinux_2_38_aarch64.whl

tiktoken-0.14.0-1.1-cp311-cp311-manylinux_2_38_aarch64.whl

tiktoken-0.14.0-1.1-cp312-cp312-manylinux_2_38_aarch64.whl

tiktoken-0.14.0-1.1-cp313-cp313-manylinux_2_38_aarch64.whl

tiktoken-0.14.0-1.1-cp314-cp314-manylinux_2_38_aarch64.whl

tiktoken-0.14.0-1.1-cp310-cp310-manylinux_2_38_x86_64.whl

tiktoken-0.14.0-1.1-cp311-cp311-manylinux_2_38_x86_64.whl

tiktoken-0.14.0-1.1-cp312-cp312-manylinux_2_38_x86_64.whl

tiktoken-0.14.0-1.1-cp313-cp313-manylinux_2_38_x86_64.whl

tiktoken-0.14.0-1.1-cp314-cp314-manylinux_2_38_x86_64.whl

tiktoken-0.13.0-2.2-cp310-cp310-manylinux_2_38_aarch64.whl

tiktoken-0.13.0-2.2-cp311-cp311-manylinux_2_38_aarch64.whl

tiktoken-0.13.0-2.2-cp312-cp312-manylinux_2_38_aarch64.whl

tiktoken-0.13.0-2.2-cp313-cp313-manylinux_2_38_aarch64.whl

tiktoken-0.13.0-2.2-cp314-cp314-manylinux_2_38_aarch64.whl

tiktoken-0.13.0-2.2-cp310-cp310-manylinux_2_38_x86_64.whl

tiktoken-0.13.0-2.2-cp311-cp311-manylinux_2_38_x86_64.whl

tiktoken-0.13.0-2.2-cp312-cp312-manylinux_2_38_x86_64.whl

tiktoken-0.13.0-2.2-cp313-cp313-manylinux_2_38_x86_64.whl

tiktoken-0.13.0-2.2-cp314-cp314-manylinux_2_38_x86_64.whl