tiktoken
Subscription: Developer
Active
Subscription: Developer
A fast BPE tokenizer for use with OpenAI's models, allowing encoding and decoding of text to match specific model requirements like gpt-4o. It provides a simple interface to work with various encodings such as o200k_base.
pip install tiktoken==0.13.0
0.13.0
Linux