Last night, OpenAI unveiled GPT-4o, along with announcing a new tokenizer that utilizes 20 source languages to compress data, leading to increased token efficiency. Despite Thai not being among the 20 languages, experiments show that Thai language tokens are compressed equally effectively.
The GPT-4o tokenizer can clearly identify words or word parts in Thai, such as “ของ” or “จำนวน,” as a single token immediately. This is in contrast to the GPT-4 tokenizer, which struggles to group multiple characters in Thai together, resulting in similar token and character counts.
The GPT-4o API costs remain the same, and with Thai benefiting from token savings, overall usage costs could potentially decrease by up to a quarter.
Source: HuggingFace: The Tokenizer Playground
TLDR: OpenAI introduced GPT-4o and a new multilingual tokenizer, improving token efficiency for various languages including Thai. The tokenizer can accurately identify Thai words and reduce token usage, potentially lowering overall usage costs by 25%.
Leave a Comment