Available on winget

Install Tokenizer

Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.

Install with winget

winget install --id Microsoft.Tokenizer

Upgrade

winget upgrade --id Microsoft.Tokenizer

Uninstall

winget uninstall --id Microsoft.Tokenizer

About Tokenizer

Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.

What's new in 1.3.3

No release notes provided.

Version history

Version	Updated	Notes
1.3.3	Unknown	No notes