tryAGI/Tiktoken

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

⭐ 87 ⑂ 7 C# MIT · 4 天前推送

https://www.nuget.org/packages/Tiktoken/

ai csharp dotnet gpt4o high-performance huggingface openai sdk tokenizer tiktoken zero-allocation bpe cl100k-base o200k-base

87

Watchers

0

贡献者

0

Commits

0

Releases

1

Open Issues

4 天前

最近推送

原文中文

暂无 README