ruby_llm-tokenizer 0.1.0
Pure-Ruby facade over Hugging Face `tokenizers` and OpenAI `tiktoken_ruby` that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).
Gemfile:
=
install:
=
Versions:
- 0.1.0 June 05, 2026 (14.5 KB)
Runtime Dependencies (2):
tiktoken_ruby
~> 0.0.9
tokenizers
~> 0.5