RubyGems Navigation menu

ruby_llm-tokenizer 0.1.0

Pure-Ruby facade over Hugging Face `tokenizers` and OpenAI `tiktoken_ruby` that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).

Gemfile:
=

install:
=

Versions:

  1. 0.1.0 June 05, 2026 (14.5 KB)

Runtime Dependencies (2):

tiktoken_ruby ~> 0.0.9
tokenizers ~> 0.5

Owners:

Pushed by:

Authors:

  • Sal Scotto

SHA 256 checksum:

=

Total downloads 254

For this version 254

Version Released:

License:

MIT

Required Ruby Version: >= 3.3.0

New versions require MFA: true

Version published with MFA: true

Links: