discourse_ai-tokenizers 0.4.2

A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.

Gemfile:
=

install:
=

Versions:

0.4.2 February 27, 2026 (20.6 MB)
0.4.1 February 26, 2026 (20.6 MB)
0.4 January 06, 2026 (20.6 MB)
0.3.2 December 10, 2025 (20.6 MB)
0.3.1 July 07, 2025 (20.6 MB)

Show all versions (10 total)

Runtime Dependencies (3):

activesupport >= 6.0

tiktoken_ruby ~> 0.0.15

tokenizers ~> 0.6.3

Development Dependencies (2):

rubocop-discourse ~> 3.8

syntax_tree ~> 6.2.0

Show all transitive dependencies

Owners:

Pushed by:

Authors:

Rafael Silva

discourse_ai-tokenizers 0.4.2

Gemfile:
=

install:
=

Versions:

Runtime Dependencies (3):

Development Dependencies (2):

Owners:

Pushed by:

Authors:

SHA 256 checksum:

Total downloads 145,996

For this version 12,089

Version Released:
February 27, 2026 1:12pm

License:
MIT

Required Ruby Version: >= 3.2.0

Links:

discourse_ai-tokenizers 0.4.2

Gemfile: =

install: =

Versions:

Runtime Dependencies (3):

Development Dependencies (2):

Owners:

Pushed by:

Authors:

SHA 256 checksum:

Total downloads 145,996

For this version 12,089

Version Released: February 27, 2026 1:12pm

License: MIT

Required Ruby Version: >= 3.2.0

Links:

Gemfile:
=

install:
=

Version Released:
February 27, 2026 1:12pm

License:
MIT