Ruby bindings to the gigatoken core crate: BPE and SentencePiece tokenization at GB/s.
Required Ruby Version
>= 3.3, < 4.1.dev
Authors
Eric Jacobs
Versions
- 0.2.0 August 11, 2026 (3.02 MB)
- 0.2.0 August 11, 2026 x86_64-linux (14.6 MB)
- 0.2.0 August 11, 2026 arm64-darwin (11.5 MB)
- 0.2.0 August 11, 2026 aarch64-linux (12.3 MB)
- 0.1.1 July 24, 2026 (304 KB)
Pushed by
