All pull requests

Pull requests

• fix(parallelism): honor RAYON_NUM_THREADS, expose set_num_threads in Python

  #2546·nov3o opened Oct 17, 2026·

  ·

• fix(decoders): Strip panics on short tokens made only of content

  #2545·spicysharkco opened Oct 16, 2026·

  ·

• feat(python): update PyO3 and rust-numpy to 0.30 and test Python 3.15(t)

  #2544·ngoldbaum opened Oct 16, 2026Contributor·

  ·

• feat(python): Support Python 3.15 via abi3t wheels

  #2543·ngoldbaum opened Oct 16, 2026Contributor·

  ·
  1

• perf(encode): encode one long input on every thread, and stream a file from Python with encode_file

  #2542·nov3o opened Oct 16, 2026·

  ·

• perf(bpe trainer): a small parallel trainer (a compact take on #2501)

  #2541·nov3o opened Oct 16, 2026·

  ·

• Restore special token masks in v1 encodings

  #2540·starx710 opened Oct 16, 2026·

  ·
  1

• Preserve token text in v1 encodings

  #2539·starx710 opened Oct 16, 2026·

  ·
  1

• fix(wordpiece): vocabs with empty tokens

  #2538·SBrandeis opened Oct 15, 2026Collaborator·

  ·
  1

  1

• chore: bump 2 deps

  #2537·SBrandeis opened Oct 15, 2026Collaborator·

  ·
  1

• fix(bindings): Enable the fancy-regex backend in all bindings

  #2536·SBrandeis opened Oct 15, 2026Collaborator·

  ·
  1

  1

• Restore vocabulary lookups for v1 tokenizers

  #2535·tuzengji opened Oct 15, 2026·

  ·
  1

• Restore Python DecodeStream for the v1 pipeline

  #2532·starx710 opened Oct 15, 2026·

  ·
  1

• Expose token-string decoding in Rust and Python

  #2531·starx710 opened Oct 15, 2026·

  ·
  1

• Fix Metaspace prefix before normalized added tokens

  #2529·starx710 opened Oct 14, 2026·

  ·
  1

• chore(tk-encode): drop the legacy Encoding

  #2509·SBrandeis opened Oct 14, 2026Collaborator·

  ·
  1

• perf(normalizers): Improve unicode and BERT normalizers

  #2508·SBrandeis opened Oct 13, 2026Collaborator·

  ·

  · v1.0.0
  1

• perf(bindings): speed up Python decode for NumPy int32 and int64 inputs

  #2507·kozistr opened Oct 13, 2026·

  ·

• fix(metaspace): keep internal spaces in the first decoded token

  #2506·aalquwayfili opened Oct 11, 2026·

  ·

• docs: correct canonical Metaspace prepend type

  #2505·j45856021-dev opened Oct 11, 2026·

  ·

• docs: correct v1 Unigram file API status

  #2504·original4422 opened Oct 11, 2026·

  ·
  3

• docs: add Simplified Chinese translation (README.zh.md)

  #2503·JasonYeYuhe opened Oct 10, 2026·

  ·
  1

• perf(bpe trainer): 10–32× faster with 8 threads

  #2501·Yikai-Liao opened Oct 9, 2026·

  ·
  5

• Share the pipeline worker pool across concurrent encode calls

  #2500·ZenAlexa opened Oct 8, 2026·

  ·
  1

  2

• feat(wasm): publish pipeline

  #2497·SBrandeis opened Oct 8, 2026Collaborator·

  ·

  · v1.0.0
  1