tokenizers

PyPI v0.23.1

48,689,869 weekly downloads · 1 Dependencies

Recently published

Source data as of:

Summary

Install pip install tokenizers

Registry values

Reproduced verbatim from the official registry, with the source named on each value.

Weekly downloads 48,689,869 Source: PyPI (Python Package Index)
Releases 100 Last release: 2026-04-27 Source: PyPI (Python Package Index)
Dependencies 1 Source: PyPI (Python Package Index)

Package details

Package
tokenizers
Registry
PyPI
Version
0.23.1
Weekly downloads
48,689,869 (weekly)
Dependencies
1: huggingface-hub
Releases
100
Last release
2026-04-27
First published
2019-11-01
Keywords
NLP, tokenizer, BPE, transformer, deep learning
Repository
https://github.com/huggingface/tokenizers

Why a field shows "—": A dash means the value is not published by the source registry for that package. Quanteta never fills in a guessed value — a missing field stays blank.

Derived indices (computation method published)

Quanteta-computed from the registry values below. This is a derived index, not a measured registry metric. See the formula on the Data & Sources page.

Q-Vitality Quanteta 78.0 / 100 Maintenance activity index (release cadence + download level).
Q-Trust Quanteta 8.5 / 100 Adoption / stability index (community size + download stability + age).
Q-Risk Quanteta 43.2 / 100 Dependency-surface index. Higher = more risk factors detected.

Data & Sources

Related packages

Other packages sharing a factual facet, for cross-reference. A descriptive neighbourhood — ordering does not imply a recommendation.

Same ecosystem

boto3 PyPI 782,306,107 packaging PyPI 469,732,079 certifi PyPI 398,890,207 urllib3 PyPI 397,909,429

Shared keyword

nltk PyPI 15,115,109 mystem3-promise npm 156 wink-embeddings-sg-100d npm 53 nvidia-nccl-cu12 PyPI 11,369,469
← Back to all packages