tokenizers

PyPI v0.22.2

32,483,598 weekly downloads · 1 Dependencies

Source data as of:

Summary

Install pip install tokenizers

Registry values

Reproduced verbatim from the official registry, with the source named on each value.

Weekly downloads 32,483,598 Source: PyPI (Python Package Index)
Releases 97 Last release: 2026-01-05 Source: PyPI (Python Package Index)
Dependencies 1 Source: PyPI (Python Package Index)

Package details

Package
tokenizers
Registry
PyPI
Version
0.22.2
Weekly downloads
32,483,598 (weekly)
Dependencies
1: huggingface-hub
Releases
97
Last release
2026-01-05
First published
2019-11-01
Keywords
NLP, tokenizer, BPE, transformer, deep learning
Repository
https://github.com/huggingface/tokenizers

Why a field shows "—": A dash means the value is not published by the source registry for that package. Quanteta never fills in a guessed value — a missing field stays blank.

Derived indices (computation method published)

Quanteta-computed from the registry values below. This is a derived index, not a measured registry metric. See the formula on the Data & Sources page.

Q-Vitality Quanteta 44.3 / 100 Maintenance activity index (release cadence + download level).
Q-Trust Quanteta 8.5 / 100 Adoption / stability index (community size + download stability + age).
Q-Risk Quanteta 46.3 / 100 Dependency-surface index. Higher = more risk factors detected.

Data & Sources

Related packages

Other packages sharing a factual facet, for cross-reference. A descriptive neighbourhood — ordering does not imply a recommendation.

Same ecosystem

boto3 PyPI 476,910,084 packaging PyPI 343,175,305 urllib3 PyPI 333,295,410 requests PyPI 322,684,829

Shared keyword

nltk PyPI 12,601,434 mystem3-promise npm 157 wink-embeddings-sg-100d npm 80 nvidia-nccl-cu12 PyPI 10,439,070
← Back to all packages