wikipedia wikimedia
Dataset Card for Wikimedia Wikipedia Dataset Summary Wikipedia dataset containing cleaned articles of all languages. The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/) with one subset per language, each containing a single train split. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections (references, etc.). All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/wikimedia/wikipedia.
- 类型
- dataset
- 许可
- cc-by-sa-3.0
- 语言
- ab
- 下载量
- 228,662
- 点赞
- 1,369
- 访问
- public
- 文件
- 0
标签
- 预训练语料
- 多语言语料
- 文本数据集
- 文本生成
- NLP
- 预训练
摘要
该数据集整理了维基百科各语言版本的清洗后文章，源自维基百科dumps，每种语言一个子集，每篇条目包含去除引用等非正文内容后的完整文章。它覆盖300多种语言，规模从千词级到千万级不等，适合作为大规模多语言语料用于语言模型预训练、掩码语言建模及文本生成等任务。
README
--- language: - ab - ace - ady - af - alt - am - ami - an - ang - anp - ar - arc - ary - arz - as - ast - atj - av - avk - awa - ay - az - azb - ba - ban - bar - bbc - bcl - be - bg - bh - bi - bjn - blk - bm - bn - bo - bpy - br - bs - bug - bxr - ca - cbk - cdo - ce - ceb - ch - chr - chy - ckb -…