AR Wikipedia corpus for rababa

Purpose

Plain-text ar Wikipedia lines used as the MLM pre-training corpus for rababa’s ar diacritization model.

For Arabic, this augments the gold Tashkeela fine-tune corpus with ~500,000 lines of unpointed Modern Standard Arabic prose.

For Hebrew, this is the unpointed source corpus for distillation — the rababa Modal distillation pipeline runs each line through the Dicta Nakdan API to produce pointed labels. The distilled result lives in a separate rababa-hebrew-distilled repo.

Source

Fetched from the wikimedia/wikipedia dataset on Hugging Face (ar config, 20231101 dump) via scripts/fetch_wiki_corpus.py in the main rababa repo.

License

Wikipedia text is © Wikipedia contributors, licensed under CC-BY-SA 4.0. This compiled corpus follows the same license.

Stats

Total lines: 500,000

Split	Lines
train	400,000
val	50,000
test	50,000

Layout

arwiki_train/train.txt
arwiki_val/val.txt
arwiki_test/test.txt