What Is a Low-Resource Language?

01 · The concept

A data condition, not a linguistic verdict

The term was born inside natural language processing (NLP) and computational linguistics, and it carries the bias of its birthplace: it measures a language by what a machine can learn from it. A language with a thousand-year literary tradition, a rich morphology, and millions of speakers can still be "low-resource" if very little of it exists as clean, licensed, machine-readable text or transcribed speech. Conversely, a language with fewer speakers but decades of public investment in corpora, spell checkers, treebanks, and translation memories may not be low-resource at all. Speaker numbers and digital resources are two different axes, and AI only sees the second one.

The most useful academic framing comes from Hedderich and colleagues (NAACL 2021), who describe a low-resource scenario rather than a low-resource language. The scenario is defined along 3 dimensions: the availability of labeled data for the specific task, the availability of unlabeled text in the language or domain, and the availability of auxiliary data (dictionaries, related languages, knowledge bases). This is why the same language can be well served for spell checking and badly served for speech recognition, or strong in news text and weak in clinical notes. Indeed, even English becomes a low-resource scenario in a narrow technical domain with no annotated data. Haddow and colleagues (Computational Linguistics, 2022) add a time dimension for machine translation: some language pairs considered scarce a decade ago now have far more data thanks to web crawling and the use of monolingual text.

Of course, the label has a cost. Some researchers point out that "low-resource" defines a language by what it lacks and can hide the knowledge, practices, and priorities of its speakers. I return to that critique later, because it has practical consequences for how data should be collected. For now, the working definition matters because procurement, research funding, and model release notes all use it, often without saying which of the 3 dimensions they mean.

04 · Consequences

Why do AI systems fail in low-resource languages?

Large language models learn from what the web contains, and the web is linguistically lopsided. Blasi, Anastasopoulos, and Neubig (ACL 2022) documented systematic performance inequalities across the world's languages in machine translation, question answering, reading comprehension, speech synthesis, parsing, and morphological inflection, and showed that the gaps track economic, demographic, and academic factors rather than linguistic difficulty. Multilingual models extend coverage, but sharing one set of parameters across hundreds of languages does not guarantee equivalent results for each of them. Meta's No Language Left Behind project (NLLB-200, published in Nature in 2024) scaled neural machine translation to 200 languages and around 40,000 translation directions precisely by combining transfer learning with large-scale data mining, and its own evaluation shows how uneven quality remains across that range.

The subtler failure is the gap between nominal coverage and effective utility. A language can appear on a model's list of supported languages and still perform far below its neighbors, or have been evaluated only on a narrow register such as Bible text or Wikipedia. Without reliable, human-built benchmarks you cannot even tell. That is why evaluation data is not a luxury for low-resource languages: it is the only instrument that separates a claim from a capability.

06 · Spain as a laboratory

Is Catalan a low-resource language? Catalan, Valencian, Basque, and Galician

Spain is one of the best places in the world to see how relative the label is, because four languages with very different histories coexist inside one state and one research system. None of them fits a single category, and each has become a showcase of a different strategy.

Catalan: well organized, still fragmentary

Catalan has a strong digital community (Softcatalà, open corpora, spell checkers, free software) that researchers describe as a case of digital resilience. Yet ELE's 2022 report on the Catalan language, authored by Maite Melero, Blanca C. Figueras, Mar Rodríguez, and Marta Villegas, placed its overall technological support in the "fragmentary" category, with large differences between areas such as text, speech, and machine translation. The Generalitat de Catalunya responded with Projecte AINA, developed by the Barcelona Supercomputing Center (BSC), which has released open corpora and models for Catalan. In April 2025 the Spanish government released the ALIA family through BSC, a public, open infrastructure for Spanish and the co-official languages; its Salamandra models were trained on 35 European languages under an Apache 2.0 license.

For us at Pangeanic this is not an abstract case study. We have worked with BSC on customised datasets of bilingual segments classified by domain and style (with particular emphasis on Catalan), on data annotation with quality control in our PECAT platform, on reinforcement learning from human feedback with reward model data, and on multilingual hate speech detection datasets to reduce bias in language models. The full scope is described in our Barcelona Supercomputing Center use case.

Valencian: the variety inside the language

Resource scarcity is not uniform within a language. The same ELE report recognizes that in the Valencian Community the traditional name of the language is valencià, and it inventories Valencian-specific resources such as the Corpus Informatitzat del Valencià and the Corpus Toponímic Valencià, while warning that several corpora are small, restricted by license, or lack comparable evaluation sets. AINA's CATalog dataset states openly that most of its material is central Catalan and that Valencian and Balearic text is being actively added to reduce variety bias. In speech, the VIVES project published a corpus of more than 270 hours from the Corts Valencianes, conceived partly to compensate for the scarcity of Valencian speech data. A system trained mostly on central Catalan can formally support Catalan and still offer weaker coverage of Valencian or Balearic forms.

This is also where speech data meets public service. We have provided AI transcription for the Valencian regional parliament since 2021, and in 2024 we were awarded the contract to introduce AI transcription in the Spanish Congress, which covers interventions in Catalan, Galician, and Basque alongside Spanish. Parliamentary speech is exactly the kind of long-form, multi-speaker, code-switched audio that exposes resource gaps between varieties.

Basque: an isolate that built its own model

Basque has no close linguistic relatives, so it cannot lean on cross-lingual transfer the way Galician leans on Portuguese or Valencian on the rest of Catalan. The HiTZ Center (University of the Basque Country) answered with Latxa, a family of open models from 7 to 70 billion parameters pretrained on a new Basque corpus of 4.3 million documents and 4.2 billion tokens, released with 4 new evaluation benchmarks built from proficiency exams, reading comprehension, trivia, and public examinations. The paper received the Best Resource Paper award at ACL 2024. The lesson for everyone else: the evaluation suite was as important a contribution as the model.

Galician: transfer from a large neighbor

Galician illustrates the opposite strategy. Proxecto Nós, led by CiTIUS at the University of Santiago de Compostela with the Xunta de Galicia, has produced Galician corpora and open generative models (Carballo among them), including a Galician-Portuguese bilingual model that exploits the proximity of the two languages. Proximity helps, but it has a known side effect: models drift toward the larger neighbor's spelling and vocabulary unless native-speaker data and evaluation pull them back.

07 · Beyond Europe

Africa, the Americas, and Asia

Africa: where most of the diversity is

Africa concentrates a very large share of the world's linguistic diversity, and it remains thinly represented in corpora and models. A review of 884 African NLP papers published between 2020 and 2025 (Alabi, Hedderich, Adelani, and Klakow, EMNLP 2025) describes sustained growth in research, with very uneven coverage across languages, tasks, and regions. Masakhane, founded in 2019 as a distributed research community, changed the method as much as the output: its participatory machine translation study (Nekoto et al., 2020) produced data and systems for more than 30 African languages with local researchers and speakers in charge. The obstacles are not only volume. For Medumba, spoken in Cameroon, researchers documented problems with writing and encoding methods, the absence of standardized technical terminology, scarce digital resources, and limited funding. When we collect African datasets for AI training, code-switching and real acoustic conditions (markets, transport, rural settings) turn out to matter as much as vocabulary.

The Americas: Indigenous languages on their own terms

In Latin America, the technological position of Spanish and Portuguese contrasts sharply with that of Indigenous languages. The difficulty rarely comes from a small number of speakers: it comes from missing digital corpora, orthographies that are not always stable, and scarce segmentation tools, parallel data, recorded speech, and evaluation sets (Mager et al., COLING 2018). The AmericasNLP workshop, started in 2021 within the Association for Computational Linguistics, organized the first shared task on open machine translation for 10 Indigenous languages paired with Spanish: Aymara, Bribri, Asháninka, Guaraní, Wixarika, Nahuatl, Hñähñu, Quechua, Shipibo-Konibo, and Rarámuri. Later editions expanded to more languages, to educational materials, and to translation metrics. In 2026 a community-centered project built speech recognition resources for Puno Quechua from dozens of hours of recordings and transcriptions, with the community involved in design and collection. That model of work (communities setting goals and validating outputs) is becoming the standard, not the exception.

Asia: scripts, segmentation, and scale

Asia shows that low-resource does not mean small. Bhojpuri appears in Joshi's class 1 despite a speaker base of tens of millions; Konkani and Lao sit in class 2; Cebuano and Indonesian in class 3. Here the gaps are often technical before they are statistical: scripts written without spaces between words (Lao, Thai, Khmer) break naive tokenizers, and closely related languages are frequently mislabeled as the dominant one, which inflates apparent coverage for the larger language and erases the smaller one. My reading, and it is an inference rather than a published measurement, is that the next wave of useful Asian language AI will come from carefully segmented, correctly labeled speech and text collected in-country, not from larger crawls. That is the principle behind our Southeast Asian datasets for AI training.

09 · The critique

Is "low-resource" the right label?

The NLP community still uses the term as standard, but its critics have sharpened what it hides. Steven Bird argues that the label lumps together very different situations (standardized languages without technological support, local languages of mainly oral tradition, contact languages) whose social functions and technological expectations are not comparable, and he asks whether NLP must be extractive at all (ACL 2024). Treating linguistic data as raw material to be mined can reproduce colonial relationships and sideline community authorities. Mager, Mager, Kann, and Vu (ACL 2023) interviewed community leaders, teachers, and activists and concluded that including speakers is essential for useful and ethical machine translation of Indigenous languages, which points toward linguistic data sovereignty: communities keep control over how the materials they contribute are used to train AI.

I agree with the critique, and I think it has a commercial translation that the industry has been slow to accept. Data whose provenance, consent, and license cannot be documented is a liability, and in low-resource languages that liability is concentrated, because the same few sources get reused everywhere. Ethical sourcing is not a tax on low-resource AI; it is the only way to build datasets that survive audit. This is why we insist on a human-in-the-center model, where native speakers are not a validation afterthought but the people who decide what "correct" means for their variety.

11 · Our work

How we approach low-resource languages at Pangeanic

Pangeanic began by building and processing bilingual data for machine translation systems, long before "training data" became a market category. That history taught us that the hard part of a low-resource language is never the model architecture: it is finding, licensing, cleaning, aligning, and validating the data, and then proving with evaluation that the result works for the people who speak it. Today that work runs through our Data for AI services (sourcing, collection, annotation, RLHF, and evaluation data), our parallel corpora [CONFIRMAR URL canónica], and our AI evaluation practice, with native-speaker validation at each step. The same data discipline sits behind our Deep Adaptive AI Translation, which adapts output to a client's terminology and style, a need that is sharper, not softer, in languages where generic engines have seen little text.

If you want a longer view of where machine translation still struggles, I wrote about the languages that defy machine translation from our production experience.