AI training data
How to Accept or Reject Your First Batch of Multilingual AI Training Data
A supplier delivers the first batch on Friday. On Monday, someone must decide whether to release the production budget. I would want that person to be able to point to a particular row and explain what would happen if a model learned from it.
What makes a pilot batch acceptable?
Accept a versioned batch when its permitted use is documented, the promised languages and tasks are present, and independent review finds that each required group meets its agreed quality threshold. Record who approved it and which data the approval covers. If a critical defect appears, hold the affected records while you determine how far it extends.
In Valencia, the language on an administrative form can change between the heading and the instructions. A dataset can count that form as one document and still tell you very little about its suitability for teaching a model to answer in Valencian Catalan. What was the user trying to do? Which language should the reply use? Could an error change what the citizen believes they are obliged to provide? Those questions determine what the buyer needs to inspect.
If you are still evaluating bids, our guide to evaluating multilingual AI data services covers the evidence to request from a supplier. Here we pick up when the files arrive.
Agree on the release conditions before the files arrive
Write down the behavior you are buying. “Bulgarian administrative text” leaves considerable latitude. “English and Bulgarian sentence pairs that preserve who may, must, or must not perform an action in a public service procedure” gives the reviewer a task. It also tells the supplier what evidence will determine payment for the next stage.
| Item | What the buyer and supplier should agree |
|---|---|
| Unit and scope | A sentence pair, conversation, audio clip, or document. Specify language direction, locale, domain, intended use, and any minimum coverage for each. |
| Critical defect | Define consequences. For this pilot, reversing an obligation or losing a prohibition triggers a hold and a wider review of the affected group. |
| Tolerable variation | List acceptable variants and examples of minor defects. Decide whether the rate is measured per record or per annotation and who resolves disputed labels. |
| Release authority | Name the reviewer, the person who can authorize training, the deadline for rework, and whether individual groups may be released separately. |
A vague criterion such as “95% quality” leaves the buyer to discover its denominator after the invoice arrives. Count reviewed records, affected records, and decisions by severity. A row with three punctuation errors is one affected row if your contract measures defects per record.
How should you sample multilingual training data?
Start with the delivery manifest. Reconcile its counts against the files by language, direction, task, and source. Check IDs, empty fields, encoding, duplicates, and split assignments across the full delivery wherever feasible. Ask for the collection or license record that supports the intended use of each source. An unresolved source goes on hold before anyone samples its prose.
Then draw two samples. For the first, select records at random within the groups you have agreed to release separately. Save the selection method, seed where applicable, and IDs so someone else can reproduce the draw. For the second, deliberately select difficult cases. In this pilot, those include negation, deadlines, exceptions, and permission. Keep the results separate. A sample enriched for troublesome cases tells you what can go wrong. Its defect rate cannot estimate the rate in the delivery.
How many rows are enough?
Choose the sample according to the failure you need to detect. As a planning example, if independent random draws each have a 1% chance of containing a particular defect, 100 draws give about a 63% chance of seeing it at least once. At 300 draws, that probability rises to about 95%. The calculation is 1 − (1 − 0.01)n. It estimates detection under those assumptions. It cannot certify that a batch is clean.
Three hundred adjacent rows from one translated form could repeat the same translator’s decisions. Spread the draw across source documents, templates, and contributors where those distinctions are available. For a small group with a consequential risk, inspecting every record may be cheaper than arguing over a percentage.
A Bulgarian example worth holding the batch for
The following pairs are constructed for this article. They illustrate how I would document a review of parallel training data . Both administrative sentences were written for this example.
English source
The applicant may submit a copy.
Bulgarian delivered
Заявителят трябва да представи копие.
Bulgarian correction
Заявителят може да представи копие.
The delivered sentence tells the applicant they must submit a copy. Both Bulgarian sentences read naturally. A fluency score would give the purchaser little reason to stop. The reviewer records the defect as a change in obligation, retains the original pair ID, and searches for other statements of permission in the same source family.
English source
The applicant must sign the declaration.
Bulgarian delivered
Заявителят трябва да подпише декларацията.
Pair B should survive the repair intact. A global replacement of трябва with може would create precisely the defect we are paying to remove. Modal verbs are a small linguistic minefield. Even a search for English may needs judgment. “The office may be closed” expresses uncertainty. The reviewer must recover the sense of the source before changing the target.
My decision in this example would be to hold the affected source family, correct Pair A, and examine permission, obligation, and prohibition across that family. A second qualified reviewer should adjudicate disputed readings with the surrounding paragraph available. If the same defect occurs in another template, widen the hold. If the supplier can trace the failure to one template, the rest of the delivery can follow its own agreed release rules.
Accept, request rework, or reject the delivery
| Decision | Evidence and next action |
|---|---|
| Accept | The agreed checks pass for the named groups, source permissions are documented, and critical findings are closed. Record the version and authorize its specified use. |
| Request rework | A recoverable defect has a traceable scope. Hold the affected group. Require corrected IDs, a description of the cause, and a new version. Retest the known failures and draw fresh records from the repaired group. |
| Reject for the intended use | The source terms exclude the intended training use, or the supplier cannot separate affected records and repeated defects invalidate the agreed scope. Record the reason and request a replacement delivery under the contract. |
When the evidence is incomplete, record a hold with an owner and a review date. Missing paperwork and confirmed incompatible rights call for different decisions. If a test document appears in the training set, quarantine it and examine related versions, translations, and near duplicates across splits. A new record ID does remarkably little to disguise the same underlying document.
What belongs in the signoff?
Keep the batch version and checksum, accepted and held groups, sample IDs, reviewer decisions, unresolved exceptions, permitted use, and approving owner together. Link the corrected version to its predecessor. A datasheet for the dataset can describe its origins, composition, collection, and intended uses. The signoff adds the buyer’s decision on this particular delivery.
What should the buyer authorize next?
Approving the pilot permits a defined next step. It might release the purchase order for another 20,000 pairs from the same sources under the same instructions. Adding a language, switching to synthetic generation, or moving from administrative notices to medical instructions changes the basis for approval. Scope those changes explicitly and review the first affected delivery.
Keep a separate decision for model performance. If your investment depends on better Bulgarian answers, reserve an independent evaluation set, compare the baseline and adapted models under the same conditions, and have reviewers inspect the errors that affect users. Dataset acceptance establishes what you are willing to train on. The evaluation establishes what the model learned.
That distinction determines what to buy next. If reviewers find a lacuna in the approved batch, such as missing examples of exceptions, commission targeted collection. If reviewers disagree about the meaning of an instruction, revise the guidelines and recalibrate the team before ordering more annotation. If the data passes inspection and the model still mishandles obligations, investigate the training setup and evaluation before paying for another batch of the same material.
For teams commissioning a first pilot, our training and evaluation datasets page covers available data options. Our AI Data Operations page describes how collection, annotation, and review fit into delivery.
Buyer questions
Questions buyers ask before approving a pilot
What should a buyer check before accepting multilingual training data?
Check the agreed language and task scope, source permissions, delivery integrity, representative random sampling, targeted sampling of difficult cases, and the agreed quality thresholds. Record the exact version and the groups covered by the approval.
When should a multilingual training data pilot be rejected?
Reject a delivery for its intended use when the source terms exclude that use, affected records cannot be separated, or repeated critical defects invalidate the agreed scope. Hold a recoverable defect while its scope is investigated.
How large should a multilingual training data sample be?
The sample depends on the failure you need to detect, the size of each release group, and the consequence of an error. Use reproducible random sampling for estimates and a separate targeted sample for difficult cases.
Does accepting a dataset prove that the model improved?
No. Dataset acceptance records what the buyer is willing to train on. An independent evaluation set and a controlled comparison establish what the model learned.