Knowledge · Data for AI · Provenance and licensing

Data provenance and licensing for AI training data

Data provenance is the documented record of where a dataset came from, who created it, how it was collected and processed, and under which permissions. Licensing is the set of rights that says what you may do with it. A dataset is safe to train on only when the 2 match: every record traces back to a source whose permissions cover your intended use, and the license you sign passes those permissions on to you in writing. This page explains how to check both before you buy.

8 questions to ask a vendor Browse datasets for AI

The evidence

Why can't you trust the license label on a dataset?

Because independent audits keep finding that the label and the underlying rights disagree. The Data Provenance Initiative, a collaboration of legal and machine learning researchers, has traced thousands of public datasets back to their sources. These are its headline findings.

1,800+

text datasets audited in the first study (Nature Machine Intelligence, 2024)

70%+

license omission rate on popular dataset hosting sites

50%+

error rate in the license categories that were listed

80%+

of source content in widely used text, speech, and video datasets carries non-commercial restrictions (ICLR 2025)

The fourth number is the one buyers miss. In the follow-up audit of nearly 4,000 text, speech, and video datasets, fewer than 33% of the datasets themselves were restrictively licensed, yet more than 80% of the source content inside them carried non-commercial restrictions. A dataset can display a permissive license while the material it was built from does not allow commercial use at all. The same study found that multimodal datasets have turned overwhelmingly to web-crawled, synthetic, and social media sources since 2019, which is exactly where the chain of permissions is hardest to reconstruct.

Of course, these audits cover public datasets, not the commercial market, and a restriction at the source does not settle any legal question by itself. But they show how the problem arises: rights are lost one repackaging at a time. The license on a dataset is only as good as the provenance behind it, and provenance is something a vendor either documented at collection time or cannot reconstruct later.

Key terms

What is the difference between provenance, lineage, consent, and licensing?

The 4 terms are often used as if they meant the same thing. They answer different questions, and a dataset can pass one and fail another.

TERM 1

Data provenance

The verifiable history of a dataset: its sources, its creators, the consent or license under which each part was obtained, and every processing step between collection and delivery.

TERM 2

Data lineage

The record of what happened to the data after it was obtained: filtering, cleaning, annotation, anonymization, merging, and versioning. Lineage is the processing part of provenance.

TERM 3

Consent and legal basis

The permission given by the people in the data, or another lawful ground, for a stated purpose. It governs personal data, and it binds every later holder of the dataset to that purpose.

TERM 4

Data licensing

The contract through which a rights holder grants a buyer defined uses of a dataset: which uses, by whom, where, for how long, and with which restrictions and warranties.

Sources

How does the source of the data change what you must check?

Every dataset reaches a buyer through one of 5 routes, or a mix of them. The European Commission's template for the public summary of training content uses a similar grouping (public datasets, licensed private data, web-scraped content, user data, and synthetic data), so a vendor that cannot tell you the mix is giving you nothing to report.

Source class	What it is	What you must check	Evidence to ask for
Commissioned collection	Data recorded or written for the project by recruited participants.	That each participant's consent and rights assignment cover AI training and commercial use.	The consent form template, the participant agreement, and the collection protocol.
Licensed from a rights holder	Existing content licensed from a publisher, archive, company, or institution.	That the licensor owns the rights it is granting, and that the grant allows onward licensing to you.	The upstream license scope in writing, including sublicensing and AI training.
Public and open-licensed datasets	Datasets published under open or research licenses.	The license of the dataset and of its sources: non-commercial, share-alike, and attribution terms travel with the data.	The license of every component, traced to the original source.
Web-collected content	Content gathered from websites and platforms.	Copyright and any rights reservation by the rights holder, site terms, and personal data inside the content.	The list of domains, the collection dates, and how opt-outs were respected.
Synthetic data	Data generated by a model.	The license terms of the generating model and the provenance of the seed data it was prompted with.	The generator, the seed data, and the share of the dataset it represents.

The permission chain

What is the permission chain, and where does it break?

The permission chain is the unbroken sequence of rights from the person or organization that created the data to the model that is trained on it. It has 4 links, and a dataset is only as usable as its weakest one.

LINK 1

Collection basis

Who collected the data, where, and under which consent, contract, or other legal basis? It breaks when nobody kept the record.

LINK 2

Privacy treatment

Which personal data does the dataset contain, what was anonymized or removed, and by what method? It breaks when "anonymized" is a label and not a documented process.

LINK 3

Onward licensing rights

Does the seller hold the right to license the data to you? It breaks when a reseller passes on data it only had permission to use itself.

LINK 4

A purpose that covers AI training

Does the original permission cover training and commercial deployment, or only the purpose the data was created for? It breaks when consent for one use is stretched to another.

Most failures happen at links 3 and 4, and they are invisible in the data itself. A recording sounds the same whether or not the speaker agreed to AI training. A document reads the same whether or not the archive that sold it had the right to. That is why provenance cannot be tested from a sample: it has to be shown on paper. If a vendor answers a rights question with a sample, the question has not been answered.

The license

Which license terms decide what you can do with a dataset?

A data license is not a formality that follows the technical evaluation. It defines the product. These are the 8 terms we recommend reading before the datasheet.

Term	The question it answers	What to look for
Permitted use	May you train, fine-tune, evaluate, and deploy commercially?	Each use named explicitly. "Research use" or "internal use" does not include a commercial product.
Exclusivity	Can the vendor sell the same data to your competitors?	Exclusive, non-exclusive, or exclusive for a period or a field of use.
Term and territory	For how long and where may you use the data?	Perpetual or time-limited, and what happens to trained models when the term ends.
Model and output rights	Who owns the models trained on the data and what they produce?	A clear statement that trained models and their outputs are yours, with no claim by the licensor.
Sublicensing and affiliates	May group companies, contractors, or customers use the data?	Named rights for affiliates and for processors working on your behalf.
Inherited restrictions	Do attribution, share-alike, or non-commercial terms come with the data?	A list of every upstream condition, or a statement that there are none.
Withdrawal and deletion	What happens if a participant withdraws consent or a rights holder objects?	A defined process for removal and replacement, and who bears the cost.
Warranties and indemnity	Who stands behind the rights if they are challenged?	A warranty that the vendor holds the rights it grants, backed by an indemnity it can honor.

Regulation

What do EU rules ask you to document about training data?

Three instruments matter most, and they ask for the same thing from different angles: an account of where the data came from. For providers of general-purpose AI models, Article 53 of the EU AI Act has applied since 2 August 2025. It requires a policy to comply with EU copyright law and a public summary of the content used for training, written on the template the Commission's AI Office published on 24 July 2025. For high-risk AI systems, Article 10 requires data governance practices that cover the data collection processes and the origin of the data. After the Digital Omnibus on AI (Regulation (EU) 2026/1744), those high-risk obligations apply from 2 December 2027 for Annex III systems and from 2 August 2028 for systems covered by Annex I.

Copyright adds a third angle. Under Article 4 of the Copyright in the Digital Single Market Directive, rights holders can reserve their works from text and data mining, for online content by machine-readable means, so web-collected data needs a record of how reservations were respected. And where a dataset contains personal data, the GDPR applies to it whatever the license says. Other jurisdictions run their own regimes, and the questions on this page apply wherever the data was collected.

The practical point for a buyer is simple. None of these rules can be met at the end of a project by the team that trained the model. The information they ask for exists only if the data supplier recorded it at the start.

AT A GLANCE

3 EU instruments, 1 question

AI Act, Article 53: a copyright policy and a public summary of training content for general-purpose AI models.
AI Act, Article 10: data governance for high-risk systems, including collection processes and data origin.
Copyright Directive, Article 4: rights holders may reserve content from text and data mining.

Whether and how these rules apply to your system is a legal determination. This page is not legal advice.

Documentation

What should a dataset arrive with?

A dataset should arrive with an evidence file: the set of documents that lets a third party verify its provenance and rights without asking the vendor again. The research community has converged on a few formats for it. Datasheets for Datasets (Gebru et al., 2021) proposed a standard set of questions about a dataset's motivation, composition, collection process, and recommended uses. Croissant, the metadata format maintained by MLCommons on top of schema.org, makes license, source, and structure machine-readable, with a Responsible AI extension for provenance and collection details. The Data & Trust Alliance's Data Provenance Standards, now being taken through the OASIS Open standards process, define cross-industry metadata for source, lineage, and permitted use.

None of these formats is mandatory, and none of them makes a dataset lawful by being filled in. Their value is that they force the questions to be answered in writing, by the party that knows the answers, at the time the answers are still known.

THE EVIDENCE FILE

6 documents to expect with a dataset

Datasheet: composition, languages, volumes, collection method, and known limits.
Source record: the source class of every component and its share of the total.
Consent and rights templates: the participant consent form or the upstream license scope.
Processing log: cleaning, annotation, anonymization, and versioning steps.
Quality report: the sampling method, the checks applied, and the results.
License: permitted uses, restrictions, term, and warranties in one document.

Buyer's checklist

What should you ask an AI data vendor about provenance and licensing?

These 8 questions fit in one email. A vendor with documented provenance can answer them in writing within days. A vendor that answers with a sample, a brochure, or a call has told you what you needed to know.

ORIGIN

Questions 1 to 4

Source: which source class does each part of the dataset belong to, and in what proportion?
Collection: who collected the data, in which countries, and when?
Consent: can we see the consent form or the upstream license, and does it name AI training and commercial use?
Privacy: which personal data does it contain, and how was it anonymized or removed?

RIGHTS

Questions 5 to 8

Ownership: do you hold the rights you are licensing, or are you reselling someone else's data?
Restrictions: which upstream conditions (non-commercial, share-alike, attribution) come with it?
Exclusivity: who else has licensed this data, and can we have it exclusively?
Warranty: do you warrant the rights in the contract, and what happens if a participant or rights holder objects?

Our practice

How does Pangeanic document provenance and licensing?

For us at Pangeanic, provenance is part of the specification, not a report written after delivery. Before a dataset is quoted, we confirm its source class, the rights we hold, and the uses we can license, and we would rather be conservative than represent material as available before the rights are clear. Where a requirement cannot be met with documented rights, we say so and propose a route that can, such as a commissioned collection with consent written for AI training.

The exact framework depends on the dataset, the jurisdiction, the source, and the intended commercial use. Depending on the program, it can include provenance documentation, participant permissions, permitted uses and licensing conditions, privacy assessment, masking or exclusion workflows, validation documentation, and controlled delivery. This discipline comes from more than 2 decades of building language resources, including European public-sector programs such as MAPA, the multilingual anonymization toolkit for public administrations that Pangeanic coordinated.

WHAT THIS PAGE DOES NOT CLAIM

This page is not legal advice, and documented provenance does not by itself make any use of a dataset lawful: that depends on your system, your jurisdiction, and your contract. The audit figures describe public datasets studied by the Data Provenance Initiative, not the commercial data market. Pangeanic sells AI training data, so we have a commercial interest in buyers asking these questions. They are written so that you can put them to us as well.

FAQ

Data provenance and licensing: frequently asked questions

What is data provenance in AI?

Data provenance is the verifiable history of a dataset: its sources, its creators, the consent or license under which each part was obtained, and every processing step between collection and delivery. In AI, it is what lets a model developer show where training data came from and that its use was permitted.

What is the difference between data provenance and data lineage?

Provenance covers the whole history of a dataset, including its origin and the permissions attached to it. Lineage is the narrower record of what happened to the data after it was obtained, such as filtering, annotation, anonymization, and versioning. Lineage is the processing part of provenance.

Can I use an open-licensed dataset to train a commercial AI model?

Only if the license of the dataset and the licenses of its sources allow commercial use. An audit of nearly 4,000 public text, speech, and video datasets found that fewer than 33% were restrictively licensed, while more than 80% of their source content carried non-commercial restrictions. Check the license of every component, traced to the original source.

What should an AI training data license include?

At a minimum: the permitted uses named explicitly (training, fine-tuning, evaluation, and commercial deployment), exclusivity, term and territory, ownership of trained models and their outputs, rights for affiliates and contractors, any inherited restrictions, a process for withdrawal and deletion, and a warranty that the vendor holds the rights it grants.

What does the EU AI Act require providers to disclose about training data?

Since 2 August 2025, Article 53 requires providers of general-purpose AI models to have a policy to comply with EU copyright law and to publish a summary of the content used for training, following the template issued by the Commission's AI Office. For high-risk AI systems, Article 10 requires data governance practices that cover the data collection processes and the origin of the data. This answer is general information, not legal advice.

Who is liable if licensed training data turns out to infringe someone's rights?

It depends on the contract and the jurisdiction, which is why warranties and indemnities matter. A license should state that the vendor holds the rights it grants and what the vendor will do if a participant or rights holder objects. A vendor that will not warrant its rights in writing is asking the buyer to carry the risk.

Does synthetic data remove the need for provenance?

No. Synthetic data has provenance of its own: the model that generated it, the license terms of that model, and the seed data it was prompted with. A dataset should state which records are synthetic, how they were generated, and what share of the total they represent.

Guides and services that build on this page

SOURCES

Longpre, S. et al. (2024). A large-scale audit of dataset licensing and attribution in AI. Nature Machine Intelligence 6. doi.org/10.1038/s42256-024-00878-8

Longpre, S. et al. (2025). Bridging the Data Provenance Gap Across Text, Speech, and Video. ICLR 2025. arxiv.org/abs/2412.17847

Gebru, T. et al. (2021). Datasheets for Datasets. Communications of the ACM 64(12). arxiv.org/abs/1803.09010

Akhtar, M. et al. (2024). Croissant: A Metadata Format for ML-Ready Datasets. arxiv.org/abs/2403.19546

Data & Trust Alliance. Data Provenance Standards. dataandtrustalliance.org

Regulation (EU) 2024/1689 (AI Act), Articles 10 and 53, as amended by Regulation (EU) 2026/1744. eur-lex.europa.eu/eli/reg/2024/1689/oj

European Commission (2025). Template for general-purpose AI model providers to summarise their training content. digital-strategy.ec.europa.eu

Directive (EU) 2019/790 on copyright in the Digital Single Market, Article 4. eur-lex.europa.eu/eli/dir/2019/790/oj

Regulation (EU) 2016/679 (GDPR). eur-lex.europa.eu/eli/reg/2016/679/oj

Send us the 8 questions

Tell us the data you need, or forward the checklist as it is. We reply in writing with the source class, the rights we hold, the license scope we can offer, and a draft datasheet.

Ask about dataset rights and licensing Browse datasets for AI