Knowledge · Data for AI · Clinical speech

Real, simulated, or synthetic: how to source clinical conversation data for AI

For most clinical speech programs, the strongest source is a hybrid. Simulated consultations recorded by practicing physicians, in realistic acoustic conditions, give you volume and coverage across languages and specialties. Real recordings belong where their permission chain is fully documented, and their best use is final evaluation. Synthetic data is for augmentation. What decides the choice is not how the audio sounds. It is whether you can show, hour by hour, where it came from and what you may do with it.

See our clinical speech data Jump to the checklist

The problem

Why does the label matter more than the audio?

Clinical conversation data is scarce for a structural reason: a recording of a real consultation is health data, and most of what exists at scale is proprietary. The authors of ACI-Bench (Scientific Data, 2023) observe that clinic encounter conversations are not routinely recorded and are difficult to share ethically because of patient confidentiality, and the comparison table in their paper shows that the majority of similar dialogue corpora are proprietary and cannot be shared for community evaluation. The team behind a dataset of simulated respiratory interviews (Scientific Data, 2022) calls access to medical conversations a major roadblock, citing confidentiality and the regulations that limit data sharing, and the authors of PriMock57 make the same point for primary care.

Scarcity pushes buyers toward a market where the vocabulary is loose. One vendor case study, for example, describes 2,000 hours of role-played conversations between practicing physicians and patients as synthetic data. Of course, nobody is trying to mislead anyone with a word. But under the taxonomy we use at Pangeanic, those recordings are simulated, because people made them, and the difference matters: a machine-generated dialogue and a physician-recorded consultation carry different rights, different acoustic realism, and different failure modes. The label on a clinical dataset is a legal and technical claim, and it deserves the same scrutiny as the audio.

THE 3 CLASSES

One label per hour of audio

Real: a recording of an actual consultation or telemedicine visit.
Simulated: a consultation acted out by people from a clinical case brief and recorded as unscripted speech.
Synthetic: a conversation generated by a machine, as text from a language model, audio from a speech synthesizer, or both.

Pangeanic labels every delivered hour with one of these 3 classes.

The public record

What does the public record of clinical conversation data look like?

A 2026 literature survey on ambient clinical scribes counted the patient-clinician conversation datasets it could find and classified each one by provenance. The breakdown is the clearest snapshot we know of.

patient-clinician conversation datasets identified

simulated, mostly actor- or clinician-generated consultations

real, but text from online medical forums rather than live encounters

from real clinical encounters (a hospital dataset), and not public

One dataset in the survey has unspecified provenance, and 12 of the 13 are publicly accessible. Read the counts as a snapshot of the public literature, not of the private market. The survey was written about ambient scribes in India, and it counts datasets of every kind, from actor-recorded consultations to text scraped from forums. But the pattern agrees with what the ACI-Bench authors describe: the open resources are mostly simulated or forum text, and real clinical recordings sit behind closed doors.

Comparison

How do real, simulated, and synthetic data compare?

Simulated consultations are not a workaround invented by vendors. The OSCE respiratory interview dataset was recorded by resident doctors and senior medical students on Microsoft Teams, produced 272 simulated cases averaging just under 12 minutes, and went through manual transcript correction and quality control. ACI-Bench itself was created with domain experts to model 3 ways clinicians document encounters. What varies from one simulated dataset to the next is the design, which is the subject of the next section.

Class	What you must clear	Strongest use	Typical failure	Ask the vendor for
Real	The collection basis, a consent or other legal basis that covers AI training, de-identification of audio and transcript, and the right to license onward.	Final evaluation and acceptance testing under live clinical conditions.	Rights that look clear but do not cover training, and voices that remain identifiable after names are removed.	The permission chain in writing, dataset by dataset.
Simulated	Speaker consent and a rights assignment, as for any voice recording. No patient data is involved.	Training and fine-tuning at volume, balanced across languages, specialties, accents, and settings.	Role-play that sounds scripted, speakers who are not clinicians, and studio-clean audio.	Who spoke, how the cases were briefed, and how the acoustics were reproduced.
Synthetic	The license terms of the generating models and the provenance of any seed data.	Augmentation of rare terminology, drug names, and edge cases, with human review.	Missing turn-taking, disfluency, and room acoustics, and generator errors that propagate into trained models.	The generator, the seed data, and the share of the dataset it represents.

Simulation design

What separates well-designed simulated consultations from role-play that sounds scripted?

Simulation is a method, and methods vary. A consultation read from a script teaches a model how people read, not how consultations sound. The difference shows up in 4 places. The first is the speaker: a practicing physician knows the questioning routine, the terminology, and the points at which a real consultation turns. The second is the case brief, which should fix the facts (complaint, history, patient profile, outcome) and never the wording. The third is the recording setting, which should match where the model will be deployed: a consulting room, a telephone line, or a video call. The fourth is the acoustic environment, because clinics are not quiet.

Setting claims can be checked from the signal instead of from the vendor's description. A narrowband telephone channel nominally carries 300 to 3,400 Hz, so a patient on a phone line and a physician in a consulting room leave measurably different spectra above roughly 4 kHz. If a dataset is sold as telemedicine, ask whether the setting was verified that way. See how we produce simulated clinical consultations

CHECKLIST

6 signs of well-designed simulated data

Practicing clinicians in the clinician role.
Case briefs that fix facts, not wording.
Unscripted dialogue in the speakers' own words.
Recording settings that match deployment (room, phone, video).
Background noise that reflects real clinical environments.
A written transcription guideline and a quality-control record.

Testing

How do you test whether simulated data works for your task?

How far simulated speech transfers to real consultations depends on the task, the language, and the setting, so the only answer that counts is one measured on your own system. For us at Pangeanic, simulated consultations are a core offer, which is exactly why you should weigh our view accordingly and run this test before you scale.

Reserve a real evaluation set

De-identified, with rights confirmed, and never used in training. Even a small set is enough to make the comparison honest.

Train with and without

Fine-tune one model with the simulated data and keep a baseline trained without it, under the same settings.

Evaluate on the real set

Use the metric that matches your task: word error rate for speech recognition, diarization error rate for speaker separation, note quality for scribes.

Break results down

Report by language, specialty, and recording setting, so a good average cannot hide a weak segment.

Rights and compliance

What does the permission chain for real recordings have to show?

Health data is a special category of personal data under Article 9 of the GDPR, and a consultation recording contains more than words: the voice itself can identify the speaker. The European Data Protection Board's guidelines on virtual voice assistants treat voice used to identify a person as biometric data, and they recommend checking that anonymization really makes a voice unidentifiable before the data is treated as anonymous. So "de-identified" is a claim that needs a method behind it, for the audio as much as for the transcript.

Documentation matters as much as consent. For high-risk AI systems trained on data, Article 10 of the EU AI Act requires data governance practices that cover the data collection processes and the origin of the data, the original purpose of the collection when personal data is involved, and preparation operations such as annotation, labelling, and cleaning. Whether a given clinical product is high-risk is a classification each provider has to make, and nothing in this guide settles it. But a dataset that arrives without an account of its origin gives you nothing to put in that file.

Licenses deserve the same attention. Some public clinical speech datasets are free only for non-commercial use (the 2026 MeDial-Speech dataset, with more than 111 hours of robot-patient and doctor-patient dialogue, is one), which rules them out as training data for a commercial product no matter how good the audio is.

LINK 1

Collection basis

Who recorded the consultation, in which jurisdiction, and under which consent or other legal basis?

LINK 2

De-identification

Which identifiers were removed from the transcript and from the audio, by what method, and how was the result checked?

LINK 3

Onward licensing rights

Does the seller hold the right to license the data onward, and to whom, for how long, and in which territory?

LINK 4

A legal basis that covers AI training

Does the original consent or legal basis genuinely cover commercial AI training, or only the purpose the recording was made for?

Other jurisdictions run their own regimes (HIPAA in the United States and the DPDP Act in India are 2 examples), and the same 4 questions apply wherever the recording was made. We would rather be conservative than represent material as available before the rights are clear.

Picture the review. Six months after launch, a customer's procurement team asks where the training data for your scribe came from. You open the vendor file for the "real" recordings and find a delivery note, an invoice, and no consent record. The model is not the problem. The file is.

Synthetic data

Where does synthetic data fit?

Synthetic data earns its place at the edges: rare terminology, drug names, unusual presentations, and scenarios that would take months to record. It does not replace recorded speech, because generated dialogue tends to lack the turn-taking, disfluency, and room acoustics that speech systems meet in clinics, and because errors in a generator can propagate into every model trained on its output.

EU law points in the same direction on one narrow point. Article 10(5) of the AI Act allows providers of high-risk systems to process special categories of personal data for bias detection and correction only when that cannot be done effectively with other data, including synthetic or anonymised data. That is not a compliance claim, but it shows that regulators treat synthetic and anonymized data as the first option wherever it suffices. For clinical conversations, we recommend labeling every synthetic hour as such and capping its share of the dataset in the specification.

Language and context

Why does language change the answer?

Clinical conversations are not the same everywhere. The 2026 survey of ambient clinical scribes in India reports that Indian consultations differ from those in the Global North in their brevity, their triadic (multi-speaker) structure, their code-mixing, and their noise. A scribe trained on English two-party consultations will not meet that structure by chance, and a translated script will not reproduce it either. Simulated data can, but only if the clinicians who record it conduct consultations the way they are conducted locally, in their own language and register, and only if evaluation is done natively rather than through translated English proxies.

Pangeanic has produced simulated clinical interactions for multilingual speech and conversational AI programs covering French, Italian, German, Spanish, Ukrainian, Russian, and Dutch. Our experience with sensitive medical data in European programs includes MAPA (CEF Action 2019-EU-IA-0013), the multilingual anonymization toolkit for public administrations that Pangeanic coordinated, and NTEU, the neural machine translation project for European public administrations.

WHAT THIS GUIDE DOES NOT CLAIM

This guide is not legal advice, and it makes no claim that simulated consultations perform identically to real ones: that is an empirical question for your task and your languages. The counts come from a single survey with its own scope. And Pangeanic sells simulated clinical conversation data, so we have a commercial interest in the answer. The test above exists so that you do not have to take our word for it.

FAQ

Clinical conversation data: frequently asked questions

What is the difference between simulated and synthetic clinical conversation data?

Simulated data is recorded by people who act out a consultation from a clinical case brief, so the speech is human even though the encounter is staged. Synthetic data is generated by a machine, as text from a language model, audio from a speech synthesizer, or both. The 2 carry different rights, different acoustic realism, and different failure modes, so a dataset should state which one it is for every hour.

Does simulated clinical conversation data contain patient health data?

No patient is involved in a simulated consultation, so it contains no patient health information and no patient consent chain. The speakers' voices are still personal data, so each speaker's consent and a rights assignment covering AI training and evaluation remain necessary.

Can I train a commercial model on a public clinical conversation dataset?

Only if its license allows it. Some public clinical speech datasets are free only for non-commercial use, for example the 2026 MeDial-Speech dataset, so check the license before the audio quality. Also check whether the dataset documents its origin, because that documentation is what a provider needs for its own data governance file.

What does the EU AI Act require for training data?

For high-risk AI systems trained on data, Article 10 requires data governance practices covering, among other things, the data collection processes and the origin of the data, the original purpose of the collection for personal data, and preparation operations such as annotation, labelling, and cleaning. Whether a particular clinical AI system is high-risk is a classification each provider must make, so this answer is not a compliance opinion.

How can I tell whether a vendor's audio is real, simulated, or synthetic?

Ask for the label per recording, not per dataset, together with who the speakers were, how the case was briefed, and what consent or rights assignment covers each speaker. You can also check the recording setting from the signal itself: a narrowband telephone channel, for example, shows almost no energy above roughly 4 kHz, which a consulting-room microphone does not.

What share of a clinical speech dataset should be real?

There is no fixed share: it depends on your task, your languages, and the rights you can document. We recommend reserving real, de-identified recordings with confirmed rights for evaluation, using simulated consultations for training volume and coverage, and limiting synthetic data to augmentation. Then measure the result on your own task with a held-out real set.

Continue from here

SOURCES

Yim, W. et al. (2023). ACI-BENCH: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Scientific Data 10. doi.org/10.1038/s41597-023-02487-3

Fareez, F. et al. (2022). A dataset of simulated patient-physician medical interviews with a focus on respiratory cases. Scientific Data. doi.org/10.1038/s41597-022-01423-1

Korfiatis, A. P. et al. (2022). PriMock57: a dataset of primary care mock consultations. arxiv.org/abs/2204.00333

Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data (2026), Table 1. arxiv.org/abs/2609.17355

Cuayáhuitl, H. and Jang, G. (2026). A dataset of robot-patient and doctor-patient medical dialogues for spoken language processing tasks (MeDial-Speech). arxiv.org/abs/2605.26747

European Data Protection Board (2021). Guidelines 02/2021 on virtual voice assistants, version 2.0. edpb.europa.eu

Regulation (EU) 2016/679 (GDPR), Article 9. eur-lex.europa.eu/eli/reg/2016/679/oj

Regulation (EU) 2024/1689 (AI Act), Article 10. AI Act Service Desk, European Commission

Vendor case study cited for terminology: Enhancing Clinical Ambient Intelligence with Synthetic Patient Physician Conversations. shaip.com

MAPA, Multilingual Anonymisation Toolkit for Public Administrations. pangeanic.com/mapa

Need clinical conversation data with the label, the rights, and the evidence file?

Send your specification or your request for information. We reply with sample recordings, a draft datasheet, and a written answer on provenance class, consent, and license scope.

Request samples and a datasheet See clinical speech data