What Does an Earnings-Call Transcript Leave Out?

A roadmap for a new series on speech, voice cloning, and the information hidden in how we speak.

A research desk with an earnings-call transcript aligned to an audio waveform and spectrogram

An earnings call can last an hour. Its transcript can be searched, summarized, and fed to a language model in seconds.

Why, then, would anyone still listen to the recording?

The obvious answer is that a transcript preserves the words but discards the voice. The less obvious questions are what exactly the voice adds, whether we can measure it, and whether it changes how a listener interprets an answer.

Imagine that an executive’s answer looks reassuring on the page but sounds hesitant when spoken. Is that a real signal absent from the transcript? Is it simply our subjective impression? Is a machine responding to the words, the tone, the speaker, or the bad conference-call codec? Most importantly, does hearing the delivery cause anyone to revise their judgment?

Over the next few weeks, I will publish a series of pedagogical posts about the research that my colleague Hamdan Al Ahbabi and I have been conducting in the context of his PhD thesis on multimodal AI at Khalifa University, under my co-supervision.

This introductory post is Post 0: a map of where we are going, why we started, and the rather pedestrian steps required before one can claim to have learned anything from an audio recording.

N.B. This is independent academic research, unrelated to my professional responsibilities. It is presented for educational purposes and should not be interpreted as investment advice. The project is ongoing. The series will include failed experiments, data problems, null results, and changes of direction, not only the parts that make attractive demonstrations and publishable papers.

Table of Contents

  1. A long-standing side interest in audio
  2. Starting from a public multimodal dataset
  3. The research question
  4. The five posts to come
  5. What would count as an answer?

A long-standing side interest in audio

This project did not begin with voice cloning.

My first exposure to audio signal processing goes back to my Master’s in Applied Mathematics and Machine Learning, the MVA, where I followed Emmanuel Bacry’s course on audio signal processing. I did not become a speech specialist, but this curiosity stayed with me.

Years later, shortly before the 2022 French presidential debate, I returned to that subject for a small “real life” experiment: who was speaking, and when, during the 2017 Macron and Le Pen debate? The tools were rudimentary by today’s standards: Mel-frequency cepstral coefficients, Gaussian mixture models, and a little manually selected audio. Nevertheless, the post already ended with questions about tone, aggressiveness, interruption and hesitation.

In 2025, this interest met with Hamdan’s multimodal-AI research focus. We explored how speech embeddings mix together what is said with how it is said. In Disentangling Speech Embeddings, we tried to regress the textual component out of a speech representation. In Serious, Sassy, or Sad?, we asked whether those residual representations made vocal tone easier to classify. This work became our paper on Residual Speech Embeddings for Tone Classification.

The new series continues that progression, but on substantially messier data and with a more difficult question. We are no longer asking only whether a model can classify a synthetic speaking style. We want to understand whether authentic vocal delivery contains information beyond an earnings-call transcript, and whether that information affects interpretation.

Starting from a public multimodal dataset

We did not record thousands of earnings calls ourselves. We started from FinCall-Surprise, a multimodal benchmark published at ACL 2026 by Dong Shu, Yanguang Liu, Huopu Zhang and Mengnan Du.

The dataset contains 2,688 earnings conference calls from 2019 to 2021, with transcripts, presentation slides and full-call audio. The transcripts are available from the project’s GitHub repository, while the larger MP3 and presentation archives can be downloaded from the authors’ Google Drive folder.

That sounds like the difficult part has already been solved. We have text. We have audio. The transcripts mark analysts and executives. What remains but to run a few models?

Quite a lot, as it turns out.

For each voice-cloning example, we need two recordings of the same executive. The first is the voice reference: a short, clean sample from the prepared remarks that the cloning model uses as an anchor for reproducing the target speaker’s voice. The second is a real answer given by that executive later in the Q&A. We provide the model with the transcript of this answer and ask it to synthesize the same words in the voice anchored by the reference clip. The analyst’s question supplies the conversational context. Together, the voice reference, the question, and the real answer form the triplet at the centre of our experiment.

A one-hour call may contain an operator, several executives, and several analysts. A transcript role such as “Executives” does not identify which executive is speaking. The prepared-remarks reference may therefore contain the CFO, while the Q&A answer may come from the CEO, even though both clips are labelled “Executives” and their transcript entries look coherent. Approximate word timestamps can also cut off the final sound of a sentence or include the next person’s “thank you.” A short reference clip supposed to contain a single speaker may even contain a handoff between two speakers.

Before studying delivery, we first have to know exactly which words were spoken, where they begin and end, how many people are in each clip, and whether the same person appears in both places. We initially moved too quickly from the source dataset to model experiments. Human listening then revealed references with several speakers, incomplete answers, poor segment boundaries, and reference-answer pairs spoken by different people. Those problems invalidated the experiments built from that version of the data. We therefore discarded those results and rebuilt the source-selection and review process. This work deserves an entire post because it explains why reliable audio research starts with careful listening and data construction, before any model comparison or statistical analysis.

The research question

Our broad question is simple:

Is there useful information in an earnings-call recording that is not present in its transcript?

But this question hides three progressively stronger claims.

First, existence: after accounting for the words, is there reproducible information left in the waveform?

Second, interpretation: can we characterize what remains as valence, certainty, hesitation, composure, guardedness, fluency, or some other paralinguistic dimension?

Third, consequence: does this vocal delivery influence what a human or a machine concludes about the answer, including its responsiveness, credibility, evasiveness, or economic implications?

These are not equivalent. An audio model outperforming a text model does not automatically prove that it discovered meaningful delivery. It may recognize the speaker, the company, the microphone, the call position, or a segmentation error. Likewise, calling an acoustic measurement “sentiment” does not make it sentiment.

The causal question is harder still. Bad news may simultaneously cause negative words, a strained delivery, a difficult analyst question and a negative listener reaction. Observational data entangles all of them.

Ideally, we would hear the same person give the same answer twice, changing only the delivery. The original call cannot provide that counterfactual. Modern voice cloning may allow us to approximate it: preserve the answer’s words and apparent speaker, vary one aspect of its delivery, randomly expose listeners to different versions, and observe whether their judgments change.

We will first benchmark several voice-cloning models to establish whether any of them can reproduce the correct words in a voice sufficiently close to the reference speaker. This comparison is interesting in itself, but its main purpose in our research is methodological. If the benchmark shows that a model is good enough, we can use it as an experimental instrument: the model can produce several versions of the same answer, with the same words and apparent speaker but different deliveries. We can then test whether these controlled changes affect how listeners interpret the answer.

The five posts to come

The series will follow the research in the order in which the problems must be solved.

Post 1: Is it worth listening when we have the transcript?

We will separate words, speaker identity, delivery, conversational dynamics and recording channel. We will review what “audio beyond text” could mean, what it cannot mean, and why subtracting a text-model score from an audio-model score is not a causal analysis.

This post will introduce the principal confounders and the counterfactual we would actually like to observe.

Post 2: Building a reliable audio triplet is surprisingly expensive

Our basic data point contains three pieces:

  1. a clean, full-sentence reference from an executive’s prepared remarks;
  2. a later analyst question;
  3. a clean answer spoken by the same executive.

We will explain how we transcribed, aligned, segmented, clustered, compared and finally listened to these clips. We will show the human-review forms, the nine quality gates, the failure modes that automatic scores missed, and the amount of skilled human time hidden behind a supposedly “ready-to-use” dataset.

This will include tools such as Whisper, Resemblyzer and CAMPPlus. More importantly, it will show why none of them can certify the complete data point on its own.

Post 3: Eight voice cloners enter an earnings call

Once the source clips are genuinely clean, we can finally compare modern open-weight voice-cloning systems on equal inputs.

We will introduce Qwen3-TTS, IndexTTS2, CosyVoice3, Chatterbox, Higgs Audio, OpenVoice and F5-TTS; explain zero-shot cloning; and distinguish several questions that are often collapsed into one score:

  • Did the model say the correct words?
  • Does it sound like the reference person?
  • Does it sound natural?
  • Does it contain one stable speaker?
  • Can a blinded listener find the authentic recording?

There will be winners, non-finishers, close calls, and at least one surprising result: an excellent speaker-similarity score can conceal serious failures to reproduce the requested content.

Post 4: Same words, same apparent speaker, different delivery

Cloning identity is not the same as controlling delivery. A model may reproduce someone’s timbre beautifully while ignoring instructions such as “concerned”, “certain”, or “guarded.” Another may produce an obvious emotional change while damaging identity or naturalness.

We will document our experiments with positive, concerned, calm and irritated delivery, along with the controls that did not work. We will ask whether a style manipulation remains audible when the text is unchanged and whether it preserves the apparent speaker closely enough to support a matched comparison.

Most importantly, we will explain why a treatment that changes both emotion and audio quality does not isolate the causal effect of emotion.

Post 5: Do listeners change their minds?

The final step is a controlled listening experiment with multiple independent participants.

Different listeners will hear different deliveries of the same answer while we measure perceived valence, uncertainty, responsiveness, evasiveness, credibility and, in a carefully defined experimental setting, their impression of the answer’s economic implications.

We will discuss randomization, manipulation checks, sample size, listener and item effects, preregistration, and why my own expert annotations are useful for developing the experiment but insufficient for making claims about a population of listeners.

The result may be an effect, a null result, or evidence that our synthetic instrument is not yet clean enough. All three would be scientifically useful if the experiment is designed and reported honestly.

What would count as an answer?

At the end of the series, we would like to be able to answer three practical questions.

  1. What can a listener hear that a transcript reader cannot see?
  2. Can we measure that difference without confusing it with words, speaker or recording artifacts?
  3. If we change only the delivery, do people or machines change their minds?

The progression is deliberately slow:

raw call
  → aligned question and answer
  → verified same-speaker reference
  → clean voice clone
  → controlled delivery
  → randomized listener judgment

There is no point discussing causal effects if the answer is cut in half. There is no point comparing voice cloners if the reference contains two people. And there is no point claiming that vocal sentiment matters if the “concerned” clip also sounds more robotic than the “positive” one.

The next post will begin with the broadest question: if the transcript already contains the words, what exactly might remain in the audio, and how could we know whether it matters?