cognitivecomputations/Dolphin3.0-R1-Mistral-24B
Text Generation
•
Updated
•
2.88k
•
134
peterciank
peterciank
AI & ML interests
GenAI, Prompt Engineering, Machine Learning Models, Models Orchestration, Skills Orchestration
Recent Activity
liked
a model
10 days ago
cognitivecomputations/Dolphin3.0-R1-Mistral-24B
upvoted
an
article
15 days ago
o3-mini & Deepseek-R1
liked
a Space
21 days ago
Qwen/Qwen2.5-Max-Demo
Organizations
None yet
peterciank's activity
liked
a
model
10 days ago
upvoted
an
article
15 days ago
Article
o3-mini & Deepseek-R1
By
•
•
30
liked
a
Space
21 days ago
liked
a
model
22 days ago
liked
a
model
about 2 months ago
upvoted
a
collection
4 months ago
reacted to
sanchit-gandhi's
post with ❤️
4 months ago
Post
Why does returning timestamps help Whisper reduce hallucinations? 🧐
Empirically, most practitioners have found that setting
But why does this work?..
My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:
Where we have a repeated hallucination for “on the”. If we ask the model to predict timestamps, then the “on the” has to contribute to the overall segment-level timing, e.g.:
However, it’s impossible to fit 3 copies of “on the” within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:
In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper Robust Speech Recognition via Large-Scale Weak Supervision (2212.04356) → it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).
Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?
Empirically, most practitioners have found that setting
return_timestamps=True helps reduce hallucinations, particularly when doing long-form evaluation with Transformers’ “chunked” algorithm. But why does this work?..
My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:
The cat sat on the on the on the mat.Where we have a repeated hallucination for “on the”. If we ask the model to predict timestamps, then the “on the” has to contribute to the overall segment-level timing, e.g.:
<|0.00|> The cat sat on the on the on the mat.<|5.02|>However, it’s impossible to fit 3 copies of “on the” within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:
<|0.00|> The cat sat on the mat.<|5.02|>In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper Robust Speech Recognition via Large-Scale Weak Supervision (2212.04356) → it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).
Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?
- 11 replies
liked
2
Spaces
4 months ago
upvoted
an
article
4 months ago
Article
Inference for PROs
•
53
updated
2
Spaces
5 months ago
liked
a
Space
5 months ago
reacted to
KingNish's
post with 👍
5 months ago
Post
5880
Introducing OpenCHAT mini: a lightweight, fast, and unlimited version of OpenGPT 4o.
KingNish/OpenCHAT-mini2
It has unlimited web search, vision and image generation.
Please take a look and share your review. Thank you! 🤗
KingNish/OpenCHAT-mini2
It has unlimited web search, vision and image generation.
Please take a look and share your review. Thank you! 🤗
- 7 replies
liked
a
Space
6 months ago
updated
a
Space
6 months ago
liked
a
model
6 months ago
liked
a
Space
6 months ago