Tootfinder

I appreciate Hidde’s ability to take preposterous claims by a major accessibility tooling maker and the rampant thievery of IP by LLM makers, consider their dehumanizing output, and propose this gentle, whisper-soft rebuke:
https://hidde.blog/teaching-llms-or-companies/

Yes, let's teach LLMs accessibility, but also provide the companies using them with better strategies
Some thoughts on teaching LLMs accessibility.

'Dem Boden ganz nah' #FotoVorschlag 'close to the ground'
I think this could fit the topic. I took this photo when we had some vacations at the #balticSea . Quite a while ago - but all fond memories. 🙂

🇺🇦 #NowPlaying on KEXP's #Audioasis
Delvon Lamarr Organ Trio:
🎵 Careless Whisper
#DelvonLamarrOrganTrio
https://delvonlamarrorgantrio.bandcamp.com/track/careless-whisper
https://open.spotify.com/track/4xmkuzw1PuJ15xCC0YP6VY

HITSZ's End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track
Xuchen Wei, Yangxin Wu, Yaoyin Zhang, Henglyu Liu, Kehai Chen, Xuefeng Bai, Min Zhang
https://arxiv.org/abs/2507.19616

HITSZ's End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track
This paper presents HITSZ's submission for the IWSLT 2025 Indic track, focusing on speech-to-text translation (ST) for English-to-Indic and Indic-to-English language pairs. To enhance translation quality in this low-resource scenario, we propose an end-to-end system integrating the pre-trained Whisper automated speech recognition (ASR) model with Krutrim, an Indic-specialized large language model (LLM). Experimental results demonstrate that our end-to-end system achieved average BLEU scores of …

Speech Emotion Recognition via Entropy-Aware Score Selection
ChenYi Chua, JunKai Wong, Chengxin Chen, Xiaoxiao Miao
https://arxiv.org/abs/2508.20796 https://

Speech Emotion Recognition via Entropy-Aware Score Selection
In this paper, we propose a multimodal framework for speech emotion recognition that leverages entropy-aware score selection to combine speech and textual predictions. The proposed method integrates a primary pipeline that consists of an acoustic model based on wav2vec2.0 and a secondary pipeline that consists of a sentiment analysis model using RoBERTa-XLM, with transcriptions generated via Whisper-large-v3. We propose a late score fusion approach based on entropy and varentropy thresholds to …

Cooles Feature. Ich bin kein großer AI Fan gleichzeitig sehe ich bei Transkription tatsächlich Potenzial. Und nahtlos und offline SRT Dateien erstellen zu lassen ist super.
FFmpeg 8.0 integriert Whisper: Lokale Audio-Transkription ohne Cloud | heise online

FFmpeg 8.0 integriert Whisper: Lokale Audio-Transkription ohne Cloud
Die kommende Version 8.0 von FFmpeg erhält optional Whisper. OpenAIs KI-Transkription werden sich flexibel einsetzen lassen.

“I sat beside a boy who tried to smile at me. I couldn’t return a real smile. Tears welled in my eyes as I realized words could never reach the horrors his soul had witnessed. All I could do was place my hand gently on his shoulder and whisper, ‘You are not alone.’”
– @…

I was walking down a street that had once been full of life—shops open,...
Then I saw them. Children. Some missing an arm, some missing a leg, some missing both, yet their small bodies moved through the debris with a courage that seemed almost impossible. Their eyes—wide, searching—asked questions no child should ever have to ask: Why me? Why here? Will anyone see me? A boy sat on the curb, rolling a broken toy car over the cracks. His laugh was fragile, sharp, yet it was still laughter. Nearby, a girl balanced on a fallen beam, her small feet gripping...

Structured Sparsity and Weight-adaptive Pruning for Memory and Compute efficient Whisper models
Prasenjit K Mudi, Anshi Sachan, Dahlia Devapriya, Sheetal Kalyani
https://arxiv.org/abs/2510.12666

Structured Sparsity and Weight-adaptive Pruning for Memory and Compute efficient Whisper models
Whisper models have achieved remarkable progress in speech recognition; yet their large size remains a bottleneck for deployment on resource-constrained edge devices. This paper proposes a framework to design fine-tuned variants of Whisper which address the above problem. Structured sparsity is enforced via the Sparse Group LASSO penalty as a loss regularizer, to reduce the number of FLOating Point operations (FLOPs). Further, a weight statistics aware pruning algorithm is proposed. We also des…

Does Audio Matter for Modern Video-LLMs and Their Benchmarks?
Geewook Kim, Minjoon Seo
https://arxiv.org/abs/2509.17901 https://arxiv.org/pdf/2509.17901

Does Audio Matter for Modern Video-LLMs and Their Benchmarks?
Modern multimodal large language models often claim "video understanding," yet most evaluations use muted videos or simply discard audio. We ask a direct question: how much does audio actually matter for contemporary Video-LLMs and the benchmarks that certify them? We audit widely used suites and observe that many items are even solvable from a single frame, rendering audio largely redundant. Building on LLaVA-OneVision architecture, we attach a speech/audio encoder (e.g., Whisper) and analyze …

How the outrage over Jimmy Kimmel's remarks on Charlie Kirk ballooned, starting with one X user, who monitors late night shows for liberal bias, posting a clip (Stuart A. Thompson/New York Times)
https://www.nytimes.com/2025/09/19/technology/kimmel-carr-outr…

How Outrage at Kimmel Grew to a Shout From a Whisper
Right-wing users on social media on Tuesday were frustrated but not yet apoplectic about Jimmy Kimmel’s monologue. Things changed.

@… Jordi, I hear Macwhisper is now blazingly fast with Nvidia tech — did you ever finish the export that we can use for the podcasts with multiple speaker speakers plus chapter marker − .srt?
I’m always waiting to use Mac whisper, but I can’t because the output file doesn’t help me.
And I am one of thousands.

Zero-shot Context Biasing with Trie-based Decoding using Synthetic Multi-Pronunciation
Changsong Liu, Yizhou Peng, Eng Siong Chng
https://arxiv.org/abs/2508.17796 https://

Zero-shot Context Biasing with Trie-based Decoding using Synthetic Multi-Pronunciation
Contextual automatic speech recognition (ASR) systems allow for recognizing out-of-vocabulary (OOV) words, such as named entities or rare words. However, it remains challenging due to limited training data and ambiguous or inconsistent pronunciations. In this paper, we propose a synthesis-driven multi-pronunciation contextual biasing method that performs zero-shot contextual ASR on a pretrained Whisper model. Specifically, we leverage text-to-speech (TTS) systems to synthesize diverse speech sa…

BaldWhisper: Faster Whisper with Head Shearing and Layer Merging
Yaya Sy, Christophe Cerisara, Irina Illina
https://arxiv.org/abs/2510.08599 https://arxiv.…

BaldWhisper: Faster Whisper with Head Shearing and Layer Merging
Pruning large pre-trained transformers for low-resource languages is challenging, as it often requires massive retraining data to recover performance. For instance, Distill-Whisper prunes Whisper by 40% and retrains on 21,000 hours of speech, far beyond what is available for most languages. Can Whisper be made lighter and faster for edge devices in data-scarce settings? Focusing on Bambara with only 32h of speech-to-text data, we propose a new pruning recipe. Instead of vocabulary pruning, whic…

The three words you can whisper into the ear of any support AI chatbot to get their attention
credit card chargeback

VOX-KRIKRI: Unifying Speech and Language through Continuous Fusion
Dimitrios Damianos, Leon Voukoutis, Georgios Paraskevopoulos, Vassilis Katsouros
https://arxiv.org/abs/2509.15667

VOX-KRIKRI: Unifying Speech and Language through Continuous Fusion
We present a multimodal fusion framework that bridges pre-trained decoder-based large language models (LLM) and acoustic encoder-decoder architectures such as Whisper, with the aim of building speech-enabled LLMs. Instead of directly using audio embeddings, we explore an intermediate audio-conditioned text space as a more effective mechanism for alignment. Our method operates fully in continuous text representation spaces, fusing Whisper's hidden decoder states with those of an LLM through cros…

🇺🇦 #NowPlaying on #BBC6Music's #RadcliffeAndMaconie
Cecile Campbell:
🎵 Whisper to Me
#CecileCampbell
https://basswiserecords.bandcamp.com/track/whisper-to-me
https://open.spotify.com/track/5xgPgAWIZHRdXJsSlJdgJp

Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition
Yujian Ma, Jinqiu Sang, Ruizhe Li
https://arxiv.org/abs/2509.08454 https:/…

Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition
Large pre-trained speech models such as Whisper offer strong generalization but pose significant challenges for resource-efficient adaptation. Low-Rank Adaptation (LoRA) has become a popular parameter-efficient fine-tuning method, yet its underlying mechanisms in speech tasks remain poorly understood. In this work, we conduct the first systematic mechanistic interpretability study of LoRA within the Whisper encoder for speech emotion recognition (SER). Using a suite of analytical tools, includi…

Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMs
Om Tailor
https://arxiv.org/abs/2510.04303 https://arxiv.org/pdf/2510.04303

Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMs
Multi-agent deployments of large language models (LLMs) are increasingly embedded in market, allocation, and governance workflows, yet covert coordination among agents can silently erode trust and social welfare. Existing audits are dominated by heuristics that lack theoretical guarantees, struggle to transfer across tasks, and seldom ship with the infrastructure needed for independent replication. We introduce \emph{Audit the Whisper}, a conference-grade research artifact that spans theory, be…

📹 Creates SRT subtitle files for videos and supports real-time live broadcast transcription
🔄 Seamless workflow integration
🔄 allowing automated processing and data transfer to other applications
🔗 Source URL
https://www.heise.de/en/news…

Why must we whisper so much? A quite sad season opener feels bad!
#CriticalRole

Programmieren per Spracheingabe LLM, ein riesen Spaß. Gerade hat whisper "grobes Scheiß-Skript" verstanden und es ist inhaltlich absolut korrekt :D Hab "shell script" gesagt, aber ersteres gedacht...

Variational Low-Rank Adaptation for Personalized Impaired Speech Recognition
Niclas Pokel, Pehu\'en Moure, Roman Boehringer, Shih-Chii Liu, Yingqiang Gao
https://arxiv.org/abs/2509.20397

Variational Low-Rank Adaptation for Personalized Impaired Speech Recognition
Speech impairments resulting from congenital disorders, such as cerebral palsy, down syndrome, or apert syndrome, as well as acquired brain injuries due to stroke, traumatic accidents, or tumors, present major challenges to automatic speech recognition (ASR) systems. Despite recent advancements, state-of-the-art ASR models like Whisper still struggle with non-normative speech due to limited training data availability and high acoustic variability. Moreover, collecting and annotating non-normati…

It's basically impossible to imagine a time where Careless Whisper was a song you'd never heard before. But here it is. #TOTP

#Python Friday #292: Extract Text From Audio Files With Whisper - #ai #nlp

#292: Extract Text From Audio Files With Whisper - Python Friday
None

Assessing the Feasibility of Lightweight Whisper Models for Low-Resource Urdu Transcription
Abdul Rehman Antall, Naveed Akhtar
https://arxiv.org/abs/2508.09865 https://

Assessing the Feasibility of Lightweight Whisper Models for Low-Resource Urdu Transcription
This study evaluates the feasibility of lightweight Whisper models (Tiny, Base, Small) for Urdu speech recognition in low-resource settings. Despite Urdu being the 10th most spoken language globally with over 230 million speakers, its representation in automatic speech recognition (ASR) systems remains limited due to dialectal diversity, code-switching, and sparse training data. We benchmark these models on a curated Urdu dataset using word error rate (WER), without fine-tuning. Results show Wh…

Leveraging Whisper Embeddings for Audio-based Lyrics Matching
Eleonora Mancini, Joan Serr\`a, Paolo Torroni, Yuki Mitsufuji
https://arxiv.org/abs/2510.08176 https://

Leveraging Whisper Embeddings for Audio-based Lyrics Matching
Audio-based lyrics matching can be an appealing alternative to other content-based retrieval approaches, but existing methods often suffer from limited reproducibility and inconsistent baselines. In this work, we introduce WEALY, a fully reproducible pipeline that leverages Whisper decoder embeddings for lyrics matching tasks. WEALY establishes robust and transparent baselines, while also exploring multimodal extensions that integrate textual and acoustic features. Through extensive experiments…

AISHELL6-whisper: A Chinese Mandarin Audio-visual Whisper Speech Dataset with Speech Recognition Baselines
Cancan Li, Fei Su, Juan Liu, Hui Bu, Yulong Wan, Hongbin Suo, Ming Li
https://arxiv.org/abs/2509.23833

AISHELL6-whisper: A Chinese Mandarin Audio-visual Whisper Speech Dataset with Speech Recognition Baselines
Whisper speech recognition is crucial not only for ensuring privacy in sensitive communications but also for providing a critical communication bridge for patients under vocal restraint and enabling discrete interaction in noise-sensitive environments. The development of Chinese mandarin audio-visual whisper speech recognition is hindered by the lack of large-scale datasets. We present AISHELL6-Whisper, a large-scale open-source audio-visual whisper speech dataset, featuring 30 hours each of wh…

SloPalSpeech: A 2,8000-Hour Slovak Speech Corpus from Parliamentary Data
Erik Bo\v{z}\'ik, Marek \v{S}uppa
https://arxiv.org/abs/2509.19270 https://arx…

Idiosyncratic Versus Normative Modeling of Atypical Speech Recognition: Dysarthric Case Studies
Vishnu Raja, Adithya V Ganesan, Anand Syamkumar, Ritwik Banerjee, H Andrew Schwartz
https://arxiv.org/abs/2509.16718

Idiosyncratic Versus Normative Modeling of Atypical Speech Recognition: Dysarthric Case Studies
State-of-the-art automatic speech recognition (ASR) models like Whisper, perform poorly on atypical speech, such as that produced by individuals with dysarthria. Past works for atypical speech have mostly investigated fully personalized (or idiosyncratic) models, but modeling strategies that can both generalize and handle idiosyncracy could be more effective for capturing atypical speech. To investigate this, we compare four strategies: (a) $\textit{normative}$ models trained on typical speech …

Crosslisted article(s) found for cs.LG. https://arxiv.org/list/cs.LG/new
[4/5]:
- Leveraging Whisper Embeddings for Audio-based Lyrics Matching
Eleonora Mancini, Joan Serr\`a, Paolo Torroni, Yuki Mitsufuji

Isoperimetric-type inequalities for Mather's $\beta$-function of convex billiards
Stefano Baranzini, Misha Bialy, Alfonso Sorrentino
https://arxiv.org/abs/2509.06915 https:/…

Isoperimetric-type inequalities for Mather's $β$-function of convex billiards
In this article we discuss pointwise spectral rigidity results for several billiard systems (e.g., Birkhoff billiards, symplectic billiards and $4$-th billiards), showing that a single value of Mather's $β$-function can determine whether a strongly convex smooth planar domain is a disk (or an ellipse, in the affine-invariant case of symplectic billiards). Evoking the famous question "Can you hear the shape of a billiard?", one could say that circular billiards can be heard by a single whisper!…

Which one Performs Better? Wav2Vec or Whisper? Applying both in Badini Kurdish Speech to Text (BKSTT)
Renas Adnan, Hossein Hassani
https://arxiv.org/abs/2508.09957 https://

Which one Performs Better? Wav2Vec or Whisper? Applying both in Badini Kurdish Speech to Text (BKSTT)
Speech-to-text (STT) systems have a wide range of applications. They are available in many languages, albeit at different quality levels. Although Kurdish is considered a less-resourced language from a processing perspective, SST is available for some of the Kurdish dialects, for instance, Sorani (Central Kurdish). However, that is not applied to other Kurdish dialects, Badini and Hawrami, for example. This research is an attempt to address this gap. Bandin, approximately, has two million speak…

Whisper Smarter, not Harder: Adversarial Attack on Partial Suppression
Zheng Jie Wong, Bingquan Shen
https://arxiv.org/abs/2508.09994 https://arxiv.org/pdf…

Whisper Smarter, not Harder: Adversarial Attack on Partial Suppression
Currently, Automatic Speech Recognition (ASR) models are deployed in an extensive range of applications. However, recent studies have demonstrated the possibility of adversarial attack on these models which could potentially suppress or disrupt model output. We investigate and verify the robustness of these attacks and explore if it is possible to increase their imperceptibility. We additionally find that by relaxing the optimisation objective from complete suppression to partial suppression, w…

Probing Whisper for Dysarthric Speech in Detection and Assessment
Zhengjun Yue, Devendra Kayande, Zoran Cvetkovic, Erfan Loweimi
https://arxiv.org/abs/2510.04219 https://…

Probing Whisper for Dysarthric Speech in Detection and Assessment
Large-scale end-to-end models such as Whisper have shown strong performance on diverse speech tasks, but their internal behavior on pathological speech remains poorly understood. Understanding how dysarthric speech is represented across layers is critical for building reliable and explainable clinical assessment tools. This study probes the Whisper-Medium model encoder for dysarthric speech for detection and assessment (i.e., severity classification). We evaluate layer-wise embeddings with a li…

Evaluating the Effectiveness of Transformer Layers in Wav2Vec 2.0, XLS-R, and Whisper for Speaker Identification Tasks
Linus Stuhlmann, Michael Alexander Saxer
https://arxiv.org/abs/2509.00230

Evaluating the Effectiveness of Transformer Layers in Wav2Vec 2.0, XLS-R, and Whisper for Speaker Identification Tasks
This study evaluates the performance of three advanced speech encoder models, Wav2Vec 2.0, XLS-R, and Whisper, in speaker identification tasks. By fine-tuning these models and analyzing their layer-wise representations using SVCCA, k-means clustering, and t-SNE visualizations, we found that Wav2Vec 2.0 and XLS-R capture speaker-specific features effectively in their early layers, with fine-tuning improving stability and performance. Whisper showed better performance in deeper layers. Additional…

Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition
Martin Kocour, Martin Karafiat, Alexander Polok, Dominik Klement, Luk\'a\v{s} Burget, Jan \v{C}ernock\'y
https://arxiv.org/abs/2510.03723

Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition
We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to…

Whispering Context: Distilling Syntax and Semantics for Long Speech Transcripts
Duygu Altinok
https://arxiv.org/abs/2508.13376 https://arxiv.org/pdf/2508.1…

Whispering Context: Distilling Syntax and Semantics for Long Speech Transcripts
ASR systems often struggle with maintaining syntactic and semantic accuracy in long audio transcripts, impacting tasks like Named Entity Recognition (NER), capitalization, and punctuation. We propose a novel approach that enhances ASR by distilling contextual knowledge from LLaMA models into Whisper. Our method uses two strategies: (1) token level distillation with optimal transport to align dimensions and sequence lengths, and (2) representation loss minimization between sentence embeddings of…

🎵 #FFmpeg 8.0 integrates #Whisper for local audio transcription 🎵 #ai
🔧 Direct integration of #OpenAI

🇺🇦 #NowPlaying on #BBC6Music's #DreamTime
Wim Mertens:
🎵 Whisper Me
#WimMertens
https://open.spotify.com/track/0gBLAW3CBwGMqBWoL99gk0

Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification
William Ravenscroft, George Close, Kit Bower-Morris, Jamie Stacey, Dmitry Sityaev, Kris Y. Hong
https://arxiv.org/abs/2507.21642

Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification
Large-scale in-the-wild speech datasets have become more prevalent in recent years due to increased interest in models that can learn useful features from unlabelled data for tasks such as speech recognition or synthesis. These datasets often contain undesirable features, such as multiple speakers, non-target languages, and music, which may impact model learning. The Whilter model is proposed as a multitask solution to identify these undesirable samples. Whilter uses a Whisper encoder with an a…

Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
Mengqi Wang, Zhan Liu, Zengrui Jin, Guangzhi Sun, Chao Zhang, Philip C. Woodland
https://arxiv.org/abs/2509.16622

Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
Diffusion-based large language models (DLLMs) have recently attracted growing interest as an alternative to autoregressive decoders. In this work, we present an empirical study on using the diffusion-based large language model LLaDA for automatic speech recognition (ASR). We first investigate its use as an external deliberation-based processing module for Whisper-LLaMA transcripts. By leveraging the bidirectional attention and denoising capabilities of LLaDA, we explore random masking, low-conf…

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers
Akshat Pandey, Karun Kumar, Raphael Tang
https://arxiv.org/abs/2509.10452 …

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers
Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen vocabulary and parlance. In many real-world settings, collecting speech data is impractical, necessitating text-only adaptation. We propose WhisTLE, a deeply supervised, text-only adaptation method for pretrained encoder-decoder ASR models. WhisTLE trains a variational autoencoder (VAE) to model encoder outputs from text and fine-tunes the decoder using the learned…

Towards Building Speech Large Language Models for Multitask Understanding in Low-Resource Languages
Mingchen Shao, Bingshen Mu, Chengyou Wang, Hai Li, Ying Yan, Zhonghua Fu, Lei Xie
https://arxiv.org/abs/2509.14804

Towards Building Speech Large Language Models for Multitask Understanding in Low-Resource Languages
Speech large language models (SLLMs) built on speech encoders, adapters, and LLMs demonstrate remarkable multitask understanding performance in high-resource languages such as English and Chinese. However, their effectiveness substantially degrades in low-resource languages such as Thai. This limitation arises from three factors: (1) existing commonly used speech encoders, like the Whisper family, underperform in low-resource languages and lack support for broader spoken language understanding …

Transsion Multilingual Speech Recognition System for MLC-SLM 2025 Challenge
Xiaoxiao Li, An Zhu, Youhai Jiang, Fengjie Zhu
https://arxiv.org/abs/2508.14916 https://

Transsion Multilingual Speech Recognition System for MLC-SLM 2025 Challenge
This paper presents the architecture and performance of a novel Multilingual Automatic Speech Recognition (ASR) system developed by the Transsion Speech Team for Track 1 of the MLC-SLM 2025 Challenge. The proposed system comprises three key components: 1) a frozen Whisper-large-v3 based speech encoder, leveraging large-scale pretraining to ensure robust acoustic feature extraction; 2) a trainable adaptor module using Linear-ReLU-Linear transformation mechanisms to effectively align speech and t…

WhiSQA: Non-Intrusive Speech Quality Prediction Using Whisper Encoder Features
George Close, Kris Hong, Thomas Hain, Stefan Goetze
https://arxiv.org/abs/2508.02210 https://

WhiSQA: Non-Intrusive Speech Quality Prediction Using Whisper Encoder Features
There has been significant research effort developing neural-network-based predictors of SQ in recent years. While a primary objective has been to develop non-intrusive, i.e.~reference-free, metrics to assess the performance of SE systems, recent work has also investigated the direct inference of neural SQ predictors within the loss function of downstream speech tasks. To aid in the training of SQ predictors, several large datasets of audio with corresponding human labels of quality have been c…

Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM
Ryandhimas E. Zezario, Dyah A. M. G. Wisnu, Hsin-Min Wang, Yu Tsao
https://arxiv.org/abs/2509.03013

Speech Intelligibility Assessment with Uncertainty-Aware Whisper Embeddings and sLSTM
Non-intrusive speech intelligibility prediction remains challenging due to variability in speakers, noise conditions, and subjective perception. We propose an uncertainty-aware approach that leverages Whisper embeddings in combination with statistical features, specifically the mean, standard deviation, and entropy computed across the embedding dimensions. The entropy, computed via a softmax over the feature dimension, serves as a proxy for uncertainty, complementing global information captured…

Large Language Model Data Generation for Enhanced Intent Recognition in German Speech
Theresa Pekarek Rosin, Burak Can Kaplan, Stefan Wermter
https://arxiv.org/abs/2508.06277 ht…

Large Language Model Data Generation for Enhanced Intent Recognition in German Speech
Intent recognition (IR) for speech commands is essential for artificial intelligence (AI) assistant systems; however, most existing approaches are limited to short commands and are predominantly developed for English. This paper addresses these limitations by focusing on IR from speech by elderly German speakers. We propose a novel approach that combines an adapted Whisper ASR model, fine-tuned on elderly German speech (SVC-de), with Transformer-based language models trained on synthetic text d…

A Study on Zero-Shot Non-Intrusive Speech Intelligibility for Hearing Aids Using Large Language Models
Ryandhimas E. Zezario, Dyah A. M. G. Wisnu, Hsin-Min Wang, Yu Tsao
https://arxiv.org/abs/2509.03021

A Study on Zero-Shot Non-Intrusive Speech Intelligibility for Hearing Aids Using Large Language Models
This work focuses on zero-shot non-intrusive speech assessment for hearing aids (HA) using large language models (LLMs). Specifically, we introduce GPT-Whisper-HA, an extension of GPT-Whisper, a zero-shot non-intrusive speech assessment model based on LLMs. GPT-Whisper-HA is designed for speech assessment for HA, incorporating MSBG hearing loss and NAL-R simulations to process audio input based on each individual's audiogram, two automatic speech recognition (ASR) modules for audio-to-text repr…

🇺🇦 #NowPlaying on #KEXP's #MiddayShow
Kae Tempest:
🎵 Prayers to Whisper
#KaeTempest
https://open.spotify.com/track/3ZcmA9UPT7SKRCSwaI9XGk

Proficiency-Aware Adaptation and Data Augmentation for Robust L2 ASR
Ling Sun, Charlotte Zhu, Shuju Shi
https://arxiv.org/abs/2510.10738 https://arxiv.org/…

Proficiency-Aware Adaptation and Data Augmentation for Robust L2 ASR
General-purpose ASR underperforms for atypical speakers, such as L2 learners, reinforcing bias and limiting use in education and accessibility. Using the CEFR-graded Speak and Improve corpus, we show that naive fine-tuning of Whisper reduces average WER but simultaneously widens disparities and disproportionately harms lower-level learners. To address this, we propose two strategies: (i) proficiency-aware multitask learning, jointly optimizing ASR with proficiency classification, and (ii) targe…

A Low-Resource Speech-Driven NLP Pipeline for Sinhala Dyslexia Assistance
Peshala Perera, Deshan Sumanathilaka
https://arxiv.org/abs/2510.04750 https://arx…

A Low-Resource Speech-Driven NLP Pipeline for Sinhala Dyslexia Assistance
Dyslexia in adults remains an under-researched and under-served area, particularly in non-English-speaking contexts, despite its significant impact on personal and professional lives. This work addresses that gap by focusing on Sinhala, a low-resource language with limited tools for linguistic accessibility. We present an assistive system explicitly designed for Sinhala-speaking adults with dyslexia. The system integrates Whisper for speech-to-text conversion, SinBERT, an open-sourced fine-tune…

Identifying and Calibrating Overconfidence in Noisy Speech Recognition
Mingyue Huo, Yuheng Zhang, Yan Tang
https://arxiv.org/abs/2509.07195 https://arxiv.o…

Identifying and Calibrating Overconfidence in Noisy Speech Recognition
Modern end-to-end automatic speech recognition (ASR) models like Whisper not only suffer from reduced recognition accuracy in noise, but also exhibit overconfidence - assigning high confidence to wrong predictions. We conduct a systematic analysis of Whisper's behavior in additive noise conditions and find that overconfident errors increase dramatically at low signal-to-noise ratios, with 10-20% of tokens incorrectly predicted with confidence above 0.7. To mitigate this, we propose a lightweigh…

Bridging ASR and LLMs for Dysarthric Speech Recognition: Benchmarking Self-Supervised and Generative Approaches
Ahmed Aboeitta, Ahmed Sharshar, Youssef Nafea, Shady Shehata
https://arxiv.org/abs/2508.08027

Bridging ASR and LLMs for Dysarthric Speech Recognition: Benchmarking Self-Supervised and Generative Approaches
Speech Recognition (ASR) due to phoneme distortions and high variability. While self-supervised ASR models like Wav2Vec, HuBERT, and Whisper have shown promise, their effectiveness in dysarthric speech remains unclear. This study systematically benchmarks these models with different decoding strategies, including CTC, seq2seq, and LLM-enhanced decoding (BART,GPT-2, Vicuna). Our contributions include (1) benchmarking ASR architectures for dysarthric speech, (2) introducing LLM-based decoding to …

Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data
Gokul Karthik Kumar, Rishabh Saraf, Ludovick Lepauloux, Abdul Muneer, Billel Mokeddem, Hakim Hacid
https://arxiv.org/abs/2509.07526

Competitive Audio-Language Models with Data-Efficient Single-Stage Training on Public Data
Large language models (LLMs) have transformed NLP, yet their integration with audio remains underexplored -- despite audio's centrality to human communication. We introduce Falcon3-Audio, a family of Audio-Language Models (ALMs) built on instruction-tuned LLMs and Whisper encoders. Using a remarkably small amount of public audio data -- less than 30K hours (5K unique) -- Falcon3-Audio-7B matches the best reported performance among open-weight models on the MMAU benchmark, with a score of 64.14,…

Crosslisted article(s) found for eess.AS. https://arxiv.org/list/eess.AS/new
[1/1]:
- Whisper Smarter, not Harder: Adversarial Attack on Partial Suppression
Zheng Jie Wong, Bingquan Shen

Keyword Spotting with Hyper-Matched Filters for Small Footprint Devices
Yael Segal-Feldman, Ann R. Bradlow, Matthew Goldrick, Joseph Keshet
https://arxiv.org/abs/2508.04857 http…

Keyword Spotting with Hyper-Matched Filters for Small Footprint Devices
Open-vocabulary keyword spotting (KWS) refers to the task of detecting words or terms within speech recordings, regardless of whether they were included in the training data. This paper introduces an open-vocabulary keyword spotting model with state-of-the-art detection accuracy for small-footprint devices. The model is composed of a speech encoder, a target keyword encoder, and a detection network. The speech encoder is either a tiny Whisper or a tiny Conformer. The target keyword encoder is i…

Feature Importance across Domains for Improving Non-Intrusive Speech Intelligibility Prediction in Hearing Aids
Ryandhimas E. Zezario, Sabato M. Siniscalchi, Fei Chen, Hsin-Min Wang, Yu Tsao
https://arxiv.org/abs/2507.23223

Feature Importance across Domains for Improving Non-Intrusive Speech Intelligibility Prediction in Hearing Aids
Given the critical role of non-intrusive speech intelligibility assessment in hearing aids (HA), this paper enhances its performance by introducing Feature Importance across Domains (FiDo). We estimate feature importance on spectral and time-domain acoustic features as well as latent representations of Whisper. Importance weights are calculated per frame, and based on these weights, features are projected into new spaces, allowing the model to focus on important areas early. Next, feature conca…