Skip to content

feat!: major performance & accuracy improvements in speech-to-text module #1132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
IgorSwat wants to merge 19 commits into
base: main
Choose a base branch
from
Open

feat!: major performance & accuracy improvements in speech-to-text module #1132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
bc31cd5
Optimal streaming algorithm
IgorSwat May 7, 2026
92b3f29
Revert back to 100ms refresh rate
IgorSwat May 7, 2026
35290db
Add CoreML whisper models
IgorSwat May 7, 2026
7473f01
Update model urls
IgorSwat May 8, 2026
9b90ea3
Change default model for iOS devices
IgorSwat May 8, 2026
9af8124
Add explicit timeout parameter
IgorSwat May 10, 2026
f7849fc
Concurrency fixes & automatic cleaunp
IgorSwat May 10, 2026
3bf68bf
Update urls & audio-api
IgorSwat May 14, 2026
27769d4
Apply review suggestions
IgorSwat May 19, 2026
c5b142d
Rebase with main
IgorSwat May 19, 2026
6bba141
Minor fixes
IgorSwat May 20, 2026
1aebae6
Fix broken test build
IgorSwat May 20, 2026
88185d5
chore(stt): drop unused transcribeStringOnly declaration
May 20, 2026
44f6931
fix(stt): preserve pending committed words in OnlineASR
May 20, 2026
aaeac64
fix(stt): use scores.size() as avgLogProb denominator
May 20, 2026
7f46540
perf(stt): hoist mt19937 out of the sampling loop
May 20, 2026
72912ad
fix(stt)!: drop quantized variants from SpeechToTextModelName
May 20, 2026
0605950
chore(stt): align stream() declaration with definition
May 20, 2026
a5c3662
fix(stt): break streamStop() out of the timeout pause immediately
May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions apps/llm/app/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,6 @@ export default function Home() {
>
<Text style={styles.buttonText}>LLM Structured Output</Text>
</TouchableOpacity>
<TouchableOpacity
style={styles.button}
onPress={() => router.navigate('voice_chat/')}
>
<Text style={styles.buttonText}>Voice Chat</Text>
</TouchableOpacity>
<TouchableOpacity
style={styles.button}
onPress={() => router.navigate('multimodal_llm/')}
Expand Down
311 changes: 0 additions & 311 deletions apps/llm/app/voice_chat/index.tsx
View file
Open in desktop

This file was deleted.

22 changes: 15 additions & 7 deletions apps/speech/screens/SpeechToTextScreen.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@ import { SafeAreaProvider, SafeAreaView } from 'react-native-safe-area-context';
import {
useSpeechToText,
WHISPER_TINY_EN,
WHISPER_TINY_EN_QUANTIZED,
WHISPER_TINY_EN_COREML,
WHISPER_BASE_EN,
WHISPER_BASE_EN_COREML,
WHISPER_SMALL_EN,
WHISPER_SMALL_EN_COREML,
TranscriptionResult,
SpeechToTextProps,
} from 'react-native-executorch';
Expand All @@ -25,10 +27,12 @@ import { ModelPicker, ModelOption } from '../components/ModelPicker';
type STTModelSources = SpeechToTextProps['model'];

const MODELS: ModelOption<STTModelSources>[] = [
{ label: 'Whisper Tiny', value: WHISPER_TINY_EN },
{ label: 'Whisper Tiny Q', value: WHISPER_TINY_EN_QUANTIZED },
{ label: 'Whisper Base', value: WHISPER_BASE_EN },
{ label: 'Whisper Small', value: WHISPER_SMALL_EN },
{ label: 'Whisper Tiny EN (XNNPACK)', value: WHISPER_TINY_EN },
{ label: 'Whisper Tiny EN (CoreML)', value: WHISPER_TINY_EN_COREML },
{ label: 'Whisper Base EN (XNNPACK)', value: WHISPER_BASE_EN },
{ label: 'Whisper Base EN (CoreML)', value: WHISPER_BASE_EN_COREML },
{ label: 'Whisper Small EN (XNNPACK)', value: WHISPER_SMALL_EN },
{ label: 'Whisper Small EN (CoreML)', value: WHISPER_SMALL_EN_COREML },
];
import FontAwesome from '@expo/vector-icons/FontAwesome';
import {
Expand All @@ -45,9 +49,12 @@ import ErrorBanner from '../components/ErrorBanner';

const isSimulator = DeviceInfo.isEmulatorSync();

const DEFAULT_MODEL =
Platform.OS === 'ios' ? WHISPER_BASE_EN_COREML : WHISPER_TINY_EN;

export const SpeechToTextScreen = ({ onBack }: { onBack: () => void }) => {
const [selectedModel, setSelectedModel] =
useState<STTModelSources>(WHISPER_TINY_EN);
useState<STTModelSources>(DEFAULT_MODEL);

const model = useSpeechToText({
model: selectedModel,
Expand Down Expand Up @@ -148,7 +155,7 @@ export const SpeechToTextScreen = ({ onBack }: { onBack: () => void }) => {
recorder.current.onAudioReady(
{
sampleRate,
bufferLength: 0.1 * sampleRate,
bufferLength: 0.1 * sampleRate, // 100 ms
channelCount: 1,
},
({ buffer }) => {
Expand Down Expand Up @@ -178,6 +185,7 @@ export const SpeechToTextScreen = ({ onBack }: { onBack: () => void }) => {
try {
const streamIter = model.stream({
verbose: enableTimestamps,
timeout: 100,
});

for await (const { committed, nonCommitted } of streamIter) {
Expand Down
3 changes: 1 addition & 2 deletions packages/react-native-executorch/common/rnexecutorch/host_objects/JsiConversions.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -591,8 +591,7 @@ inline jsi::Value getJsiValue(const Segment &seg, jsi::Runtime &runtime) {
jsi::Object wordObj(runtime);
wordObj.setProperty(
runtime, "word",
jsi::String::createFromUtf8(runtime, seg.words[i].content +
seg.words[i].punctations));
jsi::String::createFromUtf8(runtime, seg.words[i].content));
wordObj.setProperty(runtime, "start",
static_cast<double>(seg.words[i].start));
wordObj.setProperty(runtime, "end", static_cast<double>(seg.words[i].end));
Expand Down
Loading
Loading