Skip to main content

convertToCaptions()v4.0.131

danger

This API assumes a newer version of Whisper.cpp than the stable release to support tokenLevelTimestamps. As a downside, this version may crash unexpectedly. Use an older version of Whisper.cpp (1.0.54 or earlier) if you prefer to use a stable version of Whisper.cpp and forgo tokenLevelTimeStamps support.

Opinionated function that converts the output from transcribe() into easily digestable captions.
Can also combine words with close timestamps.
Useful for TikTok/Reel-type of videos that animate captions word-by-word.

transcribe.mjs
tsx
import path from "path";
import { transcribe, convertToCaptions } from "@remotion/install-whisper-cpp";
 
const { transcription } = await transcribe({
inputPath: "/path/to/audio.wav",
whisperPath: path.join(process.cwd(), "whisper.cpp"),
model: "medium.en",
tokenLevelTimestamps: true,
});
 
const { captions } = convertToCaptions({
transcription,
combineTokensWithinMilliseconds: 200,
});
 
for (const line of captions) {
console.log(line.text, line.startInSeconds);
}
transcribe.mjs
tsx
import path from "path";
import { transcribe, convertToCaptions } from "@remotion/install-whisper-cpp";
 
const { transcription } = await transcribe({
inputPath: "/path/to/audio.wav",
whisperPath: path.join(process.cwd(), "whisper.cpp"),
model: "medium.en",
tokenLevelTimestamps: true,
});
 
const { captions } = convertToCaptions({
transcription,
combineTokensWithinMilliseconds: 200,
});
 
for (const line of captions) {
console.log(line.text, line.startInSeconds);
}

Options

transcription

The transcription object that you retrieved from transcribe().
The tokenLevelTimestamps option must have been set to true.

combineTokensWithinMilliseconds

Combine words that are close to each other.
If words are not combined, they might display for a very short time if word-by-word captions are being used.
Disable combination by setting 0.
Recommendation: 200.

Return value

An array with objects of the following shape:

ts
type Caption = {
text: string;
startInSeconds: number;
};
ts
type Caption = {
text: string;
startInSeconds: number;
};

Suggested usage

This shows how, given a data structure produced by convertToCaptions(), word-by-word captions can be rendered in a Remotion project.
See our TikTok template for a full reference implementation.

note

@remotion/install-whisper-cpp cannot be imported on the frontend, it is a Node.js API.
Only the TypeScript type is imported in this example

tsx
import type { Caption } from "@remotion/install-whisper-cpp";
import { Sequence, useVideoConfig } from "remotion";
 
const Captions: React.FC<{
subtitles: Caption[];
}> = ({ subtitles }) => {
const { fps } = useVideoConfig();
 
return (
<>
{subtitles.map((subtitle, index) => {
const nextSubtitle = subtitles[index + 1] ?? null;
const subtitleStartFrame = subtitle.startInSeconds * fps;
const subtitleEndFrame = Math.min(
nextSubtitle ? nextSubtitle.startInSeconds * fps : Infinity,
subtitleStartFrame + fps,
);
 
return (
<Sequence
from={subtitleStartFrame}
durationInFrames={subtitleEndFrame - subtitleStartFrame}
>
<Subtitle key={index} text={subtitle.text} />;
</Sequence>
);
})}
</>
);
};
tsx
import type { Caption } from "@remotion/install-whisper-cpp";
import { Sequence, useVideoConfig } from "remotion";
 
const Captions: React.FC<{
subtitles: Caption[];
}> = ({ subtitles }) => {
const { fps } = useVideoConfig();
 
return (
<>
{subtitles.map((subtitle, index) => {
const nextSubtitle = subtitles[index + 1] ?? null;
const subtitleStartFrame = subtitle.startInSeconds * fps;
const subtitleEndFrame = Math.min(
nextSubtitle ? nextSubtitle.startInSeconds * fps : Infinity,
subtitleStartFrame + fps,
);
 
return (
<Sequence
from={subtitleStartFrame}
durationInFrames={subtitleEndFrame - subtitleStartFrame}
>
<Subtitle key={index} text={subtitle.text} />;
</Sequence>
);
})}
</>
);
};

See also