Gemini YouTube Dubber
GitHub ↗
v0.5.10 architecture · validated cloud run

AI dubbing that treats timing as a first-class engineering problem.

Gemini YouTube Dubber turns public YouTube content or local video into translated speech, subtitles and a synchronized dubbing package. Its core idea is simple: preserve meaning, preserve the source timeline, and never “solve” timing by making speech unnaturally fast or slow.

Public YouTube URL / local video
        ↓
Gemini video understanding + translation
        ↓
Semantic source units + speaker/timestamp locks
        ↓
AI Timing Director (pre-TTS compression only when needed)
        ↓
Edge / Gemini speech synthesis
        ↓
Measured Timing Feedback (hard natural-rate limit: 1.10×)
        ↓
FFmpeg timeline placement + silence + soundtrack mix
        ↓
WAV + SRT + transcript + manifest → final MP4
37/37speech units completed in Cloud Dub #41
1.10×hard maximum natural-rate correction
3measured AI timing-feedback passes maximum
1.50 smaximum real-silence borrowing window
Purpose

What problem does the project solve?

Normal translation is not enough for dubbing. A correct translated sentence may take 30–70% longer to speak than the original. If software blindly stretches or speeds audio to fit, the result sounds robotic. This project treats translation, timing, speech generation and timeline placement as one coordinated system.

Meaning before speed

Names, numbers, amounts, percentages, negations, claims and instructions should survive timing adaptation. AI is used to rewrite long dialogue more efficiently, not to delete facts.

Natural timing

Short speech is accepted as short. Remaining room stays silent. Overlong speech is rewritten and regenerated instead of being aggressively time-stretched.

Source-timeline fidelity

Speech stays anchored to the original onset and neighboring cues. Conservative semantic merges remove artificial micro-deadlines without moving the whole timeline.

Pipeline

How it works, step by step

1

Understand the source

Gemini analyzes the public video URL or local media, producing speaker-aware timestamps, transcription and target-language translation. A reusable checkpoint prevents repeated video analysis when the same source is processed again.

2

Build semantic speech units

Adjacent same-speaker fragments can be conservatively merged when they are clearly one unfinished sentence. v0.5.10 also bridges an ultra-short heading across a small real pause when the next same-speaker cue is the obvious longer continuation.

3

Run the AI Timing Director

Before TTS, translated text is evaluated against its source slot. Only text predicted to be too long is compressed. Cloud mode currently targets about 72% pre-TTS occupancy and processes timing work in batches of ten.

4

Synthesize natural speech

Cloud validation uses Edge Neural TTS as the primary speech engine, while the project also contains Gemini TTS support and fallback orchestration. The project does not clone the source voice.

5

Measure, do not guess

Generated audio duration is measured. If it would require more than the hard 1.10× speed ceiling, Gemini receives the exact ratio and rewrites the text again. Up to three measured feedback passes are allowed.

6

Assemble the master timeline

FFmpeg places speech at source positions, preserves silence, performs required audio operations and creates the dubbing track. Local finalization can combine the package with the source video into MP4.

Technical architecture

What is under the hood?

LayerCurrent roleEngineering choice
AI understanding / timingTranscription, translation, semantic compression and timing feedbackgemini-3.5-flash-lite in the current Cloud workflow; model names remain environment-configurable.
SpeechDubbing synthesisEdge Neural TTS is primary in the validated Cloud path; Gemini TTS support remains available in the codebase.
SynchronizationSource-timestamp alignmentsegment_locked / onset-locked design, semantic continuation merging, micro-cue bridge, real-silence borrowing ≤ 1.50 s.
Timing controlAvoid rushed or slow-motion voicePre-TTS Timing Director + measured post-TTS feedback; hard speed ceiling 1.10×; short speech is never expanded just to fill time.
MediaAudio timing, mixing and final muxFFmpeg via imageio-ffmpeg; no pydub/audioop dependency, keeping Python 3.13 compatibility.
Local UIInteractive desktop/browser workflowStreamlit, plus CLI and Windows/macOS/Linux launch paths.
CloudRemote processingGitHub Actions with encrypted GEMINI_API_KEY, reusable transcript/TTS cache and downloadable artifacts.
TestingRegression protectionCI matrix for Python 3.11, 3.12 and 3.13, including timing, API-shape and cloud-workflow tests.
Semantic LockMicro-Cue BridgeOnset Locked

Why semantic units matter

A subtitle boundary is not always a speech boundary. The system can merge only carefully defined same-speaker continuations, removing impossible 1–2 second deadlines while preserving the first onset and final end time.

Quota awareCacheCheckpoint

Why the cloud path is hybrid

GitHub runner IPs can be blocked by YouTube media delivery. The cloud workflow therefore lets Gemini understand the public URL and builds audio/SRT/JSON remotely, while final source-video download and mux can be completed locally.

Verified achievement

What has the project actually achieved?

The important milestone is not a demo screenshot; it is a completed end-to-end Cloud Dub package after repeated real failures exposed timing, quota, model-availability and micro-cue edge cases.

Cloud Dub #41 — successful validation

The v0.5.10 semantic micro-cue fix reduced the source to 37 timestamp-locked speech units and completed all 37. Difficult chunks were measured, AI-compressed and re-synthesized while the 1.10× hard limit remained unchanged. The workflow then built and uploaded the complete cloud dubbing package.

37 / 37speech chunks completed
100%Cloud workflow conclusion: success
~44.8 MBfinal artifact ZIP
45files uploaded in the package
≤ 1.10×natural-rate speed policy preserved

Resilience learned from failure

The project survived API-format changes, Python 3.13 audio dependency issues, 429 quota exhaustion, 503 capacity failures, retired model names and timing-convergence failures. Each failure became a regression rule or architectural change.

Auditable output

Successful packages include dubbed_audio.wav, dubbed.srt, transcript.json, manifest.json and timing diagnostic files so the process is inspectable rather than opaque.

Practical multilingual base

Persian is the default target, but the pipeline accepts other target languages and can be driven locally, by CLI, Docker or GitHub Actions.

Reality check

Current limitations

Not phoneme-level lip sync

Synchronization is timestamp/utterance based. It can sound naturally aligned, but it does not yet move a face or solve viseme-level mouth matching.

Cloud finalization is still hybrid

The cloud package is complete for dubbing audio and metadata, but YouTube bot protection can prevent GitHub from reliably downloading the source video. Final MP4 mux may therefore happen on the user machine.

Provider quotas can still change

Gemini availability, free-tier limits and model names are external constraints. The code is configurable and quota-aware, but it cannot create provider capacity.

No source voice cloning

The current project uses prebuilt voices. This is safer and simpler, but it does not reproduce each original speaker's identity or emotion exactly.

Responsible use matters: only dub media you own, are authorized to adapt, or are licensed to reuse. API keys belong in local environment files or encrypted GitHub secrets — never in source control.
Future roadmap

What should be added next?

These are engineering opportunities, not promises. The highest-value work is the work that makes quality measurable, reduces repeated API cost and removes the final manual step.

Priority 1

Persistent timing-rewrite memory

Cache successful AI timing rewrites by source text, target language, voice and timing slot so the same hard chunk never consumes Gemini quota twice.

Priority 1

Fully automated final MP4 path

Add reliable user-upload/object-storage ingestion or another authorized media source so GitHub can mux the final MP4 without depending on blocked YouTube runner downloads.

Priority 1

Automatic quality gate

Back-transcribe generated audio and score semantic fidelity, names/numbers, timing, loudness, clipping and missing speech before an artifact is declared successful.

Priority 2

Editable transcript & timing studio

Provide a visual timeline where users can edit translation, merge/split cues, lock terminology, choose a voice per speaker and regenerate only selected regions.

Priority 2

Terminology / translation memory

Add project dictionaries for names, brands, technical vocabulary, units and preferred translations, with explicit protection for numbers and negation.

Priority 2

Multi-speaker scaling

Go beyond two voice roles with better diarization, speaker profiles and consistent voice assignment across long videos and episode series.

Priority 3

Optional consent-based voice matching

Introduce voice similarity or cloning only with explicit consent, provenance records and clear disclosure, while retaining prebuilt voices as the safe default.

Priority 3

Viseme / lip-sync integration

Generate phoneme or viseme timing and optionally connect to a lip-sync model for close-up talking-head content where mouth alignment matters.

Priority 3

Batch queue and observability

Add a dashboard for multiple jobs, cost/request counters, cache hit rate, provider latency, failure classification and per-language quality history.

Productization

Standalone desktop builds

Package Windows, macOS and Linux binaries so non-Python users can run the local workflow, update safely and finalize cloud packages with one click.