Design YouTube — upload, process, adaptive bitrate stream

Design YouTube

Upload and stream videos at YouTube scale: presigned S3 multipart, segment + multi-format processing DAG, adaptive bitrate streaming, resumable uploads, CDN and metadata cache — plus mid/senior/staff interview bars.

Understanding the problem

YouTube is a video-sharing platform: users upload, view, and interact with video. As of this writing it is among the most visited sites on the web. There is conceptual overlap with designing Dropbox — if you are less familiar with file upload/download system design, start with that guide first.

Cinema reel versus short streaming clips with quality switching.
Whole-reel download loses to short clips you can switch quality on mid-watch.
01Upload

Presigned multipart.

02Process

Segments + formats.

03Stream

ABR + manifest.

04Scale

CDN + meta cache.

Pair with Delivery framework, Design Dropbox, Handling large blobs, Long-running tasks, and Scaling reads.

Functional requirements

Non-functional requirements

Functional and non-functional requirements for Design YouTube.
Upload + watch above the line; HA, large files, ABR, scale, resume define the hard parts.

Planning the approach

Before drawing boxes, plan: build the design sequentially through functional requirements, then use NFRs to drive deep dives. Stay focused — do not get lost in search or recommendations.

Core entities

Start with a broad overview — not every column. Align vocabulary with the interviewer.

User, Video, and VideoMetadata entities.
User · Video · VideoMetadata — details grow in HLD.

The API

Define an endpoint per functional requirement early — then expect APIs to evolve as trade-offs appear. Say that out loud: "I'll outline simple APIs and may revise them as we go deeper."

Naive upload API versus evolved presigned URL and metadata GET.
APIs evolve from POST /upload to POST /presigned_url as you adopt direct-to-S3.

Background: video streaming basics

You do not need to be a video engineer. You do need vocabulary so you can navigate trade-offs.

1) Users can upload videos

When uploading, pin three questions: where is metadata? where are bytes? what do we store for video data?

~1M uploads/day → ~365M video rows/year. Prefer a horizontally partitioned store such as Cassandra, partitioned by videoId — we mostly do point lookups, not bulk scans. Partitioning matters when you need co-located reads or domain consistency (e.g. Ticketmaster by concert); here point lookup by videoId is enough.

For bytes, reuse the Dropbox lesson: upload directly to S3 via a presigned URL with multipart upload. That changes POST /uploadPOST /presigned_url (metadata only).

Client, API gateway, Video Service, S3, and Video Metadata DB for presigned upload.
Users can upload videos — presigned URL, direct-to-S3, metadata in the DB.

What to store for video data

Presigned upload to S3, processing service stores multiple formats, metadata DB holds S3 URLs.
Store different video formats — S3 event → processing → URLs in metadata.

2) Users can watch videos

Fetch VideoMetadata (manifest URL(s)). Evolve GET /video to return metadata only — bytes live in object storage / CDN.

Segmented storage is not the same as "chunked HTTP download of one file." Some formats happen to be playable when read in order; playable segments make incremental playback explicit and unlock ABR.

ABR client loop

  1. Fetch VideoMetadata → URL to primary manifest in S3/CDN.
  2. Download the manifest.
  3. Choose a format from network / user settings; pull first segment URL from the media manifest.
  4. Play; prefetch more segments.
  5. If bandwidth worsens (or improves), switch to lower (or higher) bitrate segments without stopping playback.
Client downloads video segments from S3 after processing splits and formats them.
Download segments incrementally — play while prefetching more clips.

Deep dive: processing for ABR

Smooth playback needs ABR, which needs segments + formats + manifests produced after upload. Post-processing is a pipeline.

  1. Split the original into segments (e.g. ffmpeg).
  2. Transcode each segment (and related work: audio, transcripts) — parallel across workers.
  3. Create manifests referencing formats / segments.
  4. Mark the upload complete; update VideoMetadata.

Work forms a DAG: one-way dependencies, fan-out on independent segment transcodes (CPU-bound — parallelize hard). Orchestrate with something like Temporal. Pass temporary artifacts via S3 URLs between workers — do not ship multi-GB blobs through the orchestrator.

Video Processing Service DAG: splitter, parallel transcoding and audio, build manifests, mark done.
How can we handle processing a video to support adaptive bitrate streaming?

Deep dive: resumable uploads

Strong overlap with Dropbox large-file upload. Track progress of the original upload:

  1. Client divides the file into ~5–10MB chunks with fingerprint hashes.
  2. VideoMetadata stores chunks[] with fingerprint + status (NotUploaded / Uploaded).
  3. Client POSTs chunk list, then uploads parts to S3.
  4. S3 returns part number + ETag; client PATCHes the backend so the server can verify and mark the chunk Uploaded.
  5. On CompleteMultipartUpload, S3 emits an object-created notification once — kick processing. Chunk progress remains client-driven as above.
  6. On resume: fetch metadata, skip already-Uploaded chunks.

In practice this is AWS multipart upload — naming the steps shows depth.

Client chunks with fingerprints, VideoMetadata status, S3 multipart, then processing on complete.
Chunk fingerprints + multipart ETags · Complete kicks the DAG.

Deep dive: scale (1M upload / 100M watch)

Hot video metadata: raise RF so several nodes share load; add a distributed LRU cache partitioned by videoId in front of Cassandra.

Geo latency: CDN caches popular manifests and segments near users. Once warm, streaming may never touch your origin for the rest of playback.

Upload Monitor (Lambda): S3 part/object events can also drive an upload monitor that writes chunk progress into VideoMetadata — complements client-driven resume tracking from the resumable-uploads deep dive.

Full scale design with CDN, metadata cache, upload monitor Lambda, and processing DAG.
How do we scale to a large number of videos uploaded / watched a day?

Final design

Final YouTube design with CDN, metadata cache, upload monitor, S3, and processing DAG.
Final design — CDN + cache + processing DAG + ABR download path.

What is expected at each level?

Mid, Senior, and Staff-plus expectations for Design YouTube.
Breadth vs depth shifts from mid → staff+.
Interview takeaway

Mid: solid spine + coachable on blobs/segments. Senior: own processing DAG + multipart resume + ABR. Staff+: anticipate hot spots, CDN, and readiness races like you've shipped them.

Wrapping up

YouTube's interview spine is control plane vs data plane for video: metadata + presigns on your service, durable originals on S3, a DAG that turns one file into segments + manifests, and ABR clients that fetch quality-appropriate clips from a CDN — with resumable multipart for tens of GB and caches for viral metadata reads.

Related: Delivery framework · Design Dropbox · Handling large blobs · Long-running tasks · Scaling reads · Caching · Common patterns.

← Lattice