Video Agent

A long video, summarised and questioned

Keyframes picked by an RL policy, answers grounded in timestamps.

Built by Akshit Gupta

Read the code Read the finding

GET /tasks/{id}/stream, one upload

  1. 5Using uploaded video
  2. 38Extracting frames
  3. 48Extracting CLIP features
  4. 57Loading RL policies
  5. 61 to 80Running RL summarization policy
  6. 83Detecting scenes
  7. 90Creating summary video
  8. 94Merging audio and video
  9. 97Generating speech transcript
  10. 99Preparing highlights
  11. 100Done
Stage names and percentages as the worker reports them. The API returns a task id at once; the stream sends a heartbeat when nothing changes and gives up after 10 minutes.

Keyframes, one move at a time

Picking 25 of 80 sampled frames is a combinatorial search, so two policies edit a current selection, 40 moves per video.

  1. 01

    Which frame

    The horizontal policy picks one of the 25 to move.

  2. 02

    Which way

    The vertical policy moves it −1, +1, −5 or +5 sampled frames.

  3. 03

    Reward

    tanh(0.4·importance + 0.3·diversity + 0.3·coverage) on CLIP features.

  4. 04

    Summary

    Greedy moves from a random start, aimed at 20% of the running time.

What measurement showed

In deterministic rollout the policy collapses onto a few keyframe slots. Against TVSum’s human annotators, its CLIP-distance importance scores slightly below random: it rewards outlier frames over representative ones. The evaluation code lives in the reproduction.

That is why I rebuilt the published method from scratch: the PRLVS reproduction.

Asking a video questions

Transcript
Faster-Whisper, cached by the file’s SHA-256
Chunks
Up to 200 words, each keeps its timestamps
Index
bge-small-en-v1.5, one LlamaIndex index per video
Answer
Claude if a key is set, else the matching chunks alone

Six endpoints, one stream

EndpointPurpose
POST /summarize/uploadQueue a job for an uploaded file
POST /summarize/youtubeQueue a job for a YouTube URL
GET /tasks/{id}Job state and result
GET /tasks/{id}/streamServer-Sent Events progress
POST /rag/queryAnswers with timestamp ranges
POST /agent/transcript-insightsBullets from a LangChain chain, or the first sentences without a key

The Celery worker runs the single-process solo pool: native video libraries crash under fork on macOS.

Run it

Docker brings up the API, worker, Redis and a Streamlit front end. Without Docker, FFmpeg must be on PATH. CI runs the smoke tests on CPU-only PyTorch.

touch .env   # API keys optional
docker compose up --build

# or without Docker
pip install -r requirements.txt
redis-server &
celery -A celery_app worker --pool=solo &
uvicorn app.api:app --reload