It did not reproduce. Here is why.
A from-scratch PyTorch reproduction of progressive RL for video summarization.
| Run | τ |
|---|---|
| Human, leave-one-out | 0.312 |
| PRLVS as published | 0.080 |
| Random scores | −0.004 |
| PRLVS, repaired reward | −0.004 |
| PRLVS, paper-faithful | −0.008 |
30 rollouts. The paper averages τ per annotator; its human figure is 0.177.
F1 cannot tell it from random
Random scores land about 2 points below humans. Whole-shot selection does most of the work.
| Selector | F1, mean over annotators | vs random |
|---|---|---|
| PRLVS, paper-faithful | 56.1% [51.8, 60.9] | p = 0.71 |
| PRLVS, repaired | 54.5% [49.9, 59.5] | p = 0.15 |
| Random scores | 54.9% [50.2, 59.8] | |
| Importance model, held out | 58.1% [53.4, 63.1] | p = 0.03 |
| Humans, each against the rest | 57.0% [52.9, 61.5] | |
| PRLVS as published | 63.0% |
KTS shots, a 15% knapsack, Wilcoxon over 50 videos. The paper’s KTS settings are unstated, so 56.1 against 63.0 does not say 63 is wrong.
The reward, not the optimiser
One scoring pipeline, 150 rollouts, scale −0.05 to 0.75. Blue: the trained agent. Brackets: 95% bootstrap.
| Selector | τ |
|---|---|
| Oracle: samples by human score | 0.729 [0.714, 0.744] |
| Greedy on the paper’s reward | 0.006 [−0.011, 0.022] |
| Greedy on the repaired reward | 0.008 [−0.010, 0.026] |
| PRLVS, trained, faithful | −0.006 [−0.025, 0.013] |
| PRLVS, trained, repaired | 0.019 [0.002, 0.037] |
| PRLVS, untrained | 0.001 [−0.018, 0.020] |
| Uniform random frames | −0.012 [−0.034, 0.012] |
The pipeline is not the ceiling
Knowing the human scores reaches 0.729 through it.
The reward carries no human signal
Maximising it directly scores no better than random.
Training changes nothing measurable
Against the untrained network: p = 0.44 faithful, 0.15 repaired.
A fix, half working
A held-out importance model joins the reward. Moves of ±1 or ±5 from random starts cap even direct maximisation at 0.064; the agent learns less.
- Importance model alone [0.193, 0.291]
- 0.243
- Greedy on the new reward [0.042, 0.085]
- 0.064
- PRLVS on the new reward [−0.026, 0.012]
- −0.007
Every unstated decision, written down
The paper cannot be implemented as printed. PROVENANCE.md records each choice: from the paper, from the reference reproduction, or decided here, and where Eq. 4 and Eq. 13 are ambiguous.
At K = 10 the literal reward is a product of at most 1e-10; the repaired variant computes it in the log domain.
Run
<tvsum_dir>: the extracted ydata-tvsum50-v1_1 folder from yalesong/tvsum.
pip install -r requirements.txt
python extract_googlenet.py <tvsum_dir>
TVSUM_ROOT=<tvsum_dir> PY=python ./run_all.sh
python diagnose_eval.py features_tvsum.npz <tvsum_dir> 30 150
python std_f1.py features_tvsum.npz <tvsum_dir>
# the fix experiment: three more commands in the README