File size: 2,276 Bytes
16dc4f2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
QVHighlights Evaluation and Codalab Submission
==================
### Task Definition
Given a video and a natural language query, our task requires a system to retrieve the most relevant moments in the video, and detect the highlightness of the clips in the video.
### Evaluation
At project root, run
```
bash standalone_eval/eval_sample.sh
```
This command will use [eval.py](eval.py) to evaluate the provided prediction file [sample_val_preds.jsonl](sample_val_preds.jsonl),
the output will be written into `sample_val_preds_metrics.json`.
The content in this generated file should be similar if not the same as [sample_val_preds_metrics_raw.json](sample_val_preds_metrics_raw.json) file.
### Format
The prediction file [sample_val_preds.jsonl](sample_val_preds.jsonl) is in [JSON Line](https://jsonlines.org/) format, each row of the files can be loaded as a single `dict` in Python. Below is an example of a single line in the prediction file:
```
{
"qid": 2579,
"query": "A girl and her mother cooked while talking with each other on facetime.",
"vid": "NUsG9BgSes0_210.0_360.0",
"pred_relevant_windows": [
[0, 70, 0.9986],
[78, 146, 0.4138],
[0, 146, 0.0444],
...
],
"pred_saliency_scores": [-0.2452, -0.3779, -0.4746, ...]
}
```
| entry | description |
| --- | ----|
| `qid` | `int`, unique query id |
| `query` | `str`, natural language query, not used by the evaluation script |
| `vid` | `str`, unique video id |
| `pred_relevant_windows` | `list(list)`, moment retrieval predictions. Each sublist contains 3 elements, `[start (seconds), end (seconds), score]`|
| `pred_saliency_scores` | `list(float)`, highlight prediction scores. The higher the better. This list should contain a score for each of the 2-second clip in the videos, and is ordered. |
### Codalab Submission
To test your model's performance on `test` split,
please submit both `val` and `test` predictions to our
[Codalab evaluation server](https://codalab.lisn.upsaclay.fr/competitions/6937).
The submission file should be a single `.zip ` file (no enclosing folder)
that contains the two prediction files
`hl_val_submission.jsonl` and `hl_test_submission.jsonl`, each of the `*submission.jsonl` file
should be formatted as instructed above.
|