QVHighlights Evaluation and Codalab Submission
==================

### Task Definition
Given a video and a natural language query, our task requires a system to retrieve the most relevant moments in the video, and detect the highlightness of the clips in the video. 

### Evaluation
At project root, run
```
bash standalone_eval/eval_sample.sh 
```
This command will use [eval.py](eval.py) to evaluate the provided prediction file [sample_val_preds.jsonl](sample_val_preds.jsonl), 
the output will be written into `sample_val_preds_metrics.json`. 
The content in this generated file should be similar if not the same as [sample_val_preds_metrics_raw.json](sample_val_preds_metrics_raw.json) file.

### Format

The prediction file [sample_val_preds.jsonl](sample_val_preds.jsonl) is in [JSON Line](https://jsonlines.org/) format, each row of the files can be loaded as a single `dict` in Python. Below is an example of a single line in the prediction file:
```
{
  "qid": 2579,
  "query": "A girl and her mother cooked while talking with each other on facetime.",
  "vid": "NUsG9BgSes0_210.0_360.0",
  "pred_relevant_windows": [
    [0, 70, 0.9986],
    [78, 146, 0.4138],
    [0, 146, 0.0444],
    ...
  ],  
  "pred_saliency_scores": [-0.2452, -0.3779, -0.4746, ...]
}

```



| entry | description |
| --- | ----|
| `qid` | `int`, unique query id |
| `query` | `str`, natural language query, not used by the evaluation script | 
| `vid` | `str`, unique video id | 
| `pred_relevant_windows` | `list(list)`, moment retrieval predictions. Each sublist contains 3 elements, `[start (seconds), end (seconds), score]`| 
| `pred_saliency_scores` | `list(float)`, highlight prediction scores. The higher the better. This list should contain a score for each of the 2-second clip in the videos, and is ordered. | 


### Codalab Submission
To test your model's performance on `test` split, 
please submit both `val` and `test` predictions to our 
[Codalab evaluation server](https://codalab.lisn.upsaclay.fr/competitions/6937). 
The submission file should be a single `.zip ` file (no enclosing folder) 
that contains the two prediction files 
`hl_val_submission.jsonl` and `hl_test_submission.jsonl`, each of the `*submission.jsonl` file 
should be formatted as instructed above.