Samuel Albanie*, Gül Varol*, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox and Andrew Zisserman, BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues, ECCV 2020.

[Project page] [arXiv]

• Setup
• Demo
• Train and Test
  • Datasets
  • Pretrained models
  • Train
  • Test
• Experiments
  • Experiments on BSL-1K
  • Experiments on Transfer
• Note on BSL-1K data release
• Limitations
• Citation

Requires: python 3.6. (some non-essential pre-processing scripts require python 3.7)

# Clone this repository
git clone https://github.com/gulvarol/bsl1k.git
cd bsl1k/
# Setup symbolic links (point these to folders where you would like data and checkpoints to be stored)
ln -s <replace_with_data_path> data
ln -s <replace_with_log_path> checkpoint
# Create bsl1k_env environment with dependencies
conda env create -f environment.yml
conda activate bsl1k_env
pip install -r requirements.txt

The demo folder contains a sample script to apply sign language recognition on an input video. By default, the demo will download: (1) a model that has been pretrained on BSL-1K and then fine-tuned on WLASL, (2) a video from handspeak.com (this particular video is part of the the WLASL test set). The demo should produce the output below (you can change to other inputs):

Usage: run python demo.py.

The original video source can be found here. Copyright Jolanta Lapiak.

• This code supports I3D classification training for the following sign language video datasets:

• Please cite the original papers for WLASL, MSASL, Phoenix2014T and BSL-Corpus datasets. Here, we only provide pre-processed metadata, but not the videos, which can instead be obtained via the metadata provided by the dataset authors, as described next:

WLASL: First head to the WLASL authors' github page here and download the .json file of links. This file evolves over time, the current version is v3 and is called WLASL_v0.3.json . Place the downloaded file at the location data/wlasl/info/WLASL_v0.3.json. After this step, video files can be downloaded by running the following command:

python misc/wlasl/download_wlasl.py

Notes: some videos may no longer be accessible - you can contact the WLASL authors to address this issue (they provide an email address on the github page linked above). Also note that the v3 json may produce slightly different results from the WLASL_v0.1.json we used for our experiments.

MSASL: As for the dataset above, first download the json files of video links from the authors here and place them into data/msasl/info/. This will create a file directory structure as follows:

data/
   msasl/
       info/
          MSASL_train.json
          MSASL_val.json
          MSASL_test.json

The videos may then be downloaded via:

python misc/msasl/download_msasl.py

Phoenix2014T: video files can be downloaded from here (this file should be unpacked to the location data/PHOENIX-2014-T-release-v3). You can then run the following command script to create .mp4 video files from the provided .png frames:

python misc/phoenix2014/gather_frames.py

BSL-Corpus: can be downloaded from here upon request from the owners.

• In our folder organization, each dataset has a subfolder info/ in which most pre-extracted annotations are kept:
  • info/info.pkl
  • info/pose.pkl OpenPose is extracted for:
    • bsl1k for a subset of the videos
    • msasl and wlasl for all videos (we provide these within the .tar files)
• We have pre-processed all of the video datasets to be at 256x256 spatial resolution. The pre-processing scripts can be found under the misc folder for each dataset. Using the original videos is possible, but is slower.
• We have pre-processed WLASL and MSASL such that the video frames are stored in a pkl file, we then loaded the entire dataset in RAM. Setting --ram_data to 0 will not require this preprocessing step, and use the video files instead. The results are similar with and without this step.

You can download some of the pretrained models used in the experiments by running bash misc/pretrained_models/download.sh in the project root directory. All the other pretrained models from the experiments are provided in the Experiments section. The best BSL-1K model reported for the final experiments is the first model.

Note 2021.09.14: You might want to check an improved model here from our follow-up CVPR'21 work.

The training launch for each experiment can be found in the Experiments section by clicking "run" links. The training can be ran by directly typing python main.py <args> on terminal with the arguments. We also provide the exp/create_exp.py script that we used when launching experiments. You can use that via:

cd exp/
# Change config.json contents
python train.py

cd exp/
# Change config.json contents
python test.py

• Best model BSL-1K(m.5), last 20 frames, video pose pretrained

• Trade-off between training noise vs. size

• Contribution of individual cues (pose subset of the data)

• Effect of pretraining

• The effect of the temporal window for KWS

• The effect of the number of frames before the mouthing peak

• WLASL dataset (isolated) - 64 frames input

• MSASL dataset (isolated) - 64 frames input

• Phoenix2014T dataset (co-articulated) - 16 frames input

• BSL-Corpus dataset subset (co-articulated) - 16 frames input

We are in the process of finalising legal confirmation from our broadcasting partners before we release data.

We would like to emphasise that this research represents a working progress towards achieving automatic sign language recognition, and as such, has a number of limitations that we are aware of (and likely many that we are not aware of). Key limitations include:

• The data collected with our technique is long-tailed (this can be seen in Fig. 2 of our paper, referenced below). This reflects the nature of how signs are used in reality, but it also makes it challenging to train existing vision models (which prefer balanced data).
• All data collected here is interpreted. Interpreted data differs from conversations between native signers (see e.g. this paper for a discussion on this point).
• Our approach naturally biases the annotated data towards mouthings (signs that are not frequently mouthed, or signers who do not mouth, are less represented).

If you use this code, please cite the following:

@INPROCEEDINGS{albanie20_bsl1k,
  title     = {{BSL-1K}: {S}caling up co-articulated sign language recognition using mouthing cues},
  author    = {Albanie, Samuel and Varol, G{\"u}l and Momeni, Liliane and Afouras, Triantafyllos and Chung, Joon Son and Fox, Neil and Zisserman, Andrew},
  booktitle = {ECCV},
  year      = {2020}
}