• Linux machine with four GPUs
• Conda
• Python's dependencies: this file:

conda create --name research --file spec-file.txt
conda activate research

git clone [email protected]:NVIDIA/apex.git
cd apex
git checkout 4ef930c1c884fdca5f472ab2ce7cb9b505d26c1a  # to fix the exact library version
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch

In NLP experiments, we need similar words constructed from fasttext's pre-trained word embeddings.

cd code
wget https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip
python nlp_construct_simiar_data.py

Please fill the value of replace_data in conf/dataset/ag_news.yaml with the generated file's path. For example,

replace_data: /home/nzw/code/ag_replace_ids.npy

Please run content of all scripts in ./scripts/**/train/ under code.

After training, please run ./gather_weights.py to generate text files for evaluation.

Please run content of all scripts in ./scripts/**/eval/ under code as well.

For the AG news dataset, please run code/notebooks/filter_ag_news.ipynb after evaluation of mean classifier before the other evaluation scripts such as linear classifier and bound computation.

To obtain all figures and tables in the paper, you run notebooks in code/notebooks/. The codes save generated figures and tables into ./doc/figs and ./doc/tabs, respectively.

• code/notebooks/bound.ipynb generates Figure 1, the content of Table 2, and Tables 4 and 5.
• code/notebooks/coupon.ipynb generates Table 3.
• code/notebooks/collision.ipynb computes upper bound of the collision term used in code/notebooks/bound.ipynb.
• code/notebooks/collosion_analysis.ipynb generates Figures 3, 4, and 5.

• Openreview
• arXiv
• Slides
• NeurIPS Poster
• Video
• Video in Japanese

@inproceedings{NS2021,
    title     = {Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning},
    author    = {Kento Nozawa, Issei Sato},
    year      = {2021},
    booktitle = {NeurIPS},
    pages     = {5784--5797},
}