Improve README

Author: Niels Rogge

examples/pytorch/image-classification/README.md

@@ -14,13 +14,14 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-# Image classification examples
+# Image classification example
+
+This directory contains a script, `run_image_classification.py`, that showcases how to fine-tune any model supported by the [`AutoModelForImageClassification` API](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForImageClassification) (such as [ViT](https://huggingface.co/docs/transformers/main/en/model_doc/vit), [ConvNeXT]((https://huggingface.co/docs/transformers/main/en/model_doc/convnext)), [ResNet]((https://huggingface.co/docs/transformers/main/en/model_doc/resnet)), [Swin Transformer]((https://huggingface.co/docs/transformers/main/en/model_doc/swin))...) using PyTorch. It can be used to fine-tune models on both well-known datasets (like [CIFAR-10](https://huggingface.co/datasets/cifar10), [Fashion MNIST](https://huggingface.co/datasets/fashion_mnist), ...) as well as on your own custom data.
 
-The following examples showcase how to fine-tune a `ViT` for image-classification using PyTorch.
 
 ## Using datasets from 🤗 `datasets`
 
-Here we show how to fine-tune a `ViT` on the [beans](https://huggingface.co/datasets/beans) dataset.
+Here we show how to fine-tune a Vision Transformer (`ViT`) on the [beans](https://huggingface.co/datasets/beans) dataset, to classify the disease type of bean leaves.
 
 👀 See the results here: [nateraw/vit-base-beans](https://huggingface.co/nateraw/vit-base-beans).
 
@@ -46,36 +47,21 @@ python run_image_classification.py \
     --seed 1337
 ```
 
-Here we show how to fine-tune a `ViT` on the [cats_vs_dogs](https://huggingface.co/datasets/cats_vs_dogs) dataset.
+To fine-tune another model, simply provide the `--model_name_or_path` argument. To train on another dataset, simply set the `--dataset_name` argument. 
 
 👀 See the results here: [nateraw/vit-base-cats-vs-dogs](https://huggingface.co/nateraw/vit-base-cats-vs-dogs).
 
-```bash
-python run_image_classification.py \
-    --dataset_name cats_vs_dogs \
-    --output_dir ./cats_vs_dogs_outputs/ \
-    --remove_unused_columns False \
-    --do_train \
-    --do_eval \
-    --push_to_hub \
-    --push_to_hub_model_id vit-base-cats-vs-dogs \
-    --fp16 True \
-    --learning_rate 2e-4 \
-    --num_train_epochs 5 \
-    --per_device_train_batch_size 32 \
-    --per_device_eval_batch_size 32 \
-    --logging_strategy steps \
-    --logging_steps 10 \
-    --evaluation_strategy epoch \
-    --save_strategy epoch \
-    --load_best_model_at_end True \
-    --save_total_limit 3 \
-    --seed 1337
-```
-
 ## Using your own data
 
-To use your own dataset, the training script expects the following directory structure:
+To use your own dataset, there are 2 ways: 
+- you can either provide your own folders as `--train_dir` and/or `--validation_dir` arguments
+- you can upload your dataset to the hub (possibly as a private repo, if you prefer so), and simply pass the `--dataset_name` argument.
+
+Below, we explain both in more detail.
+
+### Provide them as folders
+
+If you provide your own folders with images, the script expects the following directory structure:
 
 ```bash
 root/dog/xxx.png
@@ -87,22 +73,55 @@ root/cat/nsdf3.png
 root/cat/[...]/asd932_.png
 ```
 
-Once you've prepared your dataset, you can can run the script like this:
+In other words, you need to organize your images in subfolders, based on their class. You can then run the script like this:
 
 ```bash
 python run_image_classification.py \
-    --dataset_name nateraw/image-folder \
     --train_dir <path-to-train-root> \
     --output_dir ./outputs/ \
     --remove_unused_columns False \
     --do_train \
     --do_eval
 ```
 
-### 💡 The above will split the train dir into training and evaluation sets
+Internally, the script will use the [`ImageFolder`](https://huggingface.co/docs/datasets/v2.0.0/en/image_process#imagefolder) feature which will automatically turn the folders into a 🤗 Dataset object.
+
+#### 💡 The above will split the train dir into training and evaluation sets
   - To control the split amount, use the `--train_val_split` flag.
   - To provide your own validation split in its own directory, you can pass the `--validation_dir <path-to-val-root>` flag.
 
+### Upload your data to the hub, as a (possibly private) repo
+
+It's very easy (and convenient) to upload your image dataset to the hub using the [`ImageFolder`](https://huggingface.co/docs/datasets/v2.0.0/en/image_process#imagefolder) feature available in 🤗 Datasets. Simply do the following:
+
+```python
+from datasets import load_dataset
+
+# example 1: local folder
+dataset = load_dataset("imagefolder", data_dir="path_to_your_folder")
+
+# example 2: local files (suppoted formats are tar, gzip, zip, xz, rar, zstd)
+dataset = load_dataset("imagefolder", data_files="path_to_zip_file")
+
+# example 3: remote files (suppoted formats are tar, gzip, zip, xz, rar, zstd)
+dataset = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip"
+
+# example 4: providing several splits
+dataset = load_dataset("imagefolder", data_files={"train": ["path/to/file1", "path/to/file2"], "test": ["path/to/file3", "path/to/file4"]})
+```
+
+`ImageFolder` will create a `label` column, and the label name is based on the directory name.
+
+Next, push it to the hub!
+
+```python
+dataset.push_to_hub("name_of_your_dataset")
+
+# if you want to push to a private repo, simply pass private=True:
+dataset.push_to_hub("name_of_your_dataset", private=True)
+```
+
+and that's it! You can now simply train your model simply by setting the `--dataset_name` argument to the name of your dataset on the hub.
 
 ## Sharing your model on 🤗 Hub
 
@@ -116,13 +135,21 @@ $ git config --global user.email "[email protected]"
 $ git config --global user.name "Your Name"
 ```
 
-2. Log in with your HuggingFace account credentials using `huggingface-cli`
+2. Log in with your HuggingFace account credentials using `huggingface-cli`:
 
 ```bash
 $ huggingface-cli login
 # ...follow the prompts
 ```
 
+or, in case you're running in a notebook:
+
+```python
+from huggingface_hub import notebook_login
+
+notebook_login()
+```
+
 3. When running the script, pass the following arguments:
 
 ```bash

examples/pytorch/image-classification/run_image_classification.py

@@ -72,14 +72,11 @@ def pil_loader(path: str):
 class DataTrainingArguments:
     """
     Arguments pertaining to what data we are going to input our model for training and eval.
-    Using `HfArgumentParser` we can turn this class
-    into argparse arguments to be able to specify them on
-    the command line.
+    Using `HfArgumentParser` we can turn this class into argparse arguments to be able to specify
+    them on the command line.
     """
 
-    dataset_name: Optional[str] = field(
-        default="nateraw/image-folder", metadata={"help": "Name of a dataset from the datasets package"}
-    )
+    dataset_name: Optional[str] = field(metadata={"help": "Name of a dataset from the hub."})
     dataset_config_name: Optional[str] = field(
         default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."}
     )
@@ -104,6 +101,11 @@ class DataTrainingArguments:
     )
 
     def __post_init__(self):
+        if self.dataset_name is None and (self.train_dir is None and self.validation_dir is None):
+            raise ValueError(
+                "You must specify either a dataset name from the hub or a train and/or validation directory."
+            )
+
         data_files = dict()
         if self.train_dir is not None:
             data_files["train"] = self.train_dir
@@ -201,14 +203,18 @@ def main():
             )
 
     # Initialize our dataset and prepare it for the 'image-classification' task.
-    ds = load_dataset(
-        data_args.dataset_name,
-        data_args.dataset_config_name,
-        data_files=data_args.data_files,
-        cache_dir=model_args.cache_dir,
-        task="image-classification",
-        use_auth_token=True if model_args.use_auth_token else None,
-    )
+    if data_args.dataset_name is not None:
+        ds = load_dataset(
+            data_args.dataset_name,
+            data_args.dataset_config_name,
+            cache_dir=model_args.cache_dir,
+            task="image-classification",
+            use_auth_token=True if model_args.use_auth_token else None,
+        )
+    else:
+        ds = load_dataset(
+            "imagefolder", data_files=data_args.data_files, cache_dir=model_args.cache_dir, task="image-classification"
+        )
 
     # If we don't have a validation split, split off a percentage of train as validation.
     data_args.train_val_split = None if "validation" in ds.keys() else data_args.train_val_split