Structuring the data pipeline in a way that it can be effortlessly linked to your deep learning model is an important aspect of any deep learning-based system. PyTorch packs everything to do just that.<\/p>\n
While in the previous tutorial<\/a>, we used simple datasets, we\u2019ll need to work with larger datasets in real world scenarios in order to fully exploit the potential of deep learning and neural networks.<\/p>\n In this tutorial, you\u2019ll learn how to build custom datasets in PyTorch. While the focus here remains only on the image data, concepts learned in this session can be applied to any form of dataset such as text or tabular datasets. So, here you\u2019ll learn:<\/p>\n Let\u2019s get started.<\/p>\n Loading and Providing Datasets in PyTorch This tutorial is in three parts; they are<\/p>\n A variety of preloaded datasets such as CIFAR-10, MNIST, Fashion-MNIST, etc. are available in the PyTorch domain library. You can import them from torchvision and perform your experiments. Additionally, you can benchmark your model using these datasets.<\/p>\n We\u2019ll move on by importing Fashion-MNIST dataset from torchvision. The Fashion-MNIST dataset includes 70,000 grayscale images in 28\u00d728 pixels, divided into ten classes, and each class contains 7,000 images. There are 60,000 images for training and 10,000 for testing.<\/p>\n Let\u2019s start by importing a few libraries we\u2019ll use in this tutorial.<\/p>\n Let\u2019s also define a helper function to display the sample elements in the dataset using matplotlib.<\/p>\n Now, we\u2019ll load the Fashion-MNIST dataset, using the function Let\u2019s check the class names along with their corresponding labels we have in the Fashion-MNIST dataset.<\/p>\n It prints<\/p>\n Similarly, for class labels:<\/p>\n It prints<\/p>\n Here is how we can visualize the first element of the dataset with its corresponding label using the helper function defined above.<\/p>\n First element of the Fashion MNIST dataset<\/p>\n<\/div>\n In many cases, we\u2019ll have to apply several transforms before feeding the images to neural networks. For instance, a lot of times we\u2019ll need to As you can see below, PyTorch enables us to choose from a variety of transforms.<\/p>\n This shows all available transform functions:<\/p>\n As an example, let\u2019s apply the This prints<\/p>\n As you can see image has now been cropped to $16times 16$ pixels. Now, let\u2019s plot the first element of the dataset to see how they have been randomly cropped.<\/p>\n This shows the following image<\/p>\n Cropped image from Fashion MNIST dataset<\/p>\n<\/div>\n Putting everything together, the complete code is as follows:<\/p>\n Until now we have been discussing prebuilt datasets in PyTorch, but what if we have to build a custom dataset class for our image dataset? While in the previous tutorial<\/a> we only had a simple overview about the components of the Firstly, in the constructor we define the parameters of the class. The and the annotation is a CSV file like the following, located under the root directory of the images (i.e., \u201cattface\u201d above):<\/p>\n where the first column of the CSV data is the path to the image and the second column is the label.<\/p>\n Similarly, we define the Now, we can create our dataset object and apply the transforms on it. We assume the image data are located under the directory named \u201cattface\u201d and the annotation CSV file is at \u201cattface\/imagedata.csv\u201d. Then the dataset is created as follows:<\/p>\n Optionally, you can add the transform function to the dataset as well:<\/p>\n You can use this custom image dataset class to any of your datasets stored in your directory and apply the transforms for your requirements.<\/p>\n In this tutorial, you learned how to work with image datasets and transforms in PyTorch. Particularly, you learned:<\/p>\n The post Loading and Providing Datasets in PyTorch<\/a> appeared first on MachineLearningMastery.com<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/11\/uriel-sc-11KDtiUWRq4-unsplash-scaled.jpg\")
<\/p>\n
Picture by Uriel SC<\/a>. Some rights reserved.<\/p>\n<\/div>\nOverview<\/h1>\n
\n
Preloaded Datasets in PyTorch<\/h1>\n
import torch\r\nfrom torch.utils.data import Dataset\r\nfrom torchvision import datasets\r\nimport torchvision.transforms as transforms\r\nimport numpy as np\r\nimport matplotlib.pyplot as plt\r\ntorch.manual_seed(42)<\/pre>\n
def imshow(sample_element, shape = (28, 28)):\r\n plt.imshow(sample_element[0].numpy().reshape(shape), cmap='gray')\r\n plt.title('Label = ' + str(sample_element[1]))\r\n plt.show()<\/pre>\nFashionMNIST()<\/code> from torchvision.datasets<\/code>. This function takes some arguments:<\/p>\n\n
root<\/code>: specifies the path where we are going to store our data.<\/li>\ntrain<\/code>: indicates whether it\u2019s train or test data. We\u2019ll set it to False as we don\u2019t yet need it for training.<\/li>\ndownload<\/code>: set to True<\/code>, meaning it will download the data from the internet.<\/li>\ntransform<\/code>: allows us to use any of the available transforms that we need to apply on our dataset.<\/li>\n<\/ul>\ndataset = datasets.FashionMNIST(\r\n root='.\/data',\r\n train=False,\r\n download=True,\r\n transform=transforms.ToTensor()\r\n)<\/pre>\n
classes = dataset.classes\r\nprint(classes)<\/pre>\n
['T-shirt\/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']<\/pre>\n
print(dataset.class_to_idx)<\/pre>\n
{'T-shirt\/top': 0, 'Trouser': 1, 'Pullover': 2, 'Dress': 3, 'Coat': 4, 'Sandal': 5, 'Shirt': 6, 'Sneaker': 7, 'Bag': 8, 'Ankle boot': 9}<\/pre>\nimshow(dataset[0])<\/pre>\n<\/p>\n
![\"First](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/11\/pytorchdata-1.png\")
<\/p>\nApplying Torchvision Transforms on Image Datasets<\/h1>\n
RandomCrop<\/code> the images for data augmentation.<\/p>\nprint(dir(transforms))<\/pre>\n
['AugMix', 'AutoAugment', 'AutoAugmentPolicy', 'CenterCrop', 'ColorJitter',\r\n 'Compose', 'ConvertImageDtype', 'ElasticTransform', 'FiveCrop', 'GaussianBlur',\r\n'Grayscale', 'InterpolationMode', 'Lambda', 'LinearTransformation',\r\n'Normalize', 'PILToTensor', 'Pad', 'RandAugment', 'RandomAdjustSharpness',\r\n'RandomAffine', 'RandomApply', 'RandomAutocontrast', 'RandomChoice', 'RandomCrop',\r\n'RandomEqualize', 'RandomErasing', 'RandomGrayscale', 'RandomHorizontalFlip',\r\n'RandomInvert', 'RandomOrder', 'RandomPerspective', 'RandomPosterize',\r\n'RandomResizedCrop', 'RandomRotation', 'RandomSolarize', 'RandomVerticalFlip',\r\n'Resize', 'TenCrop', 'ToPILImage', 'ToTensor', 'TrivialAugmentWide',\r\n...]<\/pre>\n
RandomCrop<\/code> transform to the Fashion-MNIST images and convert them to a tensor. We can use transform.Compose<\/code> to combine multiple transforms as we learned from the previous tutorial.<\/p>\nrandomcrop_totensor_transform = transforms.Compose([transforms.CenterCrop(16),\r\n transforms.ToTensor()])\r\ndataset = datasets.FashionMNIST(root='.\/data',\r\n train=False, download=True,\r\n transform=randomcrop_totensor_transform)\r\nprint(\"shape of the first data sample: \", dataset[0][0].shape)<\/pre>\n
shape of the first data sample: torch.Size([1, 16, 16])<\/pre>\n
imshow(dataset[0], shape=(16, 16))<\/pre>\n
![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/11\/pytorchdata-2.png\")
<\/p>\nimport torch\r\nfrom torch.utils.data import Dataset\r\nfrom torchvision import datasets\r\nimport torchvision.transforms as transforms\r\nimport numpy as np\r\nimport matplotlib.pyplot as plt\r\ntorch.manual_seed(42)\r\n\r\ndef imshow(sample_element, shape = (28, 28)):\r\n plt.imshow(sample_element[0].numpy().reshape(shape), cmap='gray')\r\n plt.title('Label = ' + str(sample_element[1]))\r\n plt.show()\r\n\r\ndataset = datasets.FashionMNIST(\r\n root='.\/data',\r\n train=False,\r\n download=True,\r\n transform=transforms.ToTensor()\r\n)\r\n\r\nclasses = dataset.classes\r\nprint(classes)\r\nprint(dataset.class_to_idx)\r\n\r\nimshow(dataset[0])\r\n\r\nrandomcrop_totensor_transform = transforms.Compose([transforms.CenterCrop(16),\r\n transforms.ToTensor()])\r\ndataset = datasets.FashionMNIST(\r\n root='.\/data',\r\n train=False,\r\n download=True,\r\n transform=randomcrop_totensor_transform)\r\n)\r\n\r\nprint(\"shape of the first data sample: \", dataset[0][0].shape)\r\nimshow(dataset[0], shape=(16, 16))<\/pre>\n<\/p>\nBuilding Custom Image Datasets<\/h1>\n
Dataset<\/code> class, here we\u2019ll build a custom image dataset class from scratch.<\/p>\n__init__<\/code> function in the class instantiates the Dataset<\/code> object. The directory where images and annotations are stored is initialized along with the transforms if we want to apply them on our dataset later. Here we assume we have some images in a directory structure like the following:<\/p>\nattface\/\r\n|-- imagedata.csv\r\n|-- s1\/\r\n| |-- 1.png\r\n| |-- 2.png\r\n| |-- 3.png\r\n| ...\r\n|-- s2\/\r\n| |-- 1.png\r\n| |-- 2.png\r\n| |-- 3.png\r\n| ...\r\n...<\/pre>\n
s1\/1.png,1\r\ns1\/2.png,1\r\ns1\/3.png,1\r\n...\r\ns12\/1.png,12\r\ns12\/2.png,12\r\ns12\/3.png,12<\/pre>\n
__len__<\/code> function in the class that returns the total number of samples in our image dataset while the __getitem__<\/code> method reads and returns a data element from the dataset at a given index.<\/p>\nimport os\r\nimport pandas as pd\r\nimport numpy as np\r\nfrom torchvision.io import read_image\r\n\r\n# creating object for our image dataset\r\nclass CustomDatasetForImages(Dataset):\r\n # defining constructor\r\n def __init__(self, annotations, directory, transform=None):\r\n # directory containing the images\r\n self.directory = directory\r\n annotations_file_dir = os.path.join(self.directory, annotations)\r\n # loading the csv with info about images\r\n self.labels = pd.read_csv(annotations_file_dir)\r\n # transform to be applied on images\r\n self.transform = transform\r\n\r\n # Number of images in dataset\r\n self.len = self.labels.shape[0]\r\n\r\n # getting the length\r\n def __len__(self):\r\n return len(self.labels)\r\n\r\n # getting the data items\r\n def __getitem__(self, idx):\r\n # defining the image path\r\n image_path = os.path.join(self.directory, self.labels.iloc[idx, 0])\r\n # reading the images\r\n image = read_image(image_path)\r\n # corresponding class labels of the images \r\n label = self.labels.iloc[idx, 1]\r\n\r\n # apply the transform if not set to None\r\n if self.transform:\r\n image = self.transform(image)\r\n \r\n # returning the image and label\r\n return image, label<\/pre>\n
directory = \"attface\"\r\nannotations = \"imagedata.csv\"\r\ncustom_dataset = CustomDatasetForImages(annotations=annotations,\r\n directory=directory)<\/pre>\n
randomcrop_totensor_transform = transforms.RandomCrop(16)\r\ndataset = CustomDatasetForImages(annotations=annotations,\r\n directory=directory,\r\n transform=randomcrop_totensor_transform)<\/pre>\n
Summary<\/h1>\n
\n