The fashion MNIST dataset is a replacement of the MNIST dataset. Like the 10 digits, it has 10 classes.
While it comes bundled with many deep learning libraries, I wanted to use it for my own project. One choice would have been to download it from here: https://github.com/zalandoresearch/fashion-mnist. Then extract the dataset and finally load it in my program.
As a programmer, I am not a big fan of doing things on my own. Further, this method is not scalable. If I want a different dataset in the future I will have to perform the same steps manually and create another loader for that data. So, I decided to write my own data loader.
I created two scripts: data_downloader.py and fashion_mnist.py. The idea is that data_downloader will be common utility for all the loaders to download their respective datasets. fashion_mnist contains specific code to load the data and the web urls to pass to the data_downloader to fetch the data.
The folder structure is as follows:
The datasets folder is separate to have a single directory for loader scripts of different datasets and keep them separate from the common utilities.
Both the scripts are given below:
""" data_downloader.py Common functions for downloading dataset """ import os import requests def get_file(fname, origin, cache_subdir): """Downloads the file from URL is it is not yet in cache Arguments fname: name of the file origin: Remote URL of the file Return: Path to downloaded file """ cache_dir = os.path.join(os.path.expanduser('~'), '.datasets') # create temporary download location datadir = os.path.join(cache_dir, cache_subdir) if not os.path.exists(datadir): os.makedirs(datadir) fpath = os.path.join(datadir, fname) if not os.path.exists(fpath): print(fname, "does not exist") print("Downloading data from", origin) r = requests.get(origin, stream = True) with open(fpath, 'wb') as file: for chunk in r.iter_content(chunk_size = 1024): if chunk: file.write(chunk) print("Finished downloading ", fname) return fpath
The datasets are downloaded into a directory named .datasets in the home folder of the user. Linux/OSX systems consider files and directories that start with a dot as hidden. Keeping the downloaded datasets folder reduces visible clutter in the home directory. The script also creates another folder inside the .datasets folder with the name of the dataset for storing the downloaded data. I may add file integrity check in the future but this works for now.
""" fashion_mnist.py Fashion MNIST dataset loader """ import gzip import os import numpy as np from ..utils.data_downloader import get_file def load_data(): """ Loads the Fashion MNIST dataset return: x_train, y_train, x_test, y_test """ dirname = os.path.join('datasets', 'fashion_mnist') base = 'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/' files = ['train-labels-idx1-ubyte.gz', 'train-images-idx3-ubyte.gz', 't10k-labels-idx1-ubyte.gz', 't10k-images-idx3-ubyte.gz'] paths =  for fname in files: paths.append(get_file(fname, origin=base + fname, cache_subdir=dirname)) with gzip.open(paths, 'rb') as lbpath: y_train = np.frombuffer(lbpath.read(), np.uint8, offset=8) with gzip.open(paths, 'rb') as imgpath: x_train = np.frombuffer(imgpath.read(), np.uint8, offset=16).reshape(len(y_train), 28, 28) with gzip.open(paths, 'rb') as lbpath: y_test = np.frombuffer(lbpath.read(), np.uint8, offset=8) with gzip.open(paths, 'rb') as imgpath: x_test = np.frombuffer(imgpath.read(), np.uint8, offset=16).reshape(len(y_test), 28, 28) y_train = [vectorize_data(i) for i in y_train] y_test = [vectorize_data(i) for i in y_test] return x_train, y_train, x_test, y_test def vectorize_data(y): e = np.zeros((10,1)) e[y] = 1.0 return e
The original data contains each image represented in a 2D array and the corresponding labels as a single digit. I am not flattening the image here as 2D input is convenient when using Convolutional Neural Networks. The label is vectorized because the output layer of any neural network will be of 10 units each representing a single digit. It is easier to vectorize here and convert it to pure labels later if required.
And finally you can import the dataset as:
from datasets import fashion_mnist x_train, y_train, x_test, y_test = fashion_mnist.load_data()
Thanks for reading 😎