{"id":1974,"date":"2019-04-04T18:00:57","date_gmt":"2019-04-04T18:00:57","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2019\/04\/04\/a-gentle-introduction-to-channels-first-and-channels-last-image-formats-for-deep-learning\/"},"modified":"2019-04-04T18:00:57","modified_gmt":"2019-04-04T18:00:57","slug":"a-gentle-introduction-to-channels-first-and-channels-last-image-formats-for-deep-learning","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2019\/04\/04\/a-gentle-introduction-to-channels-first-and-channels-last-image-formats-for-deep-learning\/","title":{"rendered":"A Gentle Introduction to Channels First and Channels Last Image Formats for Deep Learning"},"content":{"rendered":"<p>Author: Jason Brownlee<\/p>\n<div>\n<p>Color images have height, width, and color channel dimensions.<\/p>\n<p>When represented as three-dimensional arrays, the channel dimension for the image data is last by default, but may be moved to be the first dimension, often for performance-tuning reasons.<\/p>\n<p>The use of these two \u201c<em>channel ordering formats<\/em>\u201d and preparing data to meet a specific preferred channel ordering can be confusing to beginners.<\/p>\n<p>In this tutorial, you will discover channel ordering formats, how to prepare and manipulate image data to meet formats, and how to configure the Keras deep learning library for different channel orderings.<\/p>\n<p>After completing this tutorial, you will know:<\/p>\n<ul>\n<li>The three-dimensional array structure of images and the channels first and channels last array formats.<\/li>\n<li>How to add a channels dimension and how to convert images between the channel formats.<\/li>\n<li>How the Keras deep learning library manages a preferred channel ordering and how to change and query this preference.<\/li>\n<\/ul>\n<p>Let\u2019s get started.<\/p>\n<h2>Tutorial Overview<\/h2>\n<p>This tutorial is divided into three parts; they are:<\/p>\n<ol>\n<li>Images as 3D Arrays<\/li>\n<li>Manipulating Image Channels<\/li>\n<li>Keras Channel Ordering<\/li>\n<\/ol>\n<h2>Images as 3D Arrays<\/h2>\n<p>An image can be stored as a three-dimensional array in memory.<\/p>\n<p>Typically, the image format has one dimension for rows (height), one for columns (width) and one for channels.<\/p>\n<p>If the image is black and white (grayscale), the channels dimension may not be explicitly present, e.g. there is one unsigned integer pixel value for each (row, column) coordinate in the image.<\/p>\n<p>Colored images typically have three channels, for the pixel value at the (row, column) coordinate for the red, green, and blue components.<\/p>\n<p>Deep learning neural networks require that image data be provided as three-dimensional arrays.<\/p>\n<p>This applies even if your image is grayscale. In this case, the additional dimension for the single color channel must be added.<\/p>\n<p>There are two ways to represent the image data as a three dimensional array. The first involves having the channels as the last or third dimension in the array. This is called \u201c<em>channels last<\/em>\u201c. The second involves having the channels as the first dimension in the array, called \u201c<em>channels first<\/em>\u201c.<\/p>\n<ul>\n<li><strong>Channels Last<\/strong>. Image data is represented in a three-dimensional array where the last channel represents the color channels, e.g. <em>[rows][cols][channels]<\/em>.<\/li>\n<li><strong>Channels First<\/strong>. Image data is represented in a three-dimensional array where the first channel represents the color channels, e.g. <em>[channels][rows][cols]<\/em>.<\/li>\n<\/ul>\n<p>Some image processing and deep learning libraries prefer channels first ordering, and some prefer channels last. As such, it is important to be familiar with the two approaches to representing images.<\/p>\n<div class=\"woo-sc-hr\"><\/div>\n<p><center><\/p>\n<h3>Want Results with Deep Learning for Computer Vision?<\/h3>\n<p>Take my free 7-day email crash course now (with sample code).<\/p>\n<p>Click to sign-up and also get a free PDF Ebook version of the course.<\/p>\n<p><a href=\"https:\/\/machinelearningmastery.lpages.co\/leadbox\/1458ca1e0972a2%3A164f8be4f346dc\/4715926590455808\/\" target=\"_blank\" style=\"background: rgb(255, 206, 10); color: rgb(255, 255, 255); text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-weight: bold; font-size: 16px; line-height: 20px; padding: 10px; display: inline-block; max-width: 300px; border-radius: 5px; text-shadow: rgba(0, 0, 0, 0.25) 0px -1px 1px; box-shadow: rgba(255, 255, 255, 0.5) 0px 1px 3px inset, rgba(0, 0, 0, 0.5) 0px 1px 3px;\" rel=\"noopener noreferrer\">Download Your FREE Mini-Course<\/a><script data-leadbox=\"1458ca1e0972a2:164f8be4f346dc\" data-url=\"https:\/\/machinelearningmastery.lpages.co\/leadbox\/1458ca1e0972a2%3A164f8be4f346dc\/4715926590455808\/\" data-config=\"%7B%7D\" type=\"text\/javascript\" src=\"https:\/\/machinelearningmastery.lpages.co\/leadbox-1553357564.js\"><\/script><\/p>\n<p><\/center><\/p>\n<div class=\"woo-sc-hr\"><\/div>\n<h2>Manipulating Image Channels<\/h2>\n<p>You may need to change or manipulate the image channels or channel ordering.<\/p>\n<p>This can be achieved easily using the NumPy python library.<\/p>\n<p>Let\u2019s look at some examples.<\/p>\n<p>In this tutorial, we will use a photograph taken by Larry Koester, some rights reserved, of the <a href=\"https:\/\/www.flickr.com\/photos\/larrywkoester\/45435718605\/\">Phillip Island Penguin Parade<\/a>.<\/p>\n<div id=\"attachment_7401\" style=\"width: 650px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-7401\" class=\"size-full wp-image-7401\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/01\/penguin_arade.jpg\" alt=\"Phillip Island Penguin Parade\" width=\"640\" height=\"424\" srcset=\"http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2019\/01\/penguin_arade.jpg 640w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2019\/01\/penguin_arade-300x199.jpg 300w\" sizes=\"(max-width: 640px) 100vw, 640px\"><\/p>\n<p id=\"caption-attachment-7401\" class=\"wp-caption-text\">Phillip Island Penguin Parade<br \/>Photo by <a href=\"https:\/\/www.flickr.com\/photos\/larrywkoester\/45435718605\/\">Larry Koester<\/a>, some rights reserved.<\/p>\n<\/div>\n<p>Download the image and place it in your current working directory with the filename \u201c<em>penguin_parade.jpg<\/em>\u201c.<\/p>\n<ul>\n<li><a href=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/01\/penguin_arade.jpg\">Download Photo (penguin_parade.jpg)<\/a><\/li>\n<\/ul>\n<p>The code examples in this tutorials assume that the <a href=\"https:\/\/pillow.readthedocs.io\/en\/stable\/\">Pillow library<\/a> is installed.<\/p>\n<h3>How to Add a Channel to a Grayscale Image<\/h3>\n<p>Grayscale images are loaded as a two-dimensional array.<\/p>\n<p>Before they can be used for modeling, you may have to add an explicit channel dimension to the image. This does not add new data; instead, it changes the array data structure to have an additional third axis with one dimension to hold the grayscale pixel values.<\/p>\n<p>For example, a grayscale image with the dimensions [rows][cols] can be changed to [rows][cols][channels] or [channels][rows][cols] where the new [channels] axis has one dimension.<\/p>\n<p>This can be achieved using the <a href=\"https:\/\/docs.scipy.org\/doc\/numpy\/reference\/generated\/numpy.expand_dims.html\">expand_dims() NumPy function<\/a>. The \u201c<em>axis<\/em>\u201d argument allows you to specify where the new dimension will be added to the first, e.g. first for channels first or last for channels last.<\/p>\n<p>The example below loads the Penguin Parade photograph using the Pillow library as a grayscale image and demonstrates how to add a channel dimension.<\/p>\n<pre class=\"crayon-plain-tag\"># example of expanding dimensions\r\nfrom numpy import expand_dims\r\nfrom numpy import asarray\r\nfrom PIL import Image\r\n# load the image\r\nimg = Image.open('penguin_arade.jpg')\r\n# convert the image to grayscale\r\nimg = img.convert(mode='L')\r\n# convert to numpy array\r\ndata = asarray(img)\r\nprint(data.shape)\r\n# add channels first\r\ndata_first = expand_dims(data, axis=0)\r\nprint(data_first.shape)\r\n# add channels first\r\ndata_last = expand_dims(data, axis=2)\r\nprint(data_last.shape)<\/pre>\n<p>Running the example first loads the photograph using the Pillow library, then converts it to a grayscale image.<\/p>\n<p>The image object is converted to a NumPy array and we confirm the shape of the array is two dimensional, specifically (424, 640).<\/p>\n<p>The <em>expand_dims()<\/em> function is then used to add a channel via <em>axis=0<\/em> to the front of the array and the change is confirmed with the shape (1, 424, 640). The same function is then used to add a channel to the end or third dimension of the array with <em>axis=2<\/em> and the change is confirmed with the shape (424, 640, 1).<\/p>\n<pre class=\"crayon-plain-tag\">(424, 640)\r\n(1, 424, 640)\r\n(424, 640, 1)<\/pre>\n<p>Another popular alternative to expanding the dimensions of an array is to use the <a href=\"https:\/\/docs.scipy.org\/doc\/numpy\/reference\/generated\/numpy.reshape.html\">reshape() NumPy function<\/a> and specify a tuple with the new shape; for example:<\/p>\n<pre class=\"crayon-plain-tag\">data = data.reshape((424, 640, 1))<\/pre>\n<\/p>\n<h3>How to Change Image Channel Ordering<\/h3>\n<p>After a color image is loaded as a three-dimensional array, the channel ordering can be changed.<\/p>\n<p>This can be achieved using the <a href=\"https:\/\/docs.scipy.org\/doc\/numpy\/reference\/generated\/numpy.moveaxis.html\">moveaxis() NumPy function<\/a>. It allows you to specify the index of the source axis and the destination axis.<\/p>\n<p>This function can be used to change an array in channel last format such, as<em> [rows][cols][channels]<\/em> to channels first format, such as <em>[channels][rows][cols]<\/em>, or the reverse.<\/p>\n<p>The example below loads the Penguin Parade photograph in channel last format and uses the <em>moveaxis()<\/em> function change it to channels first format.<\/p>\n<pre class=\"crayon-plain-tag\"># change image from channels last to channels first format\r\nfrom numpy import moveaxis\r\nfrom numpy import asarray\r\nfrom PIL import Image\r\n# load the color image\r\nimg = Image.open('penguin_arade.jpg')\r\n# convert to numpy array\r\ndata = asarray(img)\r\nprint(data.shape)\r\n# change channels last to channels first format\r\ndata = moveaxis(data, 2, 0)\r\nprint(data.shape)\r\n# change channels first to channels last format\r\ndata = moveaxis(data, 0, 2)\r\nprint(data.shape)<\/pre>\n<p>Running the example first loads the photograph using the Pillow library and converts it to a NumPy array confirming that the image was loaded in channels last format with the shape (424, 640, 3).<\/p>\n<p>The <em>moveaxis()<\/em>\u00a0function is then used to move the channels axis from position 2 to position 0 and the result is confirmed showing channels first format (3, 424, 640). This is then reversed, moving the channels in position 0 to position 2 again.<\/p>\n<pre class=\"crayon-plain-tag\">(424, 640, 3)\r\n(3, 424, 640)\r\n(424, 640, 3)<\/pre>\n<\/p>\n<h2>Keras Channel Ordering<\/h2>\n<p>The Keras deep learning library is agnostic to how you wish to represent images in either channel first or last format, but the preference must be specified and adhered to when using the library.<\/p>\n<p>Keras wraps a number of mathematical libraries, and each has a preferred channel ordering. The three main libraries that Keras may wrap and their preferred channel ordering are listed below:<\/p>\n<ul>\n<li><strong>TensorFlow<\/strong>: Channels last order.<\/li>\n<li><strong>Theano<\/strong>: Channels first order.<\/li>\n<li><strong>CNTK<\/strong>: Channels last order.<\/li>\n<\/ul>\n<p>By default, Keras is configured to use TensorFlow and the channel ordering is also by default channels last. You can use either channel ordering with any library and the Keras library.<\/p>\n<p>Some libraries claim that the preferred channel ordering can result in a large difference in performance. For example, use of the MXNet mathematical library as the backend for Keras recommends using the channels first ordering for better performance.<\/p>\n<blockquote>\n<p>We strongly recommend changing the image_data_format to channels_first. MXNet is significantly faster on channels_first data.<\/p>\n<\/blockquote>\n<p>\u2014 <a href=\"https:\/\/github.com\/awslabs\/keras-apache-mxnet\/wiki\/Performance-Tuning---Keras-with-MXNet-Backend\">Performance Tuning Keras with MXNet Backend, Apache MXNet<\/a><\/p>\n<h3>Default Channel Ordering<\/h3>\n<p>The library and preferred channel ordering are listed in the Keras configuration file, stored in your home directory under <em>~\/.keras\/keras.json<\/em>.<\/p>\n<p>The preferred channel ordering is stored in the \u201c<em>image_data_format<\/em>\u201d configuration setting and can be set as either \u201c<em>channels_last<\/em>\u201d or \u201c<em>channels_first<\/em>\u201c.<\/p>\n<p>For example, below is the contents of a <em>keras.json<\/em> configuration file. In it, you can see that the system is configured to use tensorflow and <em>channels_last<\/em> order.<\/p>\n<pre class=\"crayon-plain-tag\">{\r\n    \"image_data_format\": \"channels_last\",\r\n    \"backend\": \"tensorflow\",\r\n    \"epsilon\": 1e-07,\r\n    \"floatx\": \"float32\"\r\n}<\/pre>\n<p>Based on your preferred channel ordering, you will have to prepare your image data to match the preferred ordering.<\/p>\n<p>Specifically, this will include tasks such as:<\/p>\n<ul>\n<li>Resizing or expanding the dimensions of any training, validation, and test data to meet the expectation.<\/li>\n<li>Specifying the expected input shape of samples when defining models (e.g. <em>input_shape=(28, 28, 1)<\/em>).<\/li>\n<\/ul>\n<h3>Model-Specific Channel Ordering<\/h3>\n<p>In addition, those neural network layers that are designed to work with images, such as Conv2D, also provide an argument called \u201c<em>data_format<\/em>\u201d that allows you to specify the channel ordering. For example:<\/p>\n<pre class=\"crayon-plain-tag\">...\r\nmodel.add(Conv2D(..., data_format='channels_first'))<\/pre>\n<p>By default, this will use the preferred ordering specified in the \u201c<em>image_data_format<\/em>\u201d value of the Keras configuration file. Nevertheless, you can change the channel order for a given model, and in turn, the datasets and input shape would also have to be changed to use the new channel ordering for the model.<\/p>\n<p>This can be useful when loading a model used for transfer learning that has a channel ordering different to your preferred channel ordering.<\/p>\n<h3>Query Channel Ordering<\/h3>\n<p>You can confirm your current preferred channel ordering by printing the result of the <em>image_data_format()<\/em> function. The example below demonstrates.<\/p>\n<pre class=\"crayon-plain-tag\"># show preferred channel order\r\nfrom keras import backend\r\nprint(backend.image_data_format())<\/pre>\n<p>Running the example prints your preferred channel ordering as configured in your Keras configuration file. In this case, the channels last format is used.<\/p>\n<pre class=\"crayon-plain-tag\">channels_last<\/pre>\n<p>Accessing this property can be helpful if you want to automatically construct models or prepare data differently depending on the systems preferred channel ordering; for example:<\/p>\n<pre class=\"crayon-plain-tag\">if backend.image_data_format() == 'channels_last':\r\n\t...\r\nelse:\r\n\t...<\/pre>\n<\/p>\n<h3>Force Channel Ordering<\/h3>\n<p>Finally, the channel ordering can be forced for a specific program.<\/p>\n<p>This can be achieved by calling the <em>set_image_dim_ordering()<\/em> function on the Keras backend to either \u2018<em>th<\/em>\u2018 (theano) for channel-first ordering, or \u2018<em>tf<\/em>\u2018 (tensorflow) for channel-last ordering.<\/p>\n<p>This can be useful if you want a program or model to operate consistently regardless of Keras default channel ordering configuration.<\/p>\n<pre class=\"crayon-plain-tag\"># force a channel ordering\r\nfrom keras import backend\r\n# force channels-first ordering\r\nbackend.set_image_dim_ordering('th')\r\nprint(backend.image_data_format())\r\n# force channels-last ordering\r\nbackend.set_image_dim_ordering('tf')\r\nprint(backend.image_data_format())<\/pre>\n<p>Running the example first forces channels-first ordering, then channels-last ordering, confirming each configuration by printing the channel ordering mode after the change.<\/p>\n<pre class=\"crayon-plain-tag\">channels_first\r\nchannels_last<\/pre>\n<\/p>\n<h2>Further Reading<\/h2>\n<p>This section provides more resources on the topic if you are looking to go deeper.<\/p>\n<ul>\n<li><a href=\"https:\/\/pillow.readthedocs.io\/en\/stable\/\">Pillow Python library<\/a><\/li>\n<li><a href=\"https:\/\/docs.scipy.org\/doc\/numpy\/reference\/generated\/numpy.expand_dims.html\">numpy.expand_dims API<\/a><\/li>\n<li><a href=\"https:\/\/docs.scipy.org\/doc\/numpy\/reference\/generated\/numpy.reshape.html\">numpy.reshape API<\/a><\/li>\n<li><a href=\"https:\/\/docs.scipy.org\/doc\/numpy\/reference\/generated\/numpy.moveaxis.html\">numpy.moveaxis API<\/a><\/li>\n<li><a href=\"https:\/\/keras.io\/backend\/\">Keras Backend API<\/a><\/li>\n<li><a href=\"https:\/\/keras.io\/layers\/convolutional\/\">Keras Convolutional Layers API<\/a><\/li>\n<\/ul>\n<h2>Summary<\/h2>\n<p>In this tutorial, you discovered channel ordering formats, how to prepare and manipulate image data to meet formats, and how to configure the Keras deep learning library for different channel orderings.<\/p>\n<p>Specifically, you learned:<\/p>\n<ul>\n<li>The three-dimensional array structure of images and the channels first and channels last array formats.<\/li>\n<li>How to add a channels dimension and how to convert images between the channel formats.<\/li>\n<li>How the Keras deep learning library manages a preferred channel ordering and how to change and query this preference.<\/li>\n<\/ul>\n<p>Do you have any questions?<br \/>\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n<p>The post <a rel=\"nofollow\" href=\"https:\/\/machinelearningmastery.com\/a-gentle-introduction-to-channels-first-and-channels-last-image-formats-for-deep-learning\/\">A Gentle Introduction to Channels First and Channels Last Image Formats for Deep Learning<\/a> appeared first on <a rel=\"nofollow\" href=\"https:\/\/machinelearningmastery.com\/\">Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n<p><a href=\"https:\/\/machinelearningmastery.com\/a-gentle-introduction-to-channels-first-and-channels-last-image-formats-for-deep-learning\/\">Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Author: Jason Brownlee Color images have height, width, and color channel dimensions. When represented as three-dimensional arrays, the channel dimension for the image data is [&hellip;] <span class=\"read-more-link\"><a class=\"read-more\" href=\"https:\/\/www.aiproblog.com\/index.php\/2019\/04\/04\/a-gentle-introduction-to-channels-first-and-channels-last-image-formats-for-deep-learning\/\">Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":1975,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[24],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/1974"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=1974"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/1974\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/1975"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=1974"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=1974"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=1974"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}