Python is an awesome programming language! It is one of the most popular languages for developing AI and machine learning applications. With a very easy to learn syntax, Python has some special features that distinguish it from other languages. In this tutorial, we\u2019ll talk about some special attributes of the Python programming language.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Constructs for list and dictionary comprehension<\/li>\n
- How to use zip and enumerate functions<\/li>\n
- What are function contexts and decorators<\/li>\n
- What is the purpose of generators in Python<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/12\/m-mani-1024x768.png\")
<\/a><\/p>\nPython Special Features
Photo by M Mani, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into 4 parts; they are:<\/p>\n
- \n
- List and dictionary comprehension<\/li>\n
- Zip and enumerate functions<\/li>\n
- Function contexts and decorators<\/li>\n
- Generators in Python with example of Keras generator<\/li>\n<\/ol>\n
Import Section<\/h2>\n
The libraries used in this tutorial are imported in the code below.<\/p>\n
from tensorflow import keras\r\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\r\nimport numpy as np\r\nimport matplotlib.pyplot as plt\r\nimport math<\/pre>\n<\/p>\n
List Comprehension<\/h2>\n
List comprehension provides a short, simple syntax for creating new lists from existing ones. For example, suppose we require a new list, where each new item is the old item multiplied by 3. One method is to use a
for<\/code> loop as shown below:<\/p>\noriginal_list = [1, 2, 3, 4]\r\ntimes3_list = []\r\n\r\nfor i in original_list:\r\n times3_list.append(i*3)\r\nprint(times3_list)<\/pre>\n<\/p>\n
[3, 6, 9, 12]<\/pre>\n
The shorter method using list comprehension requires only a single line of code:<\/p>\n
time3_list_awesome_method = [i*3 for i in original_list]\r\nprint(time3_list_awesome_method)<\/pre>\n<\/p>\n
[3, 6, 9, 12]<\/pre>\n
You can even create a new list based on a special criterion. For example, if we want only even numbers added to the new list.<\/p>\n
even_list_awesome_method = [i for i in original_list if i%2==0]\r\nprint(even_list_awesome_method)<\/pre>\n<\/p>\n
[2, 4]<\/pre>\n
It is also possible to have an
else<\/code> associated with the above. For example, we can leave all even numbers intact and replace the odd numbers by zero:<\/p>\nnew_list_awesome_method = [i if i%2==0 else 0 for i in original_list]\r\nprint(new_list_awesome_method)<\/pre>\n<\/p>\n
[0, 2, 0, 4]<\/pre>\n<\/p>\n
\n\n\nList comprehension can also used to replace nested loops. For example the following:<\/p>\n
colors = [\"red\", \"green\", \"blue\"]\r\nanimals = [\"cat\", \"dog\", \"bird\"]\r\nnewlist = []\r\nfor c in colors:\r\n for a in animals:\r\n newlist.append(c + \" \" + a)\r\nprint(newlist)<\/pre>\n<\/p>\n
['red cat', 'red dog', 'red bird', 'green cat', 'green dog', 'green bird', 'blue cat', 'blue dog', 'blue bird']<\/pre>\n
can be done as follows, with two \u201cfor\u201d inside the list comprehension:<\/p>\n
colors = [\"red\", \"green\", \"blue\"]\r\nanimals = [\"cat\", \"dog\", \"bird\"]\r\n\r\nnewlist = [c+\" \"+a for c in colors for a in animals]\r\nprint(newlist)<\/pre>\n<\/p>\n
Syntax<\/h3>\n
Syntax for list comprehension is given by:<\/p>\n
newlist = [expression for item in iterable if condition == True] <\/em><\/p>\n
Or<\/em><\/p>\n
newList = [expression if condition == True else expression for item in iterable]<\/em><\/p>\n<\/div>\n<\/div>\n<\/div>\n
\nDictionary Comprehension<\/h2>\n
\n\nDictionary comprehension is similar to list comprehension, except now we have (key, value) pairs. Here is an example; we\u2019ll modify each value of the dictionary by concatenating the string \u2018number \u2018 to each value:<\/p>\n
original_dict = {1: 'one', 2: 'two', 3: 'three', 4: 'four'}\r\nnew_dict = {key:'number ' + value for (key, value) in original_dict.items()}\r\nprint(new_dict)<\/pre>\n<\/p>\n{1: 'number one', 2: 'number two', 3: 'number three', 4: 'number four'}<\/pre>\nAgain, conditionals are also possible. We can choose to add (key, value) pairs based on a criterion in the new dictionary.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
#Only add keys which are greater than 2\r\nnew_dict_high_keys = {key:'number ' + value for (key, value) in original_dict.items() if key>2}\r\nprint(new_dict_high_keys)\r\n\r\n# Only change values with key>2\r\nnew_dict_2 = {key:('number ' + value if key>2 else value) for (key, value) in original_dict.items() }\r\nprint(new_dict_2)<\/pre>\n<\/p>\n{3: 'number three', 4: 'number four'}\r\n{1: 'one', 2: 'two', 3: 'number three', 4: 'number four'}<\/pre>\n<\/p>\nEnumerators and Zip in Python<\/h2>\n
In Python an iterable is defined as any data structure that can return all its items, one at a time. This way you can use a
for<\/code> loop for further processing of all items one by one. Python has two additional constructs that makefor<\/code> loops easier to use, i.e.,\u00a0enumerate()<\/code> andzip()<\/code>.<\/p>\nEnumerate<\/h3>\n
In traditional programming languages, you need a loop variable to iterate through different values of a container. In Python this is simplified by giving you access to a loop variable along with one value of the iterable object. The
enumerate(x)<\/code>\u00a0function returns two iterables. One iterable varies from 0 to len(x)-1. The other is an iterable with value equal to items of x. An example is shown below:<\/p>\nname = ['Triangle', 'Square', 'Hexagon', 'Pentagon']\r\n\r\n# enumerate returns two iterables\r\nfor i, n in enumerate(name):\r\n print(i, 'name: ', n)<\/pre>\n<\/p>\n
0 name: Triangle\r\n1 name: Square\r\n2 name: Hexagon\r\n3 name: Pentagon<\/pre>\n
By default, enumerate starts at 0 but we can start at some other number if we specified it. This is useful in some situation, for example:<\/p>\n
data = [1,4,1,5,9,2,6,5,3,5,8,9,7,9,3]\r\nfor n, digit in enumerate(data[5:], 6):\r\n print(\"The %d-th digit is %d\" % (n, digit))<\/pre>\n<\/p>\n
The 6-th digit is 2\r\nThe 7-th digit is 6\r\nThe 8-th digit is 5\r\nThe 9-th digit is 3\r\nThe 10-th digit is 5\r\nThe 11-th digit is 8\r\nThe 12-th digit is 9\r\nThe 13-th digit is 7\r\nThe 14-th digit is 9\r\nThe 15-th digit is 3<\/pre>\n<\/p>\n
Zip<\/h3>\n
Zip allows you to create an iterable object of tuples. Zip takes as argument multiple containers $(m_1, m_2, ldots, m_n)$, and creates the i-th tuple by pairing one item from each container. The i-th tuple is then $(m_{1i}, m_{2i}, ldots, m_{ni})$. If the passed objects have different lengths, then the total number of tuples formed have a length equal to the minimum length of passed objects.<\/p>\n
Below are examples of using both
zip()<\/code> andenumerate()<\/code>.<\/p>\nsides = [3, 4, 6, 5]\r\ncolors = ['red', 'green', 'yellow', 'blue']\r\nshapes = zip(name, sides, colors)\r\n\r\n# Tuples are created from one item from each list\r\nprint(set(shapes))\r\n\r\n# Easy to use enumerate and zip together for iterating through multiple lists in one go\r\nfor i, (n, s, c) in enumerate(zip(name, sides, colors)):\r\n print(i, 'Shape- ', n, '; Sides ', s)<\/pre>\n<\/p>\n
{('Triangle', 3, 'red'), ('Square', 4, 'green'), ('Hexagon', 6, 'yellow'), ('Pentagon', 5, 'blue')}\r\n0 Shape- Triangle ; Sides 3\r\n1 Shape- Square ; Sides 4\r\n2 Shape- Hexagon ; Sides 6\r\n3 Shape- Pentagon ; Sides 5<\/pre>\n<\/p>\nFunction Context<\/h2>\n
Python allows nested functions, where you can define an inner function within an outer function. There are some awesome features related to nested functions in Python.<\/p>\n
- \n
- The outer function can return a handle to the inner function<\/li>\n
- The inner function retains all its environment and variables local to it and in its enclosing function even if the outer function ends its execution.<\/li>\n<\/ul>\n
An example is given below with explanation in comments.<\/p>\n
def circle(r):\r\n area = 0\r\n def area_obj():\r\n nonlocal area\r\n area = math.pi * r * r\r\n print(\"area_obj\")\r\n return area_obj \r\n\r\ndef circle(r):\r\n area_val = math.pi * r * r\r\n def area():\r\n print(area_val)\r\n return area \r\n\r\n# returns area_obj(). The value of r passed is retained\r\ncircle_1 = circle(1)\r\ncircle_2 = circle(2)\r\n\r\n# Calling area_obj() with radius = 1\r\ncircle_1()\r\n# Calling area_obj() with radius = 2\r\ncircle_2()<\/pre>\n<\/p>\n
3.141592653589793\r\n12.566370614359172<\/pre>\n<\/p>\n
Decorators in Python<\/h2>\n
Decorators are a powerful feature of Python. You can use decorators to customize the working of a class or a function. Think of them as a function applied to another function. Use the function name with
@<\/code> symbol to define the decorator function on the decorated function. The decorator takes a function as argument, giving a lot of flexibility.<\/p>\nConsider the following function
square_decorator()<\/code> that takes a function as an argument, and also returns a function.<\/p>\n- \n
- The inner nested function
square_it()<\/code>takes an argumentarg.<\/code>\n<\/li>\n- The
square_it()<\/code>function applies the function toarg<\/code> and squares the result.<\/li>\n- We can pass a function such as
sin<\/code> tosquare_decorator()<\/code>, which in turn would return $sin^2(x)$<\/span>.<\/li>\n- You can also write your own customized function and use the\u00a0
square_decorator()<\/code> function on it using the special @symbol as shown below. The function\u00a0plus_one(x)<\/code> returnsx+1<\/code>. This function is decorated by thesquare_decorator()<\/code>\u00a0and hence, we get $(x+1)^2$.<\/li>\n<\/ul>\ndef square_decorator(function):\r\n def square_it(arg):\r\n x = function(arg)\r\n return x*x\r\n return square_it\r\n\r\nsize_sq = square_decorator(len)\r\nprint(size_sq([1,2,3]))\r\n\r\nsin_sq = square_decorator(math.sin)\r\nprint(sin_sq(math.pi\/4))\r\n\r\n@square_decorator\r\ndef plus_one(a):\r\n return a+1\r\n\r\na = plus_one(3)\r\nprint(a)<\/pre>\n<\/p>\n
9\r\n0.4999999999999999\r\n16<\/pre>\n<\/p>\n
Generators in Python<\/h2>\n
Generators in Python allow you to generate sequences. Instead of writing a
return<\/code> statement, a generator returns multiple values via multipleyield<\/code>\u00a0statements. The first call to the function returns the first value from yield. The second call returns the second value from yield and so on.<\/p>\nThe generator function can be invoked via\u00a0
next().<\/code>Every timenext()<\/code>\u00a0is called the nextyield<\/code> value is returned. An example of generating the Fibonacci sequence up till a given numberx<\/code> is shown below.<\/p>\ndef get_fibonacci(x):\r\n x0 = 0\r\n x1 = 1\r\n for i in range(x):\r\n yield x0\r\n temp = x0 + x1\r\n x0 = x1\r\n x1 = temp\r\n\r\n\r\nf = get_fibonacci(6)\r\nfor i in range(6):\r\n print(next(f))<\/pre>\n<\/p>\n
0\r\n1\r\n1\r\n2\r\n3\r\n5<\/pre>\n<\/p>\n
Example of Data Generator in Keras<\/h3>\n
One use of generator is the data generator in Keras. The reason it is useful is that we do not want to keep all data in memory but want to create it on the fly when the training loop needs it. Remember in Keras, a neural network model is trained in batches, so a generator are to emit batches of data. The function below is from our previous post \u201cUsing CNN for financial time series prediction<\/a>\u201c:<\/p>\n
def datagen(data, seq_len, batch_size, targetcol, kind):\r\n \"As a generator to produce samples for Keras model\"\r\n batch = []\r\n while True:\r\n # Pick one dataframe from the pool\r\n key = random.choice(list(data.keys()))\r\n df = data[key]\r\n input_cols = [c for c in df.columns if c != targetcol]\r\n index = df.index[df.index < TRAIN_TEST_CUTOFF]\r\n split = int(len(index) * TRAIN_VALID_RATIO)\r\n if kind == 'train':\r\n index = index[:split] # range for the training set\r\n elif kind == 'valid':\r\n index = index[split:] # range for the validation set\r\n # Pick one position, then clip a sequence length\r\n while True:\r\n t = random.choice(index) # pick one time step\r\n n = (df.index == t).argmax() # find its position in the dataframe\r\n if n-seq_len+1 < 0:\r\n continue # can't get enough data for one sequence length\r\n frame = df.iloc[n-seq_len+1:n+1]\r\n batch.append([frame[input_cols].values, df.loc[t, targetcol]])\r\n break\r\n # if we get enough for a batch, dispatch\r\n if len(batch) == batch_size:\r\n X, y = zip(*batch)\r\n X, y = np.expand_dims(np.array(X), 3), np.array(y)\r\n yield X, y\r\n batch = []<\/pre>\n
The function above is to pick a random row of a pandas dataframe as a starting point and clip next several rows as one time interval sample. This process is repeated several times to collect many time intervals into one batch. When we collected enough number of interval samples, at the second last line in the above function, the batch is dispatched using the
yield<\/code> command. As you may already noticed, generator functions do not have return statement. In this example, the function even will run forever. This is useful and necessary because it allows our Keras training process to run as many epoch as we want.<\/p>\nIf we do not use generator, we will need to convert the dataframe into all possible time intervals and keep them in the memory for the training loop. This will be a lot of repeating data (because the time intervals are overlapping) and takes up a lot of memory.<\/p>\n
Because it is useful, Keras has some generator function predefined in the library. Below is an example of
ImageDataGenerator()<\/code>. We have loaded thecifar10<\/code> dataset of 32\u00d732 images inx_train<\/code>. The data is connected to the generator viaflow()<\/code> method. Thenext()<\/code> function returns the next batch of data. In the example below, there are 4 calls tonext()<\/code>. In each case 8 images are returned as the batch size is 8.<\/p>\nBelow is the entire code that also displays all images after every call to
next()<\/code>.<\/p>\n(x_train, y_train), _ = keras.datasets.cifar10.load_data()\r\ndatagen = ImageDataGenerator()\r\ndata_iterator = datagen.flow(x_train, y_train, batch_size=8)\r\n\r\nfig,ax = plt.subplots(nrows=4, ncols=8,figsize=(18,6),subplot_kw=dict(xticks=[], yticks=[]))\r\n\r\nfor i in range(4):\r\n # The next() function will load 8 images from CIFAR\r\n X, Y = data_iterator.next()\r\n for j, img in enumerate(X):\r\n ax[i, j].imshow(img.astype('int'))<\/pre>\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/12\/Untitled-1.png\")
<\/a><\/p>\nFurther Reading<\/h2>\n
This section provides more resources on the topic if you are looking to go deeper.<\/p>\n
Python Documentation<\/h3>\n
- \n
- Python Documentation at python.org<\/a><\/li>\n<\/ul>\n
Books<\/h3>\n
- \n
- \nThink Python: How to Think Like a Computer Scientist<\/a> by Allen B. Downey<\/li>\n
- \nProgramming in Python 3: A Complete Introduction to the Python Language<\/a> by Mark Summerfield<\/li>\n
- \nPython Programming: An Introduction to Computer Science<\/a> by John Zelle<\/li>\n<\/ul>\n
API Reference<\/h3>\n
- \n
- Keras ImageDataGenerator<\/a><\/li>\n<\/ul>\n
Summary<\/h2>\n
In this tutorial, you discovered special features of Python<\/p>\n
Specifically, you learned:<\/p>\n
- \n
- The purpose of list and dictionary comprehension<\/li>\n
- How to use zip and enumerate<\/li>\n
- Nested functions, function contexts and decorators<\/li>\n
- Generators in Python and the ImageDataGenerator in Python<\/li>\n<\/ul>\n
Do you have any questions about Python features discussed in this post? Ask your questions in the comments below and I will do my best to answer.<\/p>\n
\u00a0<\/p>\n
The post More special features in Python<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
- \nProgramming in Python 3: A Complete Introduction to the Python Language<\/a> by Mark Summerfield<\/li>\n
- \nThink Python: How to Think Like a Computer Scientist<\/a> by Allen B. Downey<\/li>\n
- The
- The inner nested function
![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/12\/m-mani.png\")
<\/a><\/p>\n