Serialization refers to the process of converting a data object (e.g. Python objects, Tensorflow models) into a format that allows us store or transmit the data and then recreate the object when needed using the reverse process of deserialization.<\/p>\n
There are different formats for serialization of data, such as JSON, XML, HDF5, Python\u2019s pickle, for different purposes. JSON for instance returns a human-readable string form while Python\u2019s pickle library can return a byte array.<\/p>\n
In this post, you will discover how to use two common serialization libraries in Python to serialize data objects (namely pickle and HDF5) such as dictionaries and Tensorflow models in Python for storage and transmission<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Serialization libraries in Python such as pickle and h5py<\/li>\n
- Serializing objects such as dictionaries and Tensorflow models in Python<\/li>\n
- How to use serialization for memoization to reduce function calls<\/li>\n<\/ul>\n
Let\u2019s get started!<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/02\/little-plant-TZw891-oMio-unsplash-scaled.jpg\")
<\/p>\nCommand line arguments for your Python script. Photo by little plant<\/a>. Some rights reserved<\/p>\n<\/div>\n
Overview<\/h2>\n
The tutorial is divided into four parts, they are:<\/p>\n
- \n
- What is serialization and why do we serialize?<\/li>\n
- Using Python\u2019s pickle library<\/li>\n
- Using HDF5 in Python<\/li>\n
- Comparison between different serialization methods<\/li>\n<\/ul>\n
What is serialization and why should we care?<\/h2>\n
Think about storing an integer, how would you store that in a file or transmit it? That\u2019s easy! We can just write the integer to a file and store or transmit that file.<\/p>\n
But now, what if we think about storing a Python object (e.g. a Python dictionary or a Pandas DataFrame), which has a complex structure and many attributes (e.g., columns and index of the DataFrame, and the data type of each column)? How would you store it as a file or transmit it to another computer?<\/p>\n
This is where serialization comes in!<\/p>\n
Serialization<\/strong> is the process of converting the object into a format that can be stored or transmitted. After transmitting or storing the serialized data, we are able to later reconstruct the object and obtain the exact same structure\/object, which makes it really convenient for us to continue using the stored object later on instead of reconstructing the object from scratch.<\/p>\n
In Python, there are many different formats for serialization available. One common example for hash maps (Python dictionaries) which works across many languages is the JSON file format which is human-readable and allows us to store the dictionary and recreate it with the same structure. But JSON can only store basic structures such as list and dictionary, and it can only keep strings and numbers. We cannot ask JSON to remember the data type (e.g., numpy float32 vs float64). It also cannot distinguish between Python tuples and lists.<\/p>\n
More powerful serialization formats exists. In the following, we will explore two common serialization libraries in Python, namely pickle and h5py.<\/p>\n
Using Python\u2019s pickle library<\/h2>\n
The
pickle<\/code> module is part of the Python standard library and implements methods to serialize (pickling) and deserialize (unpickling) Python objects.<\/p>\nTo get started with
pickle<\/code>, import it in Python<\/p>\nimport pickle<\/pre>\n
afterwards, to serialize a Python object such as a dictionary and store the byte stream as a file, we can use pickle\u2019s
dump()<\/code> method.<\/p>\ntest_dict = {\"Hello\": \"World!\"}\r\nwith open(\"test.pickle\", \"wb\") as outfile:\r\n \t# \"wb\" argument opens the file in binary mode\r\n\tpickle.dump(test_dict, outfile)<\/pre>\nand the byte stream representing
test_dict<\/code> is now stored in the file \u201ctest.pickle<\/code>\u201d!<\/p>\nTo recover the original object, we read the serialized byte stream from the file using pickle\u2019s
load()<\/code> method.<\/p>\nwith open(\"test.pickle\", \"rb\") as infile:\r\n \ttest_dict_reconstructed = pickle.load(infile)<\/pre>\n
Warning: <\/strong>Only unpickle data from sources you trust as it is possible to for arbitrary malicious code to be executed during the unpickling process.<\/p>\n
Put them together, the following code helps you to verify pickle can recover the same object:<\/p>\n
import pickle\r\n\r\n# A test object\r\ntest_dict = {\"Hello\": \"World!\"}\r\n\r\n# Serialization\r\nwith open(\"test.pickle\", \"wb\") as outfile:\r\n pickle.dump(test_dict, outfile)\r\nprint(\"Written object\", test_dict)\r\n\r\n# Deserialization\r\nwith open(\"test.pickle\", \"rb\") as infile:\r\n test_dict_reconstructed = pickle.load(infile)\r\nprint(\"Reconstructed object\", test_dict_reconstructed)\r\n\r\nif test_dict == test_dict_reconstructed:\r\n print(\"Reconstruction success\")<\/pre>\nBesides writing the serialized object into a pickle file, we can also obtain the object serialized as a bytes-array type in Python using pickle\u2019s
dumps()<\/code> function:<\/p>\ntest_dict_ba = pickle.dumps(test_dict) # b'x80x04x95x15\u2026<\/pre>\n
Similarly, we can use pickle\u2019s loads method to convert from a bytes-array type back to the original object<\/p>\n
test_dict_reconstructed_ba = pickle.loads(test_dict_ba)<\/pre>\n
One useful thing about pickle is that it can serialize almost any Python object, including user-defined ones, such as the following<\/p>\n
import pickle\r\n\r\nclass NewClass:\r\n def __init__(self, data):\r\n print(data)\r\n self.data = data\r\n\r\n# Create an object of NewClass\r\nnew_class = NewClass(1)\r\n\r\n# Serialize and deserialize\r\npickled_data = pickle.dumps(new_class)\r\nreconstructed = pickle.loads(pickled_data)\r\n\r\n# Verify\r\nprint(\"Data from reconstructed object:\", reconstructed.data)<\/pre>\n
The code above will print the following<\/p>\n
1\r\nData from reconstructed object: 1<\/pre>\n
Note that the print statement in the class\u2019 constructor is not executed at the time
pickle.loads()<\/code> is invoked. Because it reconstructed the object, not recreated it.<\/p>\nPickle can even serialize Python functions since functions are first class objects in Python:<\/p>\n
import pickle\r\n\r\ndef test():\r\n return \"Hello world!\"\r\n\r\n# Serialize and deserialize\r\npickled_function = pickle.dumps(test)\r\nreconstructed_function = pickle.loads(pickled_function)\r\n\r\n# Verify\r\nprint (reconstructed_function()) #prints \u201cHello, world!\u201d<\/pre>\n
Therefore, we can make use of pickle to save our work. For example, a trained model from Keras or scikit-learn can be serialized by pickle and load it later instead of re-train the model every time we use it. The following is to show you how we can build a LeNet5 model to recognize the MNIST handwritten digits using Keras, then serialize the trained model using pickle. Afterwards, we can reconstruct the model without training it again and it should produce exactly the same result as the original model:<\/p>\n
import pickle\r\n\r\nimport numpy as np\r\nimport tensorflow as tf\r\nfrom tensorflow.keras.datasets import mnist\r\nfrom tensorflow.keras.models import Sequential\r\nfrom tensorflow.keras.layers import Conv2D, Dense, AveragePooling2D, Dropout, Flatten\r\nfrom tensorflow.keras.utils import to_categorical\r\nfrom tensorflow.keras.callbacks import EarlyStopping\r\n\r\n# Load MNIST digits\r\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\r\n\r\n# Reshape data to (n_samples, height, wiedth, n_channel)\r\nX_train = np.expand_dims(X_train, axis=3).astype(\"float32\")\r\nX_test = np.expand_dims(X_test, axis=3).astype(\"float32\")\r\n\r\n# One-hot encode the output\r\ny_train = to_categorical(y_train)\r\ny_test = to_categorical(y_test)\r\n\r\n# LeNet5 model\r\nmodel = Sequential([\r\n Conv2D(6, (5,5), input_shape=(28,28,1), padding=\"same\", activation=\"tanh\"),\r\n AveragePooling2D((2,2), strides=2),\r\n Conv2D(16, (5,5), activation=\"tanh\"),\r\n AveragePooling2D((2,2), strides=2),\r\n Conv2D(120, (5,5), activation=\"tanh\"),\r\n Flatten(),\r\n Dense(84, activation=\"tanh\"),\r\n Dense(10, activation=\"softmax\")\r\n])\r\n\r\n# Train the model\r\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\r\nearlystopping = EarlyStopping(monitor=\"val_loss\", patience=4, restore_best_weights=True)\r\nmodel.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=100, batch_size=32, callbacks=[earlystopping])\r\n\r\n# Evaluate the model\r\nprint(model.evaluate(X_test, y_test, verbose=0))\r\n\r\n# Pickle to serialize and deserialize\r\npickled_model = pickle.dumps(model)\r\nreconstructed = pickle.loads(pickled_model)\r\n\r\n# Evaluate again\r\nprint(reconstructed.evaluate(X_test, y_test, verbose=0))<\/pre>\n
The above code will produce the output as follows, note that the evaluation scores from the original and reconstructed models are tied out perfectly at the last two lines:<\/p>\n
Epoch 1\/100\r\n1875\/1875 [==============================] - 15s 7ms\/step - loss: 0.1517 - accuracy: 0.9541 - val_loss: 0.0958 - val_accuracy: 0.9661\r\nEpoch 2\/100\r\n1875\/1875 [==============================] - 15s 8ms\/step - loss: 0.0616 - accuracy: 0.9814 - val_loss: 0.0597 - val_accuracy: 0.9822\r\nEpoch 3\/100\r\n1875\/1875 [==============================] - 16s 8ms\/step - loss: 0.0493 - accuracy: 0.9846 - val_loss: 0.0449 - val_accuracy: 0.9853\r\nEpoch 4\/100\r\n1875\/1875 [==============================] - 17s 9ms\/step - loss: 0.0394 - accuracy: 0.9876 - val_loss: 0.0496 - val_accuracy: 0.9838\r\nEpoch 5\/100\r\n1875\/1875 [==============================] - 17s 9ms\/step - loss: 0.0320 - accuracy: 0.9898 - val_loss: 0.0394 - val_accuracy: 0.9870\r\nEpoch 6\/100\r\n1875\/1875 [==============================] - 16s 9ms\/step - loss: 0.0294 - accuracy: 0.9908 - val_loss: 0.0373 - val_accuracy: 0.9872\r\nEpoch 7\/100\r\n1875\/1875 [==============================] - 21s 11ms\/step - loss: 0.0252 - accuracy: 0.9921 - val_loss: 0.0370 - val_accuracy: 0.9879\r\nEpoch 8\/100\r\n1875\/1875 [==============================] - 18s 10ms\/step - loss: 0.0223 - accuracy: 0.9931 - val_loss: 0.0386 - val_accuracy: 0.9880\r\nEpoch 9\/100\r\n1875\/1875 [==============================] - 15s 8ms\/step - loss: 0.0219 - accuracy: 0.9930 - val_loss: 0.0418 - val_accuracy: 0.9871\r\nEpoch 10\/100\r\n1875\/1875 [==============================] - 15s 8ms\/step - loss: 0.0162 - accuracy: 0.9950 - val_loss: 0.0531 - val_accuracy: 0.9853\r\nEpoch 11\/100\r\n1875\/1875 [==============================] - 15s 8ms\/step - loss: 0.0169 - accuracy: 0.9941 - val_loss: 0.0340 - val_accuracy: 0.9895\r\nEpoch 12\/100\r\n1875\/1875 [==============================] - 15s 8ms\/step - loss: 0.0165 - accuracy: 0.9944 - val_loss: 0.0457 - val_accuracy: 0.9874\r\nEpoch 13\/100\r\n1875\/1875 [==============================] - 15s 8ms\/step - loss: 0.0137 - accuracy: 0.9955 - val_loss: 0.0407 - val_accuracy: 0.9879\r\nEpoch 14\/100\r\n1875\/1875 [==============================] - 16s 8ms\/step - loss: 0.0159 - accuracy: 0.9945 - val_loss: 0.0442 - val_accuracy: 0.9871\r\nEpoch 15\/100\r\n1875\/1875 [==============================] - 16s 8ms\/step - loss: 0.0125 - accuracy: 0.9956 - val_loss: 0.0434 - val_accuracy: 0.9882\r\n[0.0340442918241024, 0.9894999861717224]\r\n[0.0340442918241024, 0.9894999861717224]<\/pre>\n
While pickle is a powerful library, it still does have its own limitations to what can be pickled. For example, live connections such as database connections and opened file handles cannot be pickled. This issue arises because reconstructing these objects requires pickle to re-establish the connection with the database\/file which is something pickle cannot do for you (because it needs appropriate credentials and is out of scope of what pickle is intended for).<\/p>\n
Using HDF5 in Python<\/h2>\n
Hierarchical Data Format 5 (HDF5) is a binary data format. The
h5py<\/code> package is a Python library that provides an interface to the HDF5 format. Fromh5py<\/code> docs, HDF5 \u201clets you store huge amounts of numerical data, and easily manipulate that data from Numpy.\u201d<\/p>\nWhat HDF5 can do better than other serialization format is that it stores data in a file system like hierarchy. You can store multiple objects or dataset in HDF5 like saving multiple files in the file system. You can also read a particular dataset from HDF5 like reading one file from the file system without concerning the other. If you\u2019re using pickle for this, you will need to read and write everything each time you load or create the pickle file. Hence HDF5 is advantageous for huge amount of data that can\u2019t fit entirely into memory.<\/p>\n
To get started with
h5py<\/code>, you first need to install theh5py<\/code> library, which you can do using<\/p>\npip install h5py<\/pre>\n
Or if you are using a conda environment<\/p>\n
conda install h5py<\/pre>\n
We can then get started with creating our first dataset!<\/p>\n
import h5py\r\n\r\nwith h5py.File(\"test.hdf5\", \"w\") as file:\r\n dataset = file.create_dataset(\"test_dataset\", (100,), type=\"i4\")<\/pre>\n
This creates a new dataset in the file
test.hdf5<\/code> named \u201ctest_dataset<\/code>\u201d, with a shape of (100, ) and a type int32.h5py<\/code> datasets follow a Numpy syntax so you can do slicing, retrieval, get shape, etc. similar to Numpy arrays<\/p>\nTo retrieve at a specific index,<\/p>\n
dataset[0] #retrieves element at index 0 of dataset<\/pre>\n
To get a slice from index 0 to index 10 of dataset,<\/p>\n
dataset[:10]<\/pre>\n
If you initialized the
h5py<\/code> file object outside of awith<\/code>\u00a0statement, remember to close the file as well!<\/p>\nTo read from a previously created HDF5 file, you can open the file in \u201c
r<\/code>\u201d for read mode or \u201cr+<\/code>\u201d for read\/write mode,<\/p>\nwith h5py.File(\"test.hdf5\", \"r\") as file:\r\n print (file.keys()) #gets names of datasets that are in the file\r\n dataset = file[\"test_dataset\"]<\/pre>\n
To organize your HDF5 file, you can use groups,<\/p>\n
with h5py.File(\"test.hdf5\", \"w\") as file:\r\n # creates new group_1 in file\r\n file.create_group(\"group_1\")\r\n group1 = file[\"group_1\"]\r\n # creates dataset inside group1\r\n group1.create_dataset(\"dataset1\", shape=(10,))\r\n # to access the dataset\r\n dataset = file[\"group_1\"][\"dataset1\"]<\/pre>\n
Another way to create groups and files is by specifying the path to the dataset you want to create and
h5py<\/code> will create the groups on that path as well (if they don\u2019t exist),<\/p>\nwith h5py.File(\"test.hdf5\", \"w\") as file:\r\n # creates dataset inside group1\r\n file.create_dataset(\"group1\/dataset1\", shape=(10,))<\/pre>\n
The two snippets of code both create
group1<\/code> if it has not be created previously, and then adataset1<\/code> withingroup1<\/code>.<\/p>\nHDF5 in Tensorflow<\/h2>\n
To save a model in Tensorflow Keras using HDF5 format, we can use the
save()<\/code> function of the model with a filename having extension.h5<\/code>, like the following:<\/p>\nfrom tensorflow import keras\r\n\r\n# Create model\r\nmodel = keras.models.Sequential([\r\n \tkeras.layers.Input(shape=(10,)),\r\n \tkeras.layers.Dense(1)\r\n])\r\n\r\nmodel.compile(optimizer=\"adam\", loss=\"mse\")\r\n\r\n# using the .h5 extension in the file name specifies that the model\r\n# should be saved in HDF5 format\r\nmodel.save(\"my_model.h5\")<\/pre>\n
To load the stored HDF5 model, we can also use the function from Keras directly:<\/p>\n
...\r\nmodel = keras.models.load_model(\"my_model.h5\")\r\n\r\n# to check that the model has been successfully reconstructed\r\nprint(model.summary)<\/pre>\n
One reason we don\u2019t want to use pickle for a Keras model is that we need a more flexible format that does not tie to a particular version of Keras. If we upgraded our Tensorflow version, the model object may change and pickle may fail to give us a working model. Another reason is to keep only the essential data for our model. For example, if we check the HDF5 file
my_model.h5<\/code> created in the above, we see these are stored:<\/p>\n\/\r\n\/model_weights\r\n\/model_weights\/dense\r\n\/model_weights\/dense\/dense\r\n\/model_weights\/dense\/dense\/bias:0\r\n\/model_weights\/dense\/dense\/kernel:0\r\n\/model_weights\/top_level_model_weights<\/pre>\n
Hence Keras selected only the data that are essential to reconstruct the model. A trained model will contain more datasets, namely, there are
\/optimizer_weights\/<\/code> besides\/model_weights\/<\/code>. Keras will reconstruct the model and restore the weights appropriately to give us a model that function the same.<\/p>\nComparing between different serialization methods<\/h2>\n
In the above we saw how pickle and h5py can help serialize our Python data.<\/p>\n
We can use pickle to serialize almost any Python object, including user-defined ones and functions. But pickle is not language agnostic. You cannot unpickle it outside Python. There are even 6 versions of pickle developed so far and older Python may not be able to consume the newer version of pickle data.<\/p>\n
On the contrary, HDF5 is cross platform and works well with other language such as Java and C++. In Python, the
h5py<\/code> library implemented the Numpy interface to make it easier to manipulate the data. The data can be accessed in different language because HDF5 format supports only the Numpy data types such as float and strings. We cannot store arbitrary objects such as a Python function into HDF5.<\/p>\nFurther reading<\/h2>\n
This section provides more resources on the topic if you are looking to go deeper.<\/p>\n
Articles<\/h4>\n
- \n
- Serialization from C# programming guide, https:\/\/docs.microsoft.com\/en-us\/dotnet\/csharp\/programming-guide\/concepts\/serialization\/<\/a>\n<\/li>\n
- Save and load Keras models, https:\/\/www.tensorflow.org\/guide\/keras\/save_and_serialize<\/a>\n<\/li>\n<\/ul>\n
Libraries<\/h4>\n
- \n
- pickle, https:\/\/docs.python.org\/3\/library\/pickle.html<\/a>\n<\/li>\n
- h5py, https:\/\/docs.h5py.org\/en\/stable\/<\/li>\n<\/ul>\n
APIs<\/h4>\n
- \n
- Tensorflow tf.keras.layers.serialize, https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/keras\/layers\/serialize<\/a>\n<\/li>\n
- Tensorflow tf.keras.models.load_model, https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/keras\/models\/load_model<\/a>\n<\/li>\n
- Tensorflow tf.keras.models.save_model, https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/keras\/models\/save_model<\/a>\n<\/li>\n<\/ul>\n
Summary<\/h2>\n
In this post, you discovered what serialization is and how to use libraries in Python to serialize Python objects such as dictionaries and Tensorflow Keras models. You have also learnt the advantages and disadvantages of two Python libraries for serialization (pickle, h5py).<\/p>\n
Specifically, you learned:<\/p>\n
- \n
- what is serialization and why it is useful<\/li>\n
- how to get started with pickle and h5py serialization libraries in Python<\/li>\n
- pros and cons of different serialization methods<\/li>\n<\/ul>\n
The post A Gentle Introduction to Serialization for Python<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
- Tensorflow tf.keras.models.load_model, https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/keras\/models\/load_model<\/a>\n<\/li>\n
- h5py, https:\/\/docs.h5py.org\/en\/stable\/<\/li>\n<\/ul>\n
- Save and load Keras models, https:\/\/www.tensorflow.org\/guide\/keras\/save_and_serialize<\/a>\n<\/li>\n<\/ul>\n
- Serialization from C# programming guide, https:\/\/docs.microsoft.com\/en-us\/dotnet\/csharp\/programming-guide\/concepts\/serialization\/<\/a>\n<\/li>\n