Static analyzers are tools that help you check your code without really running your code. The most basic form of static analyzers is the syntax highlighters in your favorite editors. If you need to compile your code (say, in C++), your compiler, such as LLVM, may also provide some static analyzer functions to warn you about potential issues (e.g., mistaken assignment \u201c After finishing this tutorial, you will learn some of these tools. Specifically,<\/p>\n Let\u2019s get started.<\/p>\n Static Analyzers in Python This tutorial is in three parts; they are:<\/p>\n Lint was the name of a static analyzer for C created a long time ago. Pylint borrowed its name and is one of the most widely used static analyzers. It is available as a Python package, and we can install it with Then we have the command Pylint can check one script or the entire directory. For example, if we have the following script saved as\u00a0 We can ask Pylint to tell us how good our code is before even running it:<\/p>\n The output is as follows:<\/p>\n If you provide the root directory of a module to Pylint, all components of the module will be checked by Pylint. In that case, you will see the path of different files at the beginning of each line.<\/p>\n There are several things to note here. First, the complaints from Pylint are in different categories. Most commonly we would see issues on convention (i.e., a matter of style), warnings (i.e., the code may run in a sense not consistent with what you intended to do), and error (i.e., the code may fail to run and throw exceptions). They are identified by the code such as E0601, where the first letter is the category.<\/p>\n Pylint may give false positives. In the example above, we see Pylint flagged the import from\u00a0 Furthermore, one of the key feature of Pylint is to help us make our code align with the PEP8 coding style. When we define a function without a docstring, for instance, Pylint will complain that we didn\u2019t follow the coding convention even if the code is not doing anything wrong.<\/p>\n But the most important use of Pylint is to help us identify potential issues. For example, we misspelled All these are\u00a0hints<\/strong>\u00a0provided by Pylint. We still have to use our judgement to correct our code (or ignore Pylint\u2019s complaints).<\/p>\n But if you know what Pylint should stop complaining about, you can request to ignore those. For example, we know the\u00a0 Now, all errors of code E0611 will be ignored by Pylint. You can disable multiple codes by a comma-separated list, e.g.,<\/p>\n If you want to disable some issues on only a specific line or a specific part of the code, you can put special comments to your code, as follows:<\/p>\n The magic keyword The tool Flake8 is indeed a wrapper over PyFlakes, McCabe, and pycodestyle. When you install flake8 with:<\/p>\n you will install all these dependencies.<\/p>\n Similar to Pylint, we have the command\u00a0 The error codes beginning with letter E are from pycodestyle, and those beginning with letter F are from PyFlakes. We can see it complains about coding style issues such as the use of Similar to Pylint, we can also ask Flake8 to ignore some complaints. For example,<\/p>\n Those lines will not be printed in the output:<\/p>\n We can also use magic comments to disable some complaints, e.g.,<\/p>\n Flake8 will look for the comment Python is not a typed language so, unlike C or Java, you do not need to declare the types of some functions or variables before use. But lately, Python has introduced type hint notation, so we can specify what type a function or variable intended<\/strong> to be without enforcing its compliance like a typed language.<\/p>\n One of the biggest benefits of using type hints in Python is to provide additional information for static analyzers to check. Mypy is the tool that can understand type hints. Even without type hints, Mypy can still provide complaints similar to Pylint and Flake8.<\/p>\n We can install Mypy from PyPI:<\/p>\n Then the example above can be provided to the\u00a0 We see similar errors as Pylint above, although sometimes not as precise (e.g., the issue with the variable Let\u2019s consider another example:<\/p>\n This program is supposed to load a HDF5 file (such as a Keras model) and print every attribute and data stored in it. We used the When we save this script into We misused our function so that an opened file object is passed into However, there is one more error in this code that mypy didn\u2019t identify. Namely, the use of the variable The following is the complete, corrected code with no more errors. Note that we added the magic comment \u201c In conclusion, the three tools we introduced above can be complementary to each other. You may consider to run all of them to look for any possible bugs in your code or improve the coding style. Each tool allows some configuration, either from the command line or from a config file, to customize for your needs (e.g., how long a line should be too long to deserve a warning?). Using a static analyzer is also a way to help yourself develop better programming skills.<\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n Articles<\/strong><\/p>\n Software packages<\/strong><\/p>\n In this tutorial, you\u2019ve seen how some common static analyzers can help you write better Python code. Specifically you learned:<\/p>\n The post Static Analyzers in Python<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"=<\/code>\u201d for equality \u201c==<\/code>\u201d in C++). In Python, we have some tools to identify potential errors or point out violations of coding standards.<\/p>\n\n
![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/03\/pexels-skylar-kang-6044187-scaled.jpg\")
<\/p>\n
Photo by Skylar Kang<\/a>. Some rights reserved<\/p>\n<\/div>\nOverview<\/h2>\n
\n
Pylint<\/h2>\n
pip<\/code>:<\/p>\n$ pip install pylint<\/pre>\n
pylint<\/code>\u00a0available in our system.<\/p>\nlenet5-notworking.py<\/code>:<\/p>\nimport numpy as np\r\nimport h5py\r\nimport tensorflow as tf\r\nfrom tensorflow.keras.datasets import mnist\r\nfrom tensorflow.keras.models import Sequential\r\nfrom tensorflow.keras.layers import Conv2D, Dense, AveragePooling2D, Dropout, Flatten\r\nfrom tensorflow.keras.utils import to_categorical\r\nfrom tensorflow.keras.callbacks import EarlyStopping\r\n\r\n# Load MNIST digits\r\n(X_train, Y_train), (X_test, Y_test) = mnist.load_data()\r\n\r\n# Reshape data to (n_samples, height, wiedth, n_channel)\r\nX_train = np.expand_dims(X_train, axis=3).astype(\"float32\")\r\nX_test = np.expand_dims(X_test, axis=3).astype(\"float32\")\r\n\r\n# One-hot encode the output\r\ny_train = to_categorical(y_train)\r\ny_test = to_categorical(y_test)\r\n\r\n# LeNet5 model\r\ndef createmodel(activation):\r\n model = Sequential([\r\n Conv2D(6, (5,5), input_shape=(28,28,1), padding=\"same\", activation=activation),\r\n AveragePooling2D((2,2), strides=2),\r\n Conv2D(16, (5,5), activation=activation),\r\n AveragePooling2D((2,2), strides=2),\r\n Conv2D(120, (5,5), activation=activation),\r\n Flatten(),\r\n Dense(84, activation=activation),\r\n Dense(10, activation=\"softmax\")\r\n ])\r\n return model\r\n\r\n# Train the model\r\nmodel = createmodel(tanh)\r\nmodel.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\r\nearlystopping = EarlyStopping(monitor=\"val_loss\", patience=4, restore_best_weights=True)\r\nmodel.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=100, batch_size=32, callbacks=[earlystopping])\r\n\r\n# Evaluate the model\r\nprint(model.evaluate(X_test, y_test, verbose=0))\r\nmodel.save(\"lenet5.h5\")<\/pre>\n
$ pylint lenet5-notworking.py<\/pre>\n
************* Module lenet5-notworking\r\nlenet5-notworking.py:39:0: C0301: Line too long (115\/100) (line-too-long)\r\nlenet5-notworking.py:1:0: C0103: Module name \"lenet5-notworking\" doesn't conform to snake_case naming style (invalid-name)\r\nlenet5-notworking.py:1:0: C0114: Missing module docstring (missing-module-docstring)\r\nlenet5-notworking.py:4:0: E0611: No name 'datasets' in module 'LazyLoader' (no-name-in-module)\r\nlenet5-notworking.py:5:0: E0611: No name 'models' in module 'LazyLoader' (no-name-in-module)\r\nlenet5-notworking.py:6:0: E0611: No name 'layers' in module 'LazyLoader' (no-name-in-module)\r\nlenet5-notworking.py:7:0: E0611: No name 'utils' in module 'LazyLoader' (no-name-in-module)\r\nlenet5-notworking.py:8:0: E0611: No name 'callbacks' in module 'LazyLoader' (no-name-in-module)\r\nlenet5-notworking.py:18:25: E0601: Using variable 'y_train' before assignment (used-before-assignment)\r\nlenet5-notworking.py:19:24: E0601: Using variable 'y_test' before assignment (used-before-assignment)\r\nlenet5-notworking.py:23:4: W0621: Redefining name 'model' from outer scope (line 36) (redefined-outer-name)\r\nlenet5-notworking.py:22:0: C0116: Missing function or method docstring (missing-function-docstring)\r\nlenet5-notworking.py:36:20: E0602: Undefined variable 'tanh' (undefined-variable)\r\nlenet5-notworking.py:2:0: W0611: Unused import h5py (unused-import)\r\nlenet5-notworking.py:3:0: W0611: Unused tensorflow imported as tf (unused-import)\r\nlenet5-notworking.py:6:0: W0611: Unused Dropout imported from tensorflow.keras.layers (unused-import)\r\n\r\n-------------------------------------\r\nYour code has been rated at -11.82\/10<\/pre>\n
tensorflow.keras.datasets<\/code> as an error. It is caused by an optimization in the Tensorflow package that not everything would be scanned and loaded by Python when we import Tensorflow, but a LazyLoader is created to help load only the necessary part of a large package. This saves significant time in starting the program, but it also confuses Pylint in that we seem to import something that doesn\u2019t exist.<\/p>\ny_train<\/code>\u00a0as\u00a0Y_train<\/code> with an uppercase\u00a0Y<\/code>. Pylint will tell us that we are using a variable without assigning any value to it. It is not straightforwardly telling us what went wrong, but it definitely points us to the right spot to proofread our code. Similarly, when we define the variable model<\/code>\u00a0on line 23, Pylint told us that there is a variable of the same name at the outer scope. Hence the reference to\u00a0model<\/code> later on may not be what we were thinking. Similarly, unused imports may be just that we misspelled the name of the modules.<\/p>\nimport<\/code> statements are fine, so we can invoke Pylint with:<\/p>\n$ pylint -d E0611 lenet5-notworking.py<\/pre>\n
$ pylint -d E0611,C0301 lenet5-notworking.py<\/pre>\n
...\r\nfrom tensorflow.keras.datasets import mnist # pylint: disable=no-name-in-module\r\nfrom tensorflow.keras.models import Sequential # pylint: disable=E0611\r\nfrom tensorflow.keras.layers import Conv2D, Dense, AveragePooling2D, Dropout, Flatten\r\nfrom tensorflow.keras.utils import to_categorical<\/pre>\n
pylint:<\/code>\u00a0will introduce Pylint-specific instructions. The code E0611 and the name\u00a0no-name-in-module<\/code>\u00a0are the same. In the example above, Pylint will complain about the last two import statements but not the first two because of those special comments.<\/p>\nFlake8<\/h2>\n
$ pip install flake8<\/pre>\n
flake8<\/code> after installing this package, and we can pass in a script or a directory for analysis. But the focus of Flake8 is inclined toward coding style. Hence we would see the following output for the same code as above:<\/p>\n$ flake8 lenet5-notworking.py\r\nlenet5-notworking.py:2:1: F401 'h5py' imported but unused\r\nlenet5-notworking.py:3:1: F401 'tensorflow as tf' imported but unused\r\nlenet5-notworking.py:6:1: F401 'tensorflow.keras.layers.Dropout' imported but unused\r\nlenet5-notworking.py:6:80: E501 line too long (85 > 79 characters)\r\nlenet5-notworking.py:18:26: F821 undefined name 'y_train'\r\nlenet5-notworking.py:19:25: F821 undefined name 'y_test'\r\nlenet5-notworking.py:22:1: E302 expected 2 blank lines, found 1\r\nlenet5-notworking.py:24:21: E231 missing whitespace after ','\r\nlenet5-notworking.py:24:41: E231 missing whitespace after ','\r\nlenet5-notworking.py:24:44: E231 missing whitespace after ','\r\nlenet5-notworking.py:24:80: E501 line too long (87 > 79 characters)\r\nlenet5-notworking.py:25:28: E231 missing whitespace after ','\r\nlenet5-notworking.py:26:22: E231 missing whitespace after ','\r\nlenet5-notworking.py:27:28: E231 missing whitespace after ','\r\nlenet5-notworking.py:28:23: E231 missing whitespace after ','\r\nlenet5-notworking.py:36:1: E305 expected 2 blank lines after class or function definition, found 1\r\nlenet5-notworking.py:36:21: F821 undefined name 'tanh'\r\nlenet5-notworking.py:37:80: E501 line too long (86 > 79 characters)\r\nlenet5-notworking.py:38:80: E501 line too long (88 > 79 characters)\r\nlenet5-notworking.py:39:80: E501 line too long (115 > 79 characters)<\/pre>\n
(5,5)<\/code>\u00a0for not having a space after the comma. We can also see it can identify the use of variables before assignment. But it does not catch some\u00a0code smells<\/em>\u00a0such as the function\u00a0createmodel()<\/code>that reuses the variable model<\/code> that was already defined in outer scope.<\/p>\nflake8 --ignore E501,E231 lenet5-notworking.py<\/pre>\n
lenet5-notworking.py:2:1: F401 'h5py' imported but unused\r\nlenet5-notworking.py:3:1: F401 'tensorflow as tf' imported but unused\r\nlenet5-notworking.py:6:1: F401 'tensorflow.keras.layers.Dropout' imported but unused\r\nlenet5-notworking.py:18:26: F821 undefined name 'y_train'\r\nlenet5-notworking.py:19:25: F821 undefined name 'y_test'\r\nlenet5-notworking.py:22:1: E302 expected 2 blank lines, found 1\r\nlenet5-notworking.py:36:1: E305 expected 2 blank lines after class or function definition, found 1\r\nlenet5-notworking.py:36:21: F821 undefined name 'tanh'<\/pre>\n
...\r\nimport tensorflow as tf # noqa: F401\r\nfrom tensorflow.keras.datasets import mnist\r\nfrom tensorflow.keras.models import Sequential<\/pre>\n
# noqa:<\/code> to skip some complaints on those particular lines.<\/p>\nMypy<\/h2>\n
$ pip install mypy<\/pre>\n
mypy<\/code>\u00a0command:<\/p>\n$ mypy lenet5-notworking.py\r\nlenet5-notworking.py:2: error: Skipping analyzing \"h5py\": module is installed, but missing library stubs or py.typed marker\r\nlenet5-notworking.py:2: note: See https:\/\/mypy.readthedocs.io\/en\/stable\/running_mypy.html#missing-imports\r\nlenet5-notworking.py:3: error: Skipping analyzing \"tensorflow\": module is installed, but missing library stubs or py.typed marker\r\nlenet5-notworking.py:4: error: Skipping analyzing \"tensorflow.keras.datasets\": module is installed, but missing library stubs or py.typed marker\r\nlenet5-notworking.py:5: error: Skipping analyzing \"tensorflow.keras.models\": module is installed, but missing library stubs or py.typed marker\r\nlenet5-notworking.py:6: error: Skipping analyzing \"tensorflow.keras.layers\": module is installed, but missing library stubs or py.typed marker\r\nlenet5-notworking.py:7: error: Skipping analyzing \"tensorflow.keras.utils\": module is installed, but missing library stubs or py.typed marker\r\nlenet5-notworking.py:8: error: Skipping analyzing \"tensorflow.keras.callbacks\": module is installed, but missing library stubs or py.typed marker\r\nlenet5-notworking.py:18: error: Cannot determine type of \"y_train\"\r\nlenet5-notworking.py:19: error: Cannot determine type of \"y_test\"\r\nlenet5-notworking.py:36: error: Name \"tanh\" is not defined\r\nFound 10 errors in 1 file (checked 1 source file)<\/pre>\n
y_train<\/code>). However we see one characteristic of mypy above: It expects all libraries we used to come with a stub so the type checking can be done. This is because type hints are optional<\/strong>. In case the code from a library does not provide type hints, the code can still work, but mypy cannot verify. Some of the libraries have typing stubs<\/strong> available that enables mypy to check them better.<\/p>\nimport h5py\r\n\r\ndef dumphdf5(filename: str) -> int:\r\n \"\"\"Open a HDF5 file and print all the dataset and attributes stored\r\n\r\n Args:\r\n filename: The HDF5 filename\r\n\r\n Returns:\r\n Number of dataset found in the HDF5 file\r\n \"\"\"\r\n count: int = 0\r\n\r\n def recur_dump(obj) -> None:\r\n print(f\"{obj.name} ({type(obj).__name__})\")\r\n if obj.attrs.keys():\r\n print(\"tAttribs:\")\r\n for key in obj.attrs.keys():\r\n print(f\"tt{key}: {obj.attrs[key]}\")\r\n if isinstance(obj, h5py.Group):\r\n # Group has key-value pairs\r\n for key, value in obj.items():\r\n recur_dump(value)\r\n elif isinstance(obj, h5py.Dataset):\r\n count += 1\r\n print(obj[()])\r\n\r\n with h5py.File(filename) as obj:\r\n recur_dump(obj)\r\n print(f\"{count} dataset found\")\r\n\r\nwith open(\"my_model.h5\") as fp:\r\n dumphdf5(fp)<\/pre>\nh5py<\/code> module (which does not have a typing stub, and hence mypy cannot identify the types it used), but we added type hints to the function we defined, dumphdf5()<\/code>. This function expects the filename of a HDF5 file and prints everything stored inside. At the end, the number of datasets stored will be returned.<\/p>\ndumphdf5.py<\/code>\u00a0and pass it into mypy, we will see the following:<\/p>\n$ mypy dumphdf5.py\r\ndumphdf5.py:1: error: Skipping analyzing \"h5py\": module is installed, but missing library stubs or py.typed marker\r\ndumphdf5.py:1: note: See https:\/\/mypy.readthedocs.io\/en\/stable\/running_mypy.html#missing-imports\r\ndumphdf5.py:3: error: Missing return statement\r\ndumphdf5.py:33: error: Argument 1 to \"dumphdf5\" has incompatible type \"TextIO\"; expected \"str\"\r\nFound 3 errors in 1 file (checked 1 source file)<\/pre>\n
dumphdf5()<\/code> instead of just the filename (as a string). Mypy can identify this error. We also declared that the function should return an integer, but we didn\u2019t have the return statement in the function.<\/p>\ncount<\/code>\u00a0in the inner function\u00a0recur_dump()<\/code>\u00a0should be declared\u00a0nonlocal<\/code>\u00a0because it is defined out of scope. This error can be caught by Pylint and Flake8, but mypy missed it.<\/p>\n# type: ignore<\/code>\u201d at the first line to mute the typing stubs warning from mypy:<\/p>\nimport h5py # type: ignore\r\n\r\n\r\ndef dumphdf5(filename: str) -> int:\r\n \"\"\"Open a HDF5 file and print all the dataset and attributes stored\r\n\r\n Args:\r\n filename: The HDF5 filename\r\n\r\n Returns:\r\n Number of dataset found in the HDF5 file\r\n \"\"\"\r\n count: int = 0\r\n\r\n def recur_dump(obj) -> None:\r\n nonlocal count\r\n print(f\"{obj.name} ({type(obj).__name__})\")\r\n if obj.attrs.keys():\r\n print(\"tAttribs:\")\r\n for key in obj.attrs.keys():\r\n print(f\"tt{key}: {obj.attrs[key]}\")\r\n if isinstance(obj, h5py.Group):\r\n # Group has key-value pairs\r\n for key, value in obj.items():\r\n recur_dump(value)\r\n elif isinstance(obj, h5py.Dataset):\r\n count += 1\r\n print(obj[()])\r\n\r\n with h5py.File(filename) as obj:\r\n recur_dump(obj)\r\n print(f\"{count} dataset found\")\r\n return count\r\n\r\n\r\ndumphdf5(\"my_model.h5\")<\/pre>\nFurther reading<\/h2>\n
\n
\n
Summary<\/h2>\n
\n