implementing the general attention mechanism<\/a>, we can break into the code as follows:<\/p>\nfrom numpy import array\r\nfrom numpy import random\r\nfrom numpy import dot\r\nfrom scipy.special import softmax\r\n\r\n# importing the Python debugger module\r\nimport pdb\r\n\r\n# encoder representations of four different words\r\nword_1 = array([1, 0, 0])\r\nword_2 = array([0, 1, 0])\r\nword_3 = array([1, 1, 0])\r\nword_4 = array([0, 0, 1])\r\n\r\n# stacking the word embeddings into a single array\r\nwords = array([word_1, word_2, word_3, word_4])\r\n\r\n# generating the weight matrices\r\nrandom.seed(42)\r\nW_Q = random.randint(3, size=(3, 3))\r\nW_K = random.randint(3, size=(3, 3))\r\nW_V = random.randint(3, size=(3, 3))\r\n\r\n# generating the queries, keys and values\r\nQ = dot(words, W_Q)\r\nK = dot(words, W_K)\r\nV = dot(words, W_V)\r\n\r\n# inserting a breakpoint\r\npdb.set_trace()\r\n\r\n# scoring the query vectors against all key vectors\r\nscores = dot(Q, K.transpose())\r\n\r\n# computing the weights by a softmax operation\r\nweights = softmax(scores \/ K.shape[1] ** 0.5, axis=1)\r\n\r\n# computing the attention by a weighted sum of the value vectors\r\nattention = dot(weights, V)\r\n\r\nprint(attention)<\/pre>\nExecuting the script now opens up the pdb<\/code> debugger right before we compute the variable scores<\/code>, and we can proceed to issue any debugger commands of choice, such as
\n\t\t\tn<\/span><\/span><\/span>\u00a0to move to the next line, or
\n\t\t\tc<\/span><\/span><\/span>\u00a0to continue execution:<\/p>\n\/Users\/mlm\/main.py(33)<module>()\r\n-> scores = dot(Q, K.transpose())\r\n(Pdb) n\r\n> \/Users\/mlm\/main.py(36)<module>()\r\n-> weights = softmax(scores \/ K.shape[1] ** 0.5, axis=1)\r\n(Pdb) c\r\n[[0.98522025 1.74174051 0.75652026]\r\n [0.90965265 1.40965265 0.5 ]\r\n [0.99851226 1.75849334 0.75998108]\r\n [0.99560386 1.90407309 0.90846923]]<\/pre>\nAlthough functional, this is not the most elegant and intuitive approach of inserting a breakpoint into your code. Python 3.7 implements a more straightforward way of doing so, as we shall see next.<\/p>\n
Using the breakpoint() function in Python 3.7\u00a0<\/b><\/h3>\n
Python 3.7 comes with a built-in
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span>\u00a0function that enters the Python debugger at the call site (or the point in the code at which the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span>\u00a0statement is placed).\u00a0<\/span><\/p>\nWhen called, the default implementation of the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function will call
\n\t\t\tsys<\/span>.<\/span>breakpointhook<\/span>(<\/span>)<\/span><\/span><\/span>, which in turn calls the
\n\t\t\tpdb<\/span>.<\/span>set_trace<\/span>(<\/span>)<\/span><\/span><\/span>\u00a0function. This is convenient because we will not need to
\n\t\t\timport<\/span> <\/span>pdb<\/span><\/span><\/span>\u00a0and call
\n\t\t\tpdb<\/span>.<\/span>set_trace<\/span>(<\/span>)<\/span><\/span><\/span>\u00a0explicitly ourselves.\u00a0<\/span><\/p>\nLet\u2019s reconsider the code for implementing the general attention mechanism, and now introduce a breakpoint via the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> statement:<\/p>\nfrom numpy import array\r\nfrom numpy import random\r\nfrom scipy.special import softmax\r\n\r\n# encoder representations of four different words\r\nword_1 = array([1, 0, 0])\r\nword_2 = array([0, 1, 0])\r\nword_3 = array([1, 1, 0])\r\nword_4 = array([0, 0, 1])\r\n\r\n# stacking the word embeddings into a single array\r\nwords = array([word_1, word_2, word_3, word_4])\r\n\r\n# generating the weight matrices\r\nrandom.seed(42)\r\nW_Q = random.randint(3, size=(3, 3))\r\nW_K = random.randint(3, size=(3, 3))\r\nW_V = random.randint(3, size=(3, 3))\r\n\r\n# generating the queries, keys and values\r\nQ = words @ W_Q\r\nK = words @ W_K\r\nV = words @ W_V\r\n\r\n# inserting a breakpoint\r\nbreakpoint()\r\n\r\n# scoring the query vectors against all key vectors\r\nscores = Q @ K.transpose()\r\n\r\n# computing the weights by a softmax operation\r\nweights = softmax(scores \/ K.shape[1] ** 0.5, axis=1)\r\n\r\n# computing the attention by a weighted sum of the value vectors\r\nattention = weights @ V\r\n\r\nprint(attention)<\/pre>\nOne advantage of using the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function is that, in calling the default implementation of
\n\t\t\tsys<\/span>.<\/span>breakpointhook<\/span>(<\/span>)<\/span><\/span><\/span> the value of a new environment variable,
\n\t\t\tPYTHONBREAKPOINT<\/span><\/span><\/span>, is consulted. This environment variable can take various values, based on which different operations can be performed.\u00a0<\/span><\/p>\nFor example, setting the value of
\n\t\t\tPYTHONBREAKPOINT<\/span><\/span><\/span> to 0 disables all breakpoints. Hence, your code could contain as many breakpoints as necessary, but these can be easily stopped from\u00a0<\/span>halting the execution of the code without having to remove them physically. If (for example) the name of the script containing the code is main.py<\/em>, we would disable all breakpoints by calling it in the command line interface as follows:<\/p>\nPYTHONBREAKPOINT=0 python main.py<\/pre>\nOtherwise, we can achieve the same outcome by setting the environment variable in the code itself:<\/p>\n
from numpy import array\r\nfrom numpy import random\r\nfrom scipy.special import softmax\r\n\r\n# setting the value of the PYTHONBREAKPOINT environment variable\r\nimport os\r\nos.environ['PYTHONBREAKPOINT'] = '0'\r\n\r\n# encoder representations of four different words\r\nword_1 = array([1, 0, 0])\r\nword_2 = array([0, 1, 0])\r\nword_3 = array([1, 1, 0])\r\nword_4 = array([0, 0, 1])\r\n\r\n# stacking the word embeddings into a single array\r\nwords = array([word_1, word_2, word_3, word_4])\r\n\r\n# generating the weight matrices\r\nrandom.seed(42)\r\nW_Q = random.randint(3, size=(3, 3))\r\nW_K = random.randint(3, size=(3, 3))\r\nW_V = random.randint(3, size=(3, 3))\r\n\r\n# generating the queries, keys and values\r\nQ = words @ W_Q\r\nK = words @ W_K\r\nV = words @ W_V\r\n\r\n# inserting a breakpoint\r\nbreakpoint()\r\n\r\n# scoring the query vectors against all key vectors\r\nscores = Q @ K.transpose()\r\n\r\n# computing the weights by a softmax operation\r\nweights = softmax(scores \/ K.shape[1] ** 0.5, axis=1)\r\n\r\n# computing the attention by a weighted sum of the value vectors\r\nattention = weights @ V\r\n\r\nprint(attention)<\/pre>\nThe value of
\n\t\t\tPYTHONBREAKPOINT<\/span><\/span><\/span> is consulted every time that
\n\t\t\tsys<\/span>.<\/span>breakpointhook<\/span>(<\/span>)<\/span><\/span><\/span> is called. This means that the value of this environment variable can be changed during the code execution and the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function would respond accordingly. \u00a0<\/span><\/p>\nThe
\n\t\t\tPYTHONBREAKPOINT<\/span><\/span><\/span> environment variable can also be set to other values, such as to the name of a callable. Say, for instance, that we\u2019d like to use a different Python debugger other than pdb, such as ipdb (run
\n\t\t\tpip <\/span>install <\/span>ipdb<\/span><\/span><\/span>\u00a0first, if the debugger has not yet been installed). In this case, we would call the main.py<\/em> script in the command line interface, and hook the debugger without making any changes to the code itself:<\/p>\nPYTHONBREAKPOINT=ipdb.set_trace python main.py<\/pre>\nIn doing so, the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function enters the ipdb debugger at the next call site:<\/p>\n> \/Users\/Stefania\/Documents\/PycharmProjects\/BreakpointPy37\/main.py(33)<module>()\r\n 32 # scoring the query vectors against all key vectors\r\n---> 33 scores = Q @ K.transpose()\r\n 34 \r\n\r\nipdb> n\r\n> \/Users\/Stefania\/Documents\/PycharmProjects\/BreakpointPy37\/main.py(36)<module>()\r\n 35 # computing the weights by a softmax operation\r\n---> 36 weights = softmax(scores \/ K.shape[1] ** 0.5, axis=1)\r\n 37 \r\n\r\nipdb> c\r\n[[0.98522025 1.74174051 0.75652026]\r\n [0.90965265 1.40965265 0.5 ]\r\n [0.99851226 1.75849334 0.75998108]\r\n [0.99560386 1.90407309 0.90846923]]<\/pre>\nThe function can also take input arguments as,
\n\t\t\tbreakpoint<\/span>(<\/span>*<\/span>args<\/span>,<\/span> <\/span>*<\/span>*<\/span>kws<\/span>)<\/span><\/span><\/span>, which are then passed on to
\n\t\t\tsys<\/span>.<\/span>breakpointhook<\/span>(<\/span>)<\/span><\/span><\/span>. This is because any callable (such as a third party debugger module) might accept optional arguments, which can be passed through the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function.\u00a0<\/span><\/p>\nWriting your own breakpoint() function in earlier versions of Python<\/b><\/h2>\n
Let\u2019s return to the fact that versions of Python earlier than v3.7 do not come with the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function readily built in. We can write our own.\u00a0<\/span><\/p>\nSimilarly to how the
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function is implemented from Python 3.7 onwards, we can implement a function that checks the value of an environment variable and:<\/p>\n\n- Skips all breakpoints in the code if the value of the environment variable is set to 0.<\/li>\n
- Enters into the default Python pdb debugger if the environment variable is an empty string.<\/li>\n
- Enters into another debugger as specified by the value of the environment variable.\u00a0<\/span>\n<\/li>\n<\/ul>\n
...\r\n\r\n# defining our breakpoint() function\r\ndef breakpoint(*args, **kwargs):\r\n import importlib\r\n # reading the value of the environment variable\r\n val = os.environ.get('PYTHONBREAKPOINT')\r\n # if the value has been set to 0, skip all breakpoints\r\n if val == '0':\r\n return None\r\n # else if the value is an empty string, invoke the default pdb debugger\r\n elif len(val) == 0:\r\n hook_name = 'pdb.set_trace'\r\n # else, assign the value of the environment variable\r\n else:\r\n hook_name = val\r\n # split the string into the module name and the function name\r\n mod, dot, func = hook_name.rpartition('.')\r\n # get the function from the module\r\n module = importlib.import_module(mod)\r\n hook = getattr(module, func)\r\n\r\n return hook(*args, **kwargs)\r\n\r\n...<\/pre>\nWe can include this function into the code and run it (using a Python 2.7 interpreter, in this case). If we set the value of the environment variable to an empty string, we find that the pdb debugger stops at the point in the code at which we have placed our
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function. We can then issue debugger commands into the command line from there onwards:<\/p>\nfrom numpy import array\r\nfrom numpy import random\r\nfrom numpy import dot\r\nfrom scipy.special import softmax\r\n\r\n# setting the value of the environment variable\r\nimport os\r\nos.environ['PYTHONBREAKPOINT'] = ''\r\n\r\n\r\n# defining our breakpoint() function\r\ndef breakpoint(*args, **kwargs):\r\n import importlib\r\n # reading the value of the environment variable\r\n val = os.environ.get('PYTHONBREAKPOINT')\r\n # if the value has been set to 0, skip all breakpoints\r\n if val == '0':\r\n return None\r\n # else if the value is an empty string, invoke the default pdb debugger\r\n elif len(val) == 0:\r\n hook_name = 'pdb.set_trace'\r\n # else, assign the value of the environment variable\r\n else:\r\n hook_name = val\r\n # split the string into the module name and the function name\r\n mod, dot, func = hook_name.rpartition('.')\r\n # get the function from the module\r\n module = importlib.import_module(mod)\r\n hook = getattr(module, func)\r\n\r\n return hook(*args, **kwargs)\r\n\r\n\r\n# encoder representations of four different words\r\nword_1 = array([1, 0, 0])\r\nword_2 = array([0, 1, 0])\r\nword_3 = array([1, 1, 0])\r\nword_4 = array([0, 0, 1])\r\n\r\n# stacking the word embeddings into a single array\r\nwords = array([word_1, word_2, word_3, word_4])\r\n\r\n# generating the weight matrices\r\nrandom.seed(42)\r\nW_Q = random.randint(3, size=(3, 3))\r\nW_K = random.randint(3, size=(3, 3))\r\nW_V = random.randint(3, size=(3, 3))\r\n\r\n# generating the queries, keys and values\r\nQ = dot(words, W_Q)\r\nK = dot(words, W_K)\r\nV = dot(words, W_V)\r\n\r\n# inserting a breakpoint\r\nbreakpoint()\r\n\r\n# scoring the query vectors against all key vectors\r\nscores = dot(Q, K.transpose())\r\n\r\n# computing the weights by a softmax operation\r\nweights = softmax(scores \/ K.shape[1] ** 0.5, axis=1)\r\n\r\n# computing the attention by a weighted sum of the value vectors\r\nattention = dot(weights, V)\r\n\r\nprint(attention)<\/pre>\n<\/p>\n> \/Users\/Stefania\/Documents\/PycharmProjects\/BreakpointPy27\/main.py(32)breakpoint()->None\r\n-> return hook(*args, **kwargs)\r\n(Pdb) n\r\n> \/Users\/Stefania\/Documents\/PycharmProjects\/BreakpointPy27\/main.py(59)<module>()\r\n-> scores = dot(Q, K.transpose())\r\n(Pdb) n\r\n> \/Users\/Stefania\/Documents\/PycharmProjects\/BreakpointPy27\/main.py(62)<module>()\r\n-> weights = softmax(scores \/ K.shape[1] ** 0.5, axis=1)\r\n(Pdb) c\r\n[[0.98522025 1.74174051 0.75652026]\r\n [0.90965265 1.40965265 0.5 ]\r\n [0.99851226 1.75849334 0.75998108]\r\n [0.99560386 1.90407309 0.90846923]]<\/pre>\nSimilarly, if we set the environment variable to:<\/p>\n
os.environ['PYTHONBREAKPOINT'] = 'ipdb.set_trace'<\/pre>\nThe
\n\t\t\tbreakpoint<\/span>(<\/span>)<\/span><\/span><\/span> function that we have implemented now enters the ipdb debugger and stops at the call site:<\/p>\n> \/Users\/Stefania\/Documents\/PycharmProjects\/BreakpointPy27\/main.py(31)breakpoint()\r\n 30 \r\n---> 31 return hook(*args, **kwargs)\r\n 32 \r\n\r\nipdb> n\r\n> \/Users\/Stefania\/Documents\/PycharmProjects\/BreakpointPy27\/main.py(58)<module>()\r\n 57 # scoring the query vectors against all key vectors\r\n---> 58 scores = dot(Q, K.transpose())\r\n 59 \r\n\r\nipdb> n\r\n> \/Users\/Stefania\/Documents\/PycharmProjects\/BreakpointPy27\/main.py(61)<module>()\r\n 60 # computing the weights by a softmax operation\r\n---> 61 weights = softmax(scores \/ K.shape[1] ** 0.5, axis=1)\r\n 62 \r\n\r\nipdb> c\r\n[[0.98522025 1.74174051 0.75652026]\r\n [0.90965265 1.40965265 0.5 ]\r\n [0.99851226 1.75849334 0.75998108]\r\n [0.99560386 1.90407309 0.90846923]]<\/pre>\n
![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/01\/cover_breakpoints-scaled.jpg\")
<\/a><\/p>\n