{"id":2449,"date":"2019-08-09T06:31:55","date_gmt":"2019-08-09T06:31:55","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2019\/08\/09\/detecting-bias-with-shap\/"},"modified":"2019-08-09T06:31:55","modified_gmt":"2019-08-09T06:31:55","slug":"detecting-bias-with-shap","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2019\/08\/09\/detecting-bias-with-shap\/","title":{"rendered":"Detecting Bias with SHAP"},"content":{"rendered":"

Author: Sean Owen<\/p>\n

\n

Originally published on the Databricks blog<\/a> with accompanying notebook<\/a>.<\/i><\/p>\n

StackOverflow\u2019s annual developer survey concluded earlier this year, and they have graciously published the (anonymized) 2019 results for analysis. They\u2019re a rich view into the experience of software developers around the world \u2014 what\u2019s their favorite editor? how many years of experience? tabs or spaces? and crucially, salary. Software engineers\u2019 salaries are good, and sometimes both eye-watering and news-worthy.<\/p>\n

The tech industry is also painfully aware that it does not always live up to its purported meritocratic ideals. Pay isn\u2019t a pure function of merit, and story after story tells us that factors like name-brand school, age, race, and gender have an effect on outcomes like salary.<\/p>\n

Can machine learning do more than predict things? Can it explain salaries and so highlight cases where these factors might be undesirably causing pay differences? This example will sketch how standard models can be augmented with SHAP (SHapley Additive exPlanations) to detect individual instances whose predictions may be concerning, and then dig deeper into the specific reasons the data leads to those predictions.<\/p>\n

Model Bias or Data (about) Bias?<\/h1>\n

While this topic is often characterized as detecting \u201cmodel bias\u201d, a model is merely a mirror of the data it was trained on. If the model is \u2018biased\u2019 then it learned that from the historical facts of the data. Models are not the problem per se; they are an opportunity to analyze data for evidence of bias.<\/p>\n

Explaining models isn\u2019t new, and most libraries can assess the relative importance of the inputs to a model. These are aggregate views of inputs\u2019 effects. However, the output of some machine learning models has highly individual effects: is your loan approved? will you receive financial aid? are you a suspicious traveller?<\/p>\n

Indeed, StackOverflow offers a handy calculator to estimate one\u2019s expected salary, based on its survey. We can only speculate about how accurate the predictions are overall, but all that a developer particularly cares about is his or her own prospects.<\/p>\n

The right question may not be, does the data suggest bias overall? but rather, does the data show individual instances of bias?<\/p>\n

Assessing the Survey Data<\/h1>\n

The 2019 data is, thankfully, clean and free of data problems. It contains responses to 85 questions from about 88,000 developers.<\/p>\n

This example focuses only on full-time developers. The data set contains plenty of relevant information, like years of experience, education, role, and demographic information. Notably, this data set doesn\u2019t contain information about bonuses and equity, just salary.<\/p>\n

It also has responses to wide-ranging questions about attitudes on blockchain, fizz buzz, and the survey itself. These are excluded here as unlikely to reflect the experience and skills that presumably should<\/i> determine compensation. Likewise, for simplicity, it will also only focus on US-based developers.<\/p>\n

The data needs a little more transformation before modeling. Several questions allow multiple responses, like \u201cWhat are your greatest challenges to productivity as a developer?\u201d<\/i> These single questions yield multiple yes\/no responses and need to be broken out into multiple yes\/no features.<\/p>\n

Some multiple-choice questions like \u201cApproximately how many people are employed by the company or organization you work for?\u201d<\/i> afford responses like \u201c2-9 employees\u201d<\/i>. These are effectively binned continuous values, and it may be useful to map them back to inferred continuous values like \u201c2\u201d so that the model may consider their order and relative magnitude. This translation is unfortunately manual and entails some judgment calls.<\/p>\n

The Apache Spark code that can accomplish this is in the accompanying notebook, for the interested.<\/p>\n

Model Selection with Apache Spark<\/h1>\n

With the data in a more machine-learning-friendly form, the next step is to fit a regression model that predicts salary from these features. The data set itself, after filtering and transformation with Spark, is a mere 4MB, containing 206 features from about 12,600 developers, and could easily fit in memory as a Pandas Dataframe on your wristwatch, let alone a server.<\/p>\n

xgboost<\/code>, a popular gradient-boosted trees package, can fit a model to this data in minutes on a single machine, without Spark. xgboost<\/code> offers many tunable \u201chyperparameters\u201d that affect the quality of the model: maximum depth, learning rate, regularization, and so on. Rather than guess, simple standard practice is to try lots of settings of these values and pick the combination that results in the most accurate model.<\/p>\n

Fortunately, this is where Spark comes back in. It can build hundreds of these models in parallel and collect the results of each. Because the data set is small, it\u2019s simple to broadcast it to the workers, create a bunch of combinations of those hyperparameters to try, and use Spark to apply the same simple non-distributed xgboost<\/code> code that could build a model locally to the data with each combination.<\/p>\n

...def train_model(params):  (max_depth, learning_rate, reg_alpha, reg_lambda, gamma, min_child_weight) = params  
\n  xgb_regressor = XGBRegressor(objective='reg:squarederror', max_depth=max_depth,
\n    learning_rate=learning_rate, reg_alpha=reg_alpha, reg_lambda=reg_lambda, gamma=gamma,
\n    min_child_weight=min_child_weight, n_estimators=3000, base_score=base_score,
\n    importance_type='total_gain', random_state=0)
\n  xgb_model = xgb_regressor.fit(b_X_train.value, b_y_train.value,
\n    eval_set=[(b_X_test.value, b_y_test.value)],
\n                                eval_metric='rmse', early_stopping_rounds=30)
\n  n_estimators = len(xgb_model.evals_result()['validation_0']['rmse'])
\n  y_pred = xgb_model.predict(b_X_test.value)
\n  mae = mean_absolute_error(y_pred, b_y_test.value)
\n  rmse = sqrt(mean_squared_error(y_pred, b_y_test.value))
\n  return (params + (n_estimators,), (mae, rmse), xgb_model)

\n...

\nmax_depth =        np.unique(np.geomspace(3, 7, num=5, dtype=np.int32)).tolist()
\nlearning_rate =    np.unique(np.around(np.geomspace(0.01, 0.1, num=5), decimals=3)).tolist()
\nreg_alpha =        [0] + np.unique(np.around(np.geomspace(1, 50, num=5), decimals=3)).tolist()
\nreg_lambda =       [0] + np.unique(np.around(np.geomspace(1, 50, num=5), decimals=3)).tolist()
\ngamma =            np.unique(np.around(np.geomspace(5, 20, num=5), decimals=3)).tolist()
\nmin_child_weight = np.unique(np.geomspace(5, 30, num=5, dtype=np.int32)).tolist()

\nparallelism = 128
\nparam_grid = [(choice(max_depth), choice(learning_rate), choice(reg_alpha),
\n  choice(reg_lambda), choice(gamma), choice(min_child_weight)) for _ in range(parallelism)]

\nparams_evals_models = sc.parallelize(param_grid, parallelism).map(train_model).collect()
<\/code><\/pre>\n
That will create a lot of models. To track and evaluate the results, mlflow<\/code> can log each one with its metrics and hyperparameters, and view them in the notebook\u2019s Experiment. Here, one hyperparameter over many runs is compared to the resulting accuracy (mean absolute error):<\/p>\n
\"\"<\/p>\n
The single model that showed the lowest error on the held-out validation data set is of interest. It yielded a mean absolute error of about $28,000 on salaries that average about $119,000. Not terrible, although we should realize the model can only explain most<\/i> of the variation in salary.<\/p>\n
Interpreting the xgboost Model<\/h1>\n
Although the model can be used to predict future salaries, instead, the question is what the model says about the data. What features seem to matter most when predicting salary accurately? The xgboost<\/code> model itself computes a notion of feature importance:<\/p>\n
import mlflow.sklearnbest_run_id = \"...\"model = mlflow.sklearn.load_model(\"runs:\/\" + best_run_id + \"\/xgboost\")
\nsorted((zip(model.feature_importances_, X.columns)), reverse=True)[:6]
<\/code><\/pre>\n
\"\"<\/p>\n
Factors like years of coding professionally, organization size, and using Windows are most \u201cimportant\u201d. This is interesting, but hard to interpret. The values reflect relative and not absolute importance. That is, the effect isn\u2019t measured in dollars. The definition of importance here (total gain) is also specific to how decision trees are built and are hard to map to an intuitive interpretation. The important features don\u2019t even necessarily correlate positively with salary, either.<\/p>\n
More importantly, this is a \u2018global\u2019 view of how much features matter in aggregate. Factors like gender and ethnicity don\u2019t show up on this list until farther along. This doesn\u2019t mean these factors aren\u2019t still significant. For one, features can be correlated, or interact. It\u2019s possible that factors like gender correlate with other features that the trees selected instead, and this to some degree masks their effect.<\/p>\n
The more interesting question is not so much whether these factors matter overall \u2014 it\u2019s possible that their average effect is relatively small \u2014 but, whether they have a significant effect in some individual cases. These are the instances where the model is telling us something important about individuals\u2019 experience, and to those individuals, that experience is what matters.<\/p>\n
Applying SHAP for Developer-Level Explanations<\/h1>\n
Fortunately, a set of techniques for more theoretically sound model interpretation at the individual prediction level has emerged over the past five years or so. They are collectively \u201cSh<\/b>apley A<\/b>dditive Exp<\/b>lanations\u201d, and conveniently, are implemented in the Python package shap<\/code>.<\/p>\n
Given any model, this library computes \u201cSHAP values\u201d from the model. These values are readily interpretable, as each value is a feature\u2019s effect on the prediction, in its units. A SHAP value of 1000 here means \u201cexplained +$1,000 of predicted salary\u201d. SHAP values are computed in a way that attempts to isolate away of correlation and interaction, as well.<\/p>\n
import shapexplainer = shap.TreeExplainer(model)shap_values = explainer.shap_values(X, y=y.values)
<\/code><\/pre>\n
SHAP values are also computed for every input, not the model as a whole, so these explanations are available for each input individually. It can also estimate the effect of feature interactions separately from the main effect of each feature, for each prediction.<\/p>\n
Explaining the Features\u2019 Effects Overall<\/h1>\n
Developer-level explanations can aggregate into explanations of the features\u2019 effects on salary over the whole data set by simply averaging their absolute values. SHAP\u2019s assessment of the overall most important features is similar:<\/p>\n
\"\"<\/p>\n
The SHAP values tell a similar story. First, SHAP is able to quantify the effect on salary in dollars, which greatly improves the interpretation of the results. Above is a plot the absolute<\/i> effect of each feature on predicted salary, averaged across developers. Years of professional coding experience still dominates, explaining on average almost $15,000 of effect on salary.<\/p>\n
Examining the Effects of Gender with SHAP Values<\/h1>\n
We came to look specifically at the effects of gender, race, and other factors that presumably should not be predictive per se of salary at all. This example will examine the effect of gender, though this by no means suggests that it\u2019s the only or most important, type of bias to look for.<\/p>\n
Gender is not binary, and the survey recognizes responses of \u201cMan\u201d, \u201cWoman\u201d, and \u201cNon-binary, genderqueer, or gender non-conforming\u201d as well as \u201cTrans\u201d separately. (Note that while the survey also separately records responses about sexuality, these are not considered here.) SHAP computes the effect on predicted salary for each of these. For a male developer (identifying only as male), the effect of gender is not just the effect of being male, but of not identifying as female, transgender, and so on.<\/p>\n
SHAP values let us read off the sum of these effects for developers identifying as each of the four categories:<\/p>\n
\"\"<\/p>\n
While male developers\u2019 gender explains about a modest -$230 to +$890 with mean about $225, for females, the range is wider, from about -$4,260 to -$690 with mean -$1,320. The results for transgender and non-binary developers is similar, though slightly less negative.<\/p>\n
When evaluating what this means below, it\u2019s important to recall the limitations of the data and model here:<\/p>\n