Time series forecasting can be challenging as there are many different methods you could use and many different hyperparameters for each method.<\/p>\n
The Prophet library is an open-source library designed for making forecasts for univariate time series datasets. It is easy to use and designed to automatically find a good set of hyperparameters for the model in an effort to make skillful forecasts for data with trends and seasonal structure by default.<\/p>\n
In this tutorial, you will discover how to use the Facebook Prophet library for time series forecasting.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Prophet is an open-source library developed by Facebook and designed for automatic forecasting of univariate time series data.<\/li>\n
- How to fit Prophet models and use them to make in-sample and out-of-sample forecasts.<\/li>\n
- How to evaluate a Prophet model on a hold-out dataset.<\/li>\n<\/ul>\n
Let’s get started.<\/p>\n
![\"Time](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/06\/Time-Series-Forecasting-With-Prophet-in-Python.jpg\")
<\/p>\nTime Series Forecasting With Prophet in Python
Photo by Rinaldo Wurglitsch<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into three parts; they are:<\/p>\n
- \n
- Prophet Forecasting Library<\/li>\n
- Car Sales Dataset\n
- \n
- Load and Summarize Dataset<\/li>\n
- Load and Plot Dataset<\/li>\n<\/ol>\n<\/li>\n
- Forecast Car Sales With Prophet\n
- \n
- Fit Prophet Model<\/li>\n
- Make an In-Sample Forecast<\/li>\n
- Make an Out-of-Sample Forecast<\/li>\n
- Manually Evaluate Forecast Model<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
Prophet Forecasting Library<\/h2>\n
Prophet<\/a>, or “Facebook Prophet<\/em>,” is an open-source library for univariate (one variable) time series forecasting developed by Facebook.<\/p>\n
Prophet implements what they refer to as an additive time series forecasting model<\/a>, and the implementation supports trends, seasonality, and holidays.<\/p>\n
\n
Implements a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects<\/p>\n<\/blockquote>\n
— Package ‘prophet’<\/a>, 2019.<\/p>\n
It is designed to be easy and completely automatic, e.g. point it at a time series and get a forecast. As such, it is intended for internal company use, such as forecasting sales, capacity, etc.<\/p>\n
For a great overview of Prophet and its capabilities, see the post:<\/p>\n
- \n
- Prophet: forecasting at scale<\/a>, 2017.<\/li>\n<\/ul>\n
The library provides two interfaces, including R and Python. We will focus on the Python interface in this tutorial.<\/p>\n
The first step is to install the Prophet library using Pip, as follows:<\/p>\n
sudo pip install fbprophet<\/pre>\n
Next, we can confirm that the library was installed correctly.<\/p>\n
To do this, we can import the library and print the version number in Python. The complete example is listed below.<\/p>\n
# check prophet version\r\nimport fbprophet\r\n# print version number\r\nprint('Prophet %s' % fbprophet.__version__)<\/pre>\nRunning the example prints the installed version of Prophet.<\/p>\n
You should have the same version or higher.<\/p>\n
Prophet 0.5<\/pre>\n
Now that we have Prophet installed, let’s select a dataset we can use to explore using the library.<\/p>\n
Car Sales Dataset<\/h2>\n
We will use the monthly car sales dataset.<\/p>\n
It is a standard univariate time series dataset that contains both a trend and seasonality. The dataset has 108 months of data and a naive persistence forecast can achieve a mean absolute error of about 3,235 sales, providing a lower error limit.<\/p>\n
No need to download the dataset as we will download it automatically as part of each example.<\/p>\n
- \n
- Monthly Car Sales Dataset (csv)<\/a><\/li>\n
- Monthly Car Sales Dataset Description<\/a><\/li>\n<\/ul>\n
Load and Summarize Dataset<\/h3>\n
First, let’s load and summarize the dataset.<\/p>\n
Prophet requires data to be in Pandas DataFrames. Therefore, we will load and summarize the data using Pandas.<\/p>\n
We can load the data directly from the URL by calling the read_csv() Pandas function<\/a>, then summarize the shape (number of rows and columns) of the data and view the first few rows of data.<\/p>\n
The complete example is listed below.<\/p>\n
# load the car sales dataset\r\nfrom pandas import read_csv\r\n# load data\r\npath = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/monthly-car-sales.csv'\r\ndf = read_csv(path, header=0)\r\n# summarize shape\r\nprint(df.shape)\r\n# show first few rows\r\nprint(df.head())<\/pre>\n
Running the example first reports the number of rows and columns, then lists the first five rows of data.<\/p>\n
We can see that as we expected, there are 108 months worth of data and two columns. The first column is the date and the second is the number of sales.<\/p>\n
Note that the first column in the output is a row index and is not a part of the dataset, just a helpful tool that Pandas uses to order rows.<\/p>\n
(108, 2)\r\n Month Sales\r\n0 1960-01 6550\r\n1 1960-02 8728\r\n2 1960-03 12026\r\n3 1960-04 14395\r\n4 1960-05 14587<\/pre>\n<\/p>\n
Load and Plot Dataset<\/h3>\n
A time-series dataset does not make sense to us until we plot it.<\/p>\n
Plotting a time series helps us actually see if there is a trend, a seasonal cycle, outliers, and more. It gives us a feel for the data.<\/p>\n
We can plot the data easily in Pandas by calling the plot()<\/em> function on the DataFrame.<\/p>\n
The complete example is listed below.<\/p>\n
# load and plot the car sales dataset\r\nfrom pandas import read_csv\r\nfrom matplotlib import pyplot\r\n# load data\r\npath = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/monthly-car-sales.csv'\r\ndf = read_csv(path, header=0)\r\n# plot the time series\r\ndf.plot()\r\npyplot.show()<\/pre>\n
Running the example creates a plot of the time series.<\/p>\n
We can clearly see the trend in sales over time and a monthly seasonal pattern to the sales. These are patterns we expect the forecast model to take into account.<\/p>\n
![\"Line](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Line-Plot-of-Car-Sales-Dataset.png\")
<\/p>\nLine Plot of Car Sales Dataset<\/p>\n<\/div>\n
Now that we are familiar with the dataset, let’s explore how we can use the Prophet library to make forecasts.<\/p>\n
Forecast Car Sales With Prophet<\/h2>\n
In this section, we will explore using the Prophet to forecast the car sales dataset.<\/p>\n
Let’s start by fitting a model on the dataset<\/p>\n
Fit Prophet Model<\/h3>\n
To use Prophet for forecasting, first, a Prophet()<\/em> object is defined and configured, then it is fit on the dataset by calling the fit()<\/em> function and passing the data.<\/p>\n
The Prophet()<\/em> object takes arguments to configure the type of model you want, such as the type of growth, the type of seasonality, and more. By default, the model will work hard to figure out almost everything automatically.<\/p>\n
The fit()<\/em> function takes a DataFrame<\/em> of time series data. The DataFrame<\/em> must have a specific format. The first column must have the name ‘ds<\/em>‘ and contain the date-times. The second column must have the name ‘y<\/em>‘ and contain the observations.<\/p>\n
This means we change the column names in the dataset. It also requires that the first column be converted to date-time objects, if they are not already (e.g. this can be down as part of loading the dataset with the right arguments to read_csv<\/em>).<\/p>\n
For example, we can modify our loaded car sales dataset to have this expected structure, as follows:<\/p>\n
...\r\n# prepare expected column names\r\ndf.columns = ['ds', 'y']\r\ndf['ds']= to_datetime(df['ds'])<\/pre>\n
The complete example of fitting a Prophet model on the car sales dataset is listed below.<\/p>\n
# fit prophet model on the car sales dataset\r\nfrom pandas import read_csv\r\nfrom pandas import to_datetime\r\nfrom fbprophet import Prophet\r\n# load data\r\npath = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/monthly-car-sales.csv'\r\ndf = read_csv(path, header=0)\r\n# prepare expected column names\r\ndf.columns = ['ds', 'y']\r\ndf['ds']= to_datetime(df['ds'])\r\n# define the model\r\nmodel = Prophet()\r\n# fit the model\r\nmodel.fit(df)<\/pre>\n
Running the example loads the dataset, prepares the DataFrame in the expected format, and fits a Prophet model.<\/p>\n
By default, the library provides a lot of verbose output during the fit process. I think it’s a bad idea in general as it trains developers to ignore output.<\/p>\n
Nevertheless, the output summarizes what happened during the model fitting process, specifically the optimization processes that ran.<\/p>\n
INFO:fbprophet:Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this.\r\nINFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.\r\nInitial log joint probability = -4.39613\r\n Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes\r\n 99 270.121 0.00413718 75.7289 1 1 120\r\n Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes\r\n 179 270.265 0.00019681 84.1622 2.169e-06 0.001 273 LS failed, Hessian reset\r\n 199 270.283 1.38947e-05 87.8642 0.3402 1 299\r\n Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes\r\n 240 270.296 1.6343e-05 89.9117 1.953e-07 0.001 381 LS failed, Hessian reset\r\n 299 270.3 4.73573e-08 74.9719 0.3914 1 455\r\n Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes\r\n 300 270.3 8.25604e-09 74.4478 0.3522 0.3522 456\r\nOptimization terminated normally:\r\n Convergence detected: absolute parameter change was below tolerance<\/pre>\n
I will not reproduce this output in subsequent sections when we fit the model.<\/p>\n
Next, let’s make a forecast.<\/p>\n
Make an In-Sample Forecast<\/h3>\n
It can be useful to make a forecast on historical data.<\/p>\n
That is, we can make a forecast on data used as input to train the model. Ideally, the model has seen the data before and would make a perfect prediction.<\/p>\n
Nevertheless, this is not the case as the model tries to generalize across all cases in the data.<\/p>\n
This is called making an in-sample (in training set sample) forecast and reviewing the results can give insight into how good the model is. That is, how well it learned the training data.<\/p>\n
A forecast is made by calling the predict()<\/em> function and passing a DataFrame<\/em> that contains one column named ‘ds<\/em>‘ and rows with date-times for all the intervals to be predicted.<\/p>\n
There are many ways to create this “forecast<\/em>” DataFrame<\/em>. In this case, we will loop over one year of dates, e.g. the last 12 months in the dataset, and create a string for each month. We will then convert the list of dates into a DataFrame<\/em> and convert the string values into date-time objects.<\/p>\n
...\r\n# define the period for which we want a prediction\r\nfuture = list()\r\nfor i in range(1, 13):\r\n\tdate = '1968-%02d' % i\r\n\tfuture.append([date])\r\nfuture = DataFrame(future)\r\nfuture.columns = ['ds']\r\nfuture['ds']= to_datetime(future['ds'])<\/pre>\n
This DataFrame<\/em> can then be provided to the predict()<\/em> function to calculate a forecast.<\/p>\n
The result of the predict() function is a DataFrame<\/em> that contains many columns. Perhaps the most important columns are the forecast date time (‘ds<\/em>‘), the forecasted value (‘yhat<\/em>‘), and the lower and upper bounds on the predicted value (‘yhat_lower<\/em>‘ and ‘yhat_upper<\/em>‘) that provide uncertainty of the forecast.<\/p>\n
For example, we can print the first few predictions as follows:<\/p>\n
...\r\n# summarize the forecast\r\nprint(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].head())<\/pre>\n
Prophet also provides a built-in tool for visualizing the prediction in the context of the training dataset.<\/p>\n
This can be achieved by calling the plot()<\/em> function on the model and passing it a result DataFrame. It will create a plot of the training dataset and overlay the prediction with the upper and lower bounds for the forecast dates.<\/p>\n
...\r\nprint(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].head())\r\n# plot forecast\r\nmodel.plot(forecast)\r\npyplot.show()<\/pre>\n
Tying this all together, a complete example of making an in-sample forecast is listed below.<\/p>\n
# make an in-sample forecast\r\nfrom pandas import read_csv\r\nfrom pandas import to_datetime\r\nfrom pandas import DataFrame\r\nfrom fbprophet import Prophet\r\nfrom matplotlib import pyplot\r\n# load data\r\npath = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/monthly-car-sales.csv'\r\ndf = read_csv(path, header=0)\r\n# prepare expected column names\r\ndf.columns = ['ds', 'y']\r\ndf['ds']= to_datetime(df['ds'])\r\n# define the model\r\nmodel = Prophet()\r\n# fit the model\r\nmodel.fit(df)\r\n# define the period for which we want a prediction\r\nfuture = list()\r\nfor i in range(1, 13):\r\n\tdate = '1968-%02d' % i\r\n\tfuture.append([date])\r\nfuture = DataFrame(future)\r\nfuture.columns = ['ds']\r\nfuture['ds']= to_datetime(future['ds'])\r\n# use the model to make a forecast\r\nforecast = model.predict(future)\r\n# summarize the forecast\r\nprint(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].head())\r\n# plot forecast\r\nmodel.plot(forecast)\r\npyplot.show()<\/pre>\n
Running the example forecasts the last 12 months of the dataset.<\/p>\n
The first five months of the prediction are reported and we can see that values are not too different from the actual sales values in the dataset.<\/p>\n
ds yhat yhat_lower yhat_upper\r\n0 1968-01-01 14364.866157 12816.266184 15956.555409\r\n1 1968-02-01 14940.687225 13299.473640 16463.811658\r\n2 1968-03-01 20858.282598 19439.403787 22345.747821\r\n3 1968-04-01 22893.610396 21417.399440 24454.642588\r\n4 1968-05-01 24212.079727 22667.146433 25816.191457<\/pre>\n
Next, a plot is created. We can see the training data are represented as black dots and the forecast is a blue line with upper and lower bounds in a blue shaded area.<\/p>\n
We can see that the forecasted 12 months is a good match for the real observations, especially when the bounds are taken into account.<\/p>\n
![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Plot-of-Time-Series-and-In-Sample-Forecast-With-Prophet.png\")
<\/p>\nPlot of Time Series and In-Sample Forecast With Prophet<\/p>\n<\/div>\n
Make an Out-of-Sample Forecast<\/h3>\n
In practice, we really want a forecast model to make a prediction beyond the training data.<\/p>\n
This is called an out-of-sample forecast.<\/p>\n
We can achieve this in the same way as an in-sample forecast and simply specify a different forecast period.<\/p>\n
In this case, a period beyond the end of the training dataset, starting 1969-01.<\/p>\n
...\r\n# define the period for which we want a prediction\r\nfuture = list()\r\nfor i in range(1, 13):\r\n\tdate = '1969-%02d' % i\r\n\tfuture.append([date])\r\nfuture = DataFrame(future)\r\nfuture.columns = ['ds']\r\nfuture['ds']= to_datetime(future['ds'])<\/pre>\n
Tying this together, the complete example is listed below.<\/p>\n
# make an out-of-sample forecast\r\nfrom pandas import read_csv\r\nfrom pandas import to_datetime\r\nfrom pandas import DataFrame\r\nfrom fbprophet import Prophet\r\nfrom matplotlib import pyplot\r\n# load data\r\npath = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/monthly-car-sales.csv'\r\ndf = read_csv(path, header=0)\r\n# prepare expected column names\r\ndf.columns = ['ds', 'y']\r\ndf['ds']= to_datetime(df['ds'])\r\n# define the model\r\nmodel = Prophet()\r\n# fit the model\r\nmodel.fit(df)\r\n# define the period for which we want a prediction\r\nfuture = list()\r\nfor i in range(1, 13):\r\n\tdate = '1969-%02d' % i\r\n\tfuture.append([date])\r\nfuture = DataFrame(future)\r\nfuture.columns = ['ds']\r\nfuture['ds']= to_datetime(future['ds'])\r\n# use the model to make a forecast\r\nforecast = model.predict(future)\r\n# summarize the forecast\r\nprint(forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].head())\r\n# plot forecast\r\nmodel.plot(forecast)\r\npyplot.show()<\/pre>\n
Running the example makes an out-of-sample forecast for the car sales data.<\/p>\n
The first five rows of the forecast are printed, although it is hard to get an idea of whether they are sensible or not.<\/p>\n
ds yhat yhat_lower yhat_upper\r\n0 1969-01-01 15406.401318 13751.534121 16789.969780\r\n1 1969-02-01 16165.737458 14486.887740 17634.953132\r\n2 1969-03-01 21384.120631 19738.950363 22926.857539\r\n3 1969-04-01 23512.464086 21939.204670 25105.341478\r\n4 1969-05-01 25026.039276 23544.081762 26718.820580<\/pre>\n
A plot is created to help us evaluate the prediction in the context of the training data.<\/p>\n
The new one-year forecast does look sensible, at least by eye.<\/p>\n
![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Plot-of-Time-Series-and-Out-Of-Sample-Forecast-With-Prophet.png\")
<\/p>\nPlot of Time Series and Out-of-Sample Forecast With Prophet<\/p>\n<\/div>\n
Manually Evaluate Forecast Model<\/h3>\n
It is critical to develop an objective estimate of a forecast model’s performance.<\/p>\n
This can be achieved by holding some data back from the model, such as the last 12 months. Then, fitting the model on the first portion of the data, using it to make predictions on the held-pack portion, and calculating an error measure, such as the mean absolute error across the forecasts. E.g. a simulated out-of-sample forecast.<\/p>\n
The score gives an estimate of how well we might expect the model to perform on average when making an out-of-sample forecast.<\/p>\n
We can do this with the samples data by creating a new DataFrame<\/em> for training with the last 12 months removed.<\/p>\n
...\r\n# create test dataset, remove last 12 months\r\ntrain = df.drop(df.index[-12:])\r\nprint(train.tail())<\/pre>\n
A forecast can then be made on the last 12 months of date-times.<\/p>\n
We can then retrieve the forecast values and the expected values from the original dataset and calculate a mean absolute error metric using the scikit-learn library.<\/p>\n
...\r\n# calculate MAE between expected and predicted values for december\r\ny_true = df['y'][-12:].values\r\ny_pred = forecast['yhat'].values\r\nmae = mean_absolute_error(y_true, y_pred)\r\nprint('MAE: %.3f' % mae)<\/pre>\nIt can also be helpful to plot the expected vs. predicted values to see how well the out-of-sample prediction matches the known values.<\/p>\n
...\r\n# plot expected vs actual\r\npyplot.plot(y_true, label='Actual')\r\npyplot.plot(y_pred, label='Predicted')\r\npyplot.legend()\r\npyplot.show()<\/pre>\n
Tying this together, the example below demonstrates how to evaluate a Prophet model on a hold-out dataset.<\/p>\n
# evaluate prophet time series forecasting model on hold out dataset\r\nfrom pandas import read_csv\r\nfrom pandas import to_datetime\r\nfrom pandas import DataFrame\r\nfrom fbprophet import Prophet\r\nfrom sklearn.metrics import mean_absolute_error\r\nfrom matplotlib import pyplot\r\n# load data\r\npath = 'https:\/\/raw.githubusercontent.com\/jbrownlee\/Datasets\/master\/monthly-car-sales.csv'\r\ndf = read_csv(path, header=0)\r\n# prepare expected column names\r\ndf.columns = ['ds', 'y']\r\ndf['ds']= to_datetime(df['ds'])\r\n# create test dataset, remove last 12 months\r\ntrain = df.drop(df.index[-12:])\r\nprint(train.tail())\r\n# define the model\r\nmodel = Prophet()\r\n# fit the model\r\nmodel.fit(train)\r\n# define the period for which we want a prediction\r\nfuture = list()\r\nfor i in range(1, 13):\r\n\tdate = '1968-%02d' % i\r\n\tfuture.append([date])\r\nfuture = DataFrame(future)\r\nfuture.columns = ['ds']\r\nfuture['ds'] = to_datetime(future['ds'])\r\n# use the model to make a forecast\r\nforecast = model.predict(future)\r\n# calculate MAE between expected and predicted values for december\r\ny_true = df['y'][-12:].values\r\ny_pred = forecast['yhat'].values\r\nmae = mean_absolute_error(y_true, y_pred)\r\nprint('MAE: %.3f' % mae)\r\n# plot expected vs actual\r\npyplot.plot(y_true, label='Actual')\r\npyplot.plot(y_pred, label='Predicted')\r\npyplot.legend()\r\npyplot.show()<\/pre>\nRunning the example first reports the last few rows of the training dataset.<\/p>\n
It confirms the training ends in the last month of 1967 and 1968 will be used as the hold-out dataset.<\/p>\n
ds y\r\n91 1967-08-01 13434\r\n92 1967-09-01 13598\r\n93 1967-10-01 17187\r\n94 1967-11-01 16119\r\n95 1967-12-01 13713<\/pre>\n
Next, a mean absolute error is calculated for the forecast period.<\/p>\n
In this case we can see that the error is approximately 1,336 sales, which is much lower (better) than a naive persistence model that achieves an error of 3,235 sales over the same period.<\/p>\n
MAE: 1336.814<\/pre>\n
Finally, a plot is created comparing the actual vs. predicted values. In this case, we can see that the forecast is a good fit. The model has skill and forecast that looks sensible.<\/p>\n
![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/02\/Plot-of-Actual-vs.-Predicted-Values-for-Last-12-Months-of-Car-Sales.png\")
<\/p>\nPlot of Actual vs. Predicted Values for Last 12 Months of Car Sales<\/p>\n<\/div>\n
The Prophet library also provides tools to automatically evaluate models and plot results, although those tools don’t appear to work well with data above one day in resolution.<\/p>\n
Further Reading<\/h2>\n
This section provides more resources on the topic if you are looking to go deeper.<\/p>\n
- \n
- Prophet Homepage<\/a>.<\/li>\n
- Prophet GitHub Project<\/a>.<\/li>\n
- Prophet API Documentation<\/a>.<\/li>\n
- Prophet: forecasting at scale<\/a>, 2017.<\/li>\n
- Forecasting at scale<\/a>, 2017.<\/li>\n
- Car Sales Dataset<\/a>.<\/li>\n
- Package ‘prophet’, R Documentation<\/a>.<\/li>\n<\/ul>\n
Summary<\/h2>\n
In this tutorial, you discovered how to use the Facebook Prophet library for time series forecasting.<\/p>\n
Specifically, you learned:<\/p>\n
- \n
- Prophet is an open-source library developed by Facebook and designed for automatic forecasting of univariate time series data.<\/li>\n
- How to fit Prophet models and use them to make in-sample and out-of-sample forecasts.<\/li>\n
- How to evaluate a Prophet model on a hold-out dataset.<\/li>\n<\/ul>\n
Do you have any questions?<\/strong>
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\nThe post Time Series Forecasting With Prophet in Python<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
- Prophet GitHub Project<\/a>.<\/li>\n
- Monthly Car Sales Dataset Description<\/a><\/li>\n<\/ul>\n
- Monthly Car Sales Dataset (csv)<\/a><\/li>\n
- Prophet: forecasting at scale<\/a>, 2017.<\/li>\n<\/ul>\n