![](\"https:\/\/images.unsplash.com\/photo-1493946740644-2d8a1f1a6aff?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=800&q=60&profile=RESIZE_710x\")
<\/a><\/em><\/h3>\nSource<\/a><\/em><\/h3>\nComprehensive and Comparative List of Feature Store Architectures for Data Scientists and Big Data Professionals<\/em><\/h3>\n<\/div>\n\n\n<\/div>\n
\nIntroduction & Motivation – Why Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\nFeature store has become an important unit of organizations developing predictive services across any industry domain. Some of the earlier challenges in deploying ML solutions at scale involves :<\/p>\n<\/div>\n
\n\n- Developing<\/strong> and maintaining customized systems<\/strong> by individual teams with little or no coordination.<\/li>\n
- No collaborative<\/strong> system for sharing features for similar type ML models (models from a similar domain or models addressing. same business use-cases or customer domains).<\/li>\n
- Increased cognitive burden<\/strong> without the proper scope of scalability<\/li>\n
- Limited integration<\/strong> with big-data ecosystems.<\/li>\n
- Limited scope for model retraining, comparison, model governance, and traceability<\/strong>, limiting agile development life-cycle.<\/li>\n
- Difficult to track and retrain model<\/strong> which exhibits seasonality<\/li>\n<\/ul>\n<\/div>\n\n
To overcome the above limitations, Architects. Data scientists, Big Data, and Analytics professionals have felt the necessity to walk under one roof with one unified framework to facilitate easier collaboration, sharing of data, results, reports.<\/p>\n<\/div>\n
\nDepartments, teams and organizations shared some of the similar notions of Feature Engineering:<\/p>\n<\/div>\n
\n\n- Feature Engineering is expensive<\/strong> and amortization<\/strong> happens over time and across models.<\/li>\n
- The increase in cost is non-linear\/exponential with the increase<\/strong> in the number of features.<\/li>\n
- Triggers\/Alerts due to addition\/removal of feature is high.<\/li>\n
- Most often dependencies are not documented\/tracked<\/strong> which results in an increase<\/strong> of implicit<\/strong> and explicit dependencies<\/strong> getting added over time.<\/li>\n<\/ul>\n<\/div>\n\n
While sharing a similar opinion, it became easier to come together and create a Unified Framework<\/strong> called Feature Store.<\/strong> This would enhance the speed of ML model deployment life-cycle<\/strong> along with the creation of proper documents, required version analysis,<\/strong> and model performance<\/strong> in order to save time and effort.<\/p>\n<\/div>\n\nIn this blog, we highlight on the features supported by different Feature Store frameworks, that are primarily developed by different leading industry giants.<\/p>\n
\n<\/div>\n
\nAdvantages of Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n- Ability to re-use and discover features<\/strong> between teams across the organization.<\/li>\n
- Features should be governed<\/strong> by adding features like access control and versioning<\/strong>.<\/li>\n
- Ability to precompute<\/strong> and automatically backfill features<\/strong> — including online computation and offline aggregation<\/li>\n
- Helping to create a collaborative environment<\/strong> between data scientists and big data engineers<\/li>\n
- Save effort<\/strong> and cost<\/strong> by sharing not only features but also related artifacts<\/strong>, documents, marketing insights of models developed from these features.<\/li>\n
- Enable consistency<\/strong> between training and serving.<\/li>\n<\/ul>\n<\/div>\n\n
Michaelengelo From Uber<\/strong><\/span><\/h2>\n<\/div>\n\n\n![\"This](\"https:\/\/techairesearch.com\/wp-content\/uploads\/2020\/08\/image-2-1024x574.png\")
<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nMichaelangelo<\/strong> – a framework developed by Uber<\/strong> that allows feature integration\/joining in both offline and online pipelines. Here Hive (Offline)<\/strong> and Cassandra (Online)<\/strong> acts as the main storage unit for raw\/transformed features. It provides a horizontally scalable multi-tenant<\/strong> architecture for multiple models with suitable scaling and monitoring. Training jobs can be configured and managed through a web UI or an API, via Jupyter notebook.<\/a><\/p>\n<\/div>\n\nIt further provides options to define hierarchical partitioning schema to train models per partition, that can be deployed as a single logical model. This provides easy bootstrapping<\/strong> and helps to overcome challenges when several models need to be trained based on the hierarchical structure of the data.<\/p>\n<\/div>\n\nAt runtime during serving, it finds root to the best model for each node. Further its best known for its ability to support continuous learning, providing integration with AutoML<\/strong>, along with its support for distributed deep learning<\/strong>.<\/p>\n\n<\/div>\n
\nFeast Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n![\"This](\"https:\/\/techairesearch.com\/wp-content\/uploads\/2020\/08\/image-1024x378.png\")
<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nGoogle released Feast<\/strong> which is primarily built around Google Cloud services: Big Query (offline) and Big Table (online) and Redis (low-latency)<\/strong>, with Apache Beam<\/strong> for feature engineering. It allows a clear separation between big data and model development. This online predictive service allows feature sharing among teams with strong consistency<\/strong> between model training and serving.<\/p>\n<\/div>\n\nFurther Feast comes with centralized feature management, discovery, feature validation,<\/strong> and feature aggregation. The feature columns reside inside wide-entity tables. In addition, the composite entities separate individual features.<\/p>\n<\/div>\n\n<\/h2>\nWix Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n![\"This](\"https:\/\/techairesearch.com\/wp-content\/uploads\/2020\/08\/image-7-1024x501.png\")
<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nWix<\/strong> provides a platform for feature-sharing across different ML models for both batch<\/strong> and real-time<\/strong> datasets. It supports a pre-configured set of feature families on the site and user-level for both training and serving models. The different stages of data management, model training and deployment are marked and show in the figure above. It further uses S3<\/strong> to store real-time extracted features.<\/p>\n\n<\/div>\n
\nFeatureStore from Comcast<\/strong><\/span><\/h2>\n<\/div>\n\n\n![\"This](\"https:\/\/i0.wp.com\/techairesearch.com\/wp-content\/uploads\/2020\/08\/image-1.png?fit=525,286&ssl=1\")
<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe Feature Store<\/strong> developed by Comcast<\/strong> helps data scientists to reuse versioned features, upload online (real-time)\/streaming data, and review feature metrics<\/strong> by models. The product is available in multiple pluggable feature store components. The built-in model repository contains artifacts related to data pre-processing (normalization, scaling)<\/strong> displaying the required mapping to the features needed to execute the model. Further, the architecture is built using Spark on Alluxio (open source data orchestration layer that brings data close to compute for big data and AI\/ML workloads in the cloud), S3, HDFS, RDBMS, Kafka, Kinesis<\/strong>. The Model deployment with Kubeflow<\/strong> helps to build a resilient, highly available distributed systems with support for rate-limiting, shadow deployments, and auto-scaling<\/strong>.<\/p>\n<\/div>\n\nThe integration with Data Lake with suitable API s helps data scientists to use SQL and create training\/validation\/test datasets<\/strong> that can be versioned<\/strong> and integrated into the full model pipeline. In addition, the framework comes with the support of Seldon Inference Graphs for A\/B Testing, Ensembles, Multi-armed bandits, Custom combinations<\/strong>. The end to end system not only provides traceability from use-case, models, features, model to features mapping, versioned datasets, model training codebase, model deployment containers, and prediction\/outcome sinks, it is also known for integration with Feature-Store, Container Repository, and Git to integrate data, code and run-time artifacts for CI\/CD integration<\/strong>.<\/p>\n<\/div>\n\nJust like any other architecture, it has continuous Feature Aggregation on streaming data + on-demand<\/strong> features. The Online Feature Store uses the following sequences before giving a prediction:<\/p>\n<\/div>\n\n\n- Payload only contains Model Name & Account Number<\/li>\n
- Model Metadata informs which features are needed for the model<\/li>\n
- Pull required by features by Account Number<\/li>\n
- Pass a full set of assembled features for model execution<\/li>\n<\/ul>\n<\/div>\n\n\n
![\"This](\"https:\/\/techairesearch.com\/wp-content\/uploads\/2020\/08\/Screenshot-2020-08-02-at-1.01.55-AM-1024x857.png\")
<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nHopWorks Enterprise Edition<\/a> is a multi-tenant architecture that integrates AWS Sagemaker, Databricks, Kubernetes, and Jupyter Notebook. It also supports integration with Authentication frameworks like LDAP, Kerberos, and Oauth2<\/strong>.<\/p>\n<\/div>\n\nThe Batch \/ Live Streaming functionality is facilitated by Apache Beam, Apache Flink, and Apache Spark,<\/strong> whereas the model governance and monitoring pipeline are built using Kafka and Spark Streaming.<\/p>\n<\/div>\n\nThe architecture is composed of several building blocks namely<\/p>\n<\/div>\n
\n\n- The Feature Store API – For reading\/writing to\/from the feature store<\/li>\n
- The Feature Store Registry – User-Interface to discover features<\/li>\n
- Feature Metadata – Documentation, Analysis and Versioning<\/li>\n
- Feature Engineering Job – For computation<\/li>\n
- Storage Layer – For feature storage<\/li>\n<\/ul>\n<\/div>\n\n
Netflix Feature Store<\/strong><\/a><\/span><\/h2>\n<\/div>\n\n\n![\"This](\"https:\/\/techairesearch.com\/wp-content\/uploads\/2020\/08\/image-3-1024x596.png\")
<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\n\n![\"This](\"https:\/\/techairesearch.com\/wp-content\/uploads\/2020\/08\/image-4-1024x482.png\")
<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe feature store developed by Netflix supports both online and offline model training and development. The online micro-services enables the framework to collect the data elements required by the feature encoders in a model. It further passes this downstream for future use by offline predictions. The Fact Logging<\/strong> service of Netflix logs user-related, video-related, and computation specific features<\/strong> in a serialized<\/strong> format in appropriate storage units (S3<\/strong>).<\/p>\n<\/div>\n\nThe unique point of this architecture is the presence of components that help to:<\/p>\n<\/div>\n
\n\n- Develop\/Create contexts to snapshot<\/li>\n
- Snapshot data of various micro-services for the selected context<\/li>\n
- Build APIs to serve this data for a given time coordinate in the past<\/li>\n<\/ul>\n<\/div>\n\n
As snapshotting data for all contexts (e.g all member profiles, devices, times of day) would incur overhead and cost, Netflix relies on selecting samples of contexts to snapshot<\/strong> periodically (at regular intervals – daily\/twice daily), though different algorithms. It achieves this through Spark, by training data on different distributions, and by using stratified samples<\/strong> based on properties such as viewing patterns, devices, time spent on the service, region, etc.<\/p>\n<\/div>\n\nNetflix embraces a fine-grained Service Oriented Architecture for cloud-based deployment model.<\/p>\n<\/div>\n
\nFBLearner from Facebook<\/strong><\/span><\/h2>\n<\/div>\n\n\n![\"This](\"https:\/\/techairesearch.com\/wp-content\/uploads\/2020\/07\/Screenshot-2020-07-27-at-4.44.02-PM-1024x418.png\")
<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe FBLearner designed by Facebook is a framework for AI WorkFlow<\/strong> with Model Management and Deployment<\/strong>. It is mainly composed of 3 components – FB Learner Feature Store (runs on CPU), FB Learner Flow (runs on CPU +GPU), and FB Learner Predictor (runs on CPU). I<\/strong>t supports building all kinds of deep learning models (Caffe2, Pytorch, Tensorflow, MxNet, CNTK)<\/strong> and models can be stored in ONNX format (standardizes portability<\/a> across converters, runtimes, compilers, and visualizers. supports and to) across different hardware\/software platforms.<\/p>\n<\/div>\n\nThe above broad categories can be seen as creating logical units from hardware to application software.<\/p>\n<\/div>\n
\n\n- Frameworks<\/strong> (FB Learner Feature Store)<\/strong> needed to create, migrate and train models<\/li>\n
- Platforms<\/strong>
Comprehensive and Comparative List of Feature Store Architectures for Data Scientists and Big Data Professionals<\/em><\/h3>\n<\/div>\n\n\n<\/div>\n
\nIntroduction & Motivation – Why Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\nFeature store has become an important unit of organizations developing predictive services across any industry domain. Some of the earlier challenges in deploying ML solutions at scale involves :<\/p>\n<\/div>\n
\n\n- Developing<\/strong> and maintaining customized systems<\/strong> by individual teams with little or no coordination.<\/li>\n
- No collaborative<\/strong> system for sharing features for similar type ML models (models from a similar domain or models addressing. same business use-cases or customer domains).<\/li>\n
- Increased cognitive burden<\/strong> without the proper scope of scalability<\/li>\n
- Limited integration<\/strong> with big-data ecosystems.<\/li>\n
- Limited scope for model retraining, comparison, model governance, and traceability<\/strong>, limiting agile development life-cycle.<\/li>\n
- Difficult to track and retrain model<\/strong> which exhibits seasonality<\/li>\n<\/ul>\n<\/div>\n\n
To overcome the above limitations, Architects. Data scientists, Big Data, and Analytics professionals have felt the necessity to walk under one roof with one unified framework to facilitate easier collaboration, sharing of data, results, reports.<\/p>\n<\/div>\n
\nDepartments, teams and organizations shared some of the similar notions of Feature Engineering:<\/p>\n<\/div>\n
\n\n- Feature Engineering is expensive<\/strong> and amortization<\/strong> happens over time and across models.<\/li>\n
- The increase in cost is non-linear\/exponential with the increase<\/strong> in the number of features.<\/li>\n
- Triggers\/Alerts due to addition\/removal of feature is high.<\/li>\n
- Most often dependencies are not documented\/tracked<\/strong> which results in an increase<\/strong> of implicit<\/strong> and explicit dependencies<\/strong> getting added over time.<\/li>\n<\/ul>\n<\/div>\n\n
While sharing a similar opinion, it became easier to come together and create a Unified Framework<\/strong> called Feature Store.<\/strong> This would enhance the speed of ML model deployment life-cycle<\/strong> along with the creation of proper documents, required version analysis,<\/strong> and model performance<\/strong> in order to save time and effort.<\/p>\n<\/div>\n\nIn this blog, we highlight on the features supported by different Feature Store frameworks, that are primarily developed by different leading industry giants.<\/p>\n
\n<\/div>\n
\nAdvantages of Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n- Ability to re-use and discover features<\/strong> between teams across the organization.<\/li>\n
- Features should be governed<\/strong> by adding features like access control and versioning<\/strong>.<\/li>\n
- Ability to precompute<\/strong> and automatically backfill features<\/strong> — including online computation and offline aggregation<\/li>\n
- Helping to create a collaborative environment<\/strong> between data scientists and big data engineers<\/li>\n
- Save effort<\/strong> and cost<\/strong> by sharing not only features but also related artifacts<\/strong>, documents, marketing insights of models developed from these features.<\/li>\n
- Enable consistency<\/strong> between training and serving.<\/li>\n<\/ul>\n<\/div>\n\n
Michaelengelo From Uber<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nMichaelangelo<\/strong> – a framework developed by Uber<\/strong> that allows feature integration\/joining in both offline and online pipelines. Here Hive (Offline)<\/strong> and Cassandra (Online)<\/strong> acts as the main storage unit for raw\/transformed features. It provides a horizontally scalable multi-tenant<\/strong> architecture for multiple models with suitable scaling and monitoring. Training jobs can be configured and managed through a web UI or an API, via Jupyter notebook.<\/a><\/p>\n<\/div>\n\nIt further provides options to define hierarchical partitioning schema to train models per partition, that can be deployed as a single logical model. This provides easy bootstrapping<\/strong> and helps to overcome challenges when several models need to be trained based on the hierarchical structure of the data.<\/p>\n<\/div>\n\nAt runtime during serving, it finds root to the best model for each node. Further its best known for its ability to support continuous learning, providing integration with AutoML<\/strong>, along with its support for distributed deep learning<\/strong>.<\/p>\n\n<\/div>\n
\nFeast Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nGoogle released Feast<\/strong> which is primarily built around Google Cloud services: Big Query (offline) and Big Table (online) and Redis (low-latency)<\/strong>, with Apache Beam<\/strong> for feature engineering. It allows a clear separation between big data and model development. This online predictive service allows feature sharing among teams with strong consistency<\/strong> between model training and serving.<\/p>\n<\/div>\n\nFurther Feast comes with centralized feature management, discovery, feature validation,<\/strong> and feature aggregation. The feature columns reside inside wide-entity tables. In addition, the composite entities separate individual features.<\/p>\n<\/div>\n\n<\/h2>\nWix Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nWix<\/strong> provides a platform for feature-sharing across different ML models for both batch<\/strong> and real-time<\/strong> datasets. It supports a pre-configured set of feature families on the site and user-level for both training and serving models. The different stages of data management, model training and deployment are marked and show in the figure above. It further uses S3<\/strong> to store real-time extracted features.<\/p>\n\n<\/div>\n
\nFeatureStore from Comcast<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe Feature Store<\/strong> developed by Comcast<\/strong> helps data scientists to reuse versioned features, upload online (real-time)\/streaming data, and review feature metrics<\/strong> by models. The product is available in multiple pluggable feature store components. The built-in model repository contains artifacts related to data pre-processing (normalization, scaling)<\/strong> displaying the required mapping to the features needed to execute the model. Further, the architecture is built using Spark on Alluxio (open source data orchestration layer that brings data close to compute for big data and AI\/ML workloads in the cloud), S3, HDFS, RDBMS, Kafka, Kinesis<\/strong>. The Model deployment with Kubeflow<\/strong> helps to build a resilient, highly available distributed systems with support for rate-limiting, shadow deployments, and auto-scaling<\/strong>.<\/p>\n<\/div>\n\nThe integration with Data Lake with suitable API s helps data scientists to use SQL and create training\/validation\/test datasets<\/strong> that can be versioned<\/strong> and integrated into the full model pipeline. In addition, the framework comes with the support of Seldon Inference Graphs for A\/B Testing, Ensembles, Multi-armed bandits, Custom combinations<\/strong>. The end to end system not only provides traceability from use-case, models, features, model to features mapping, versioned datasets, model training codebase, model deployment containers, and prediction\/outcome sinks, it is also known for integration with Feature-Store, Container Repository, and Git to integrate data, code and run-time artifacts for CI\/CD integration<\/strong>.<\/p>\n<\/div>\n\nJust like any other architecture, it has continuous Feature Aggregation on streaming data + on-demand<\/strong> features. The Online Feature Store uses the following sequences before giving a prediction:<\/p>\n<\/div>\n\n\n- Payload only contains Model Name & Account Number<\/li>\n
- Model Metadata informs which features are needed for the model<\/li>\n
- Pull required by features by Account Number<\/li>\n
- Pass a full set of assembled features for model execution<\/li>\n<\/ul>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\n
Source<\/a><\/p>\n<\/div>\n<\/div>\n\nHopWorks Enterprise Edition<\/a> is a multi-tenant architecture that integrates AWS Sagemaker, Databricks, Kubernetes, and Jupyter Notebook. It also supports integration with Authentication frameworks like LDAP, Kerberos, and Oauth2<\/strong>.<\/p>\n<\/div>\n\nThe Batch \/ Live Streaming functionality is facilitated by Apache Beam, Apache Flink, and Apache Spark,<\/strong> whereas the model governance and monitoring pipeline are built using Kafka and Spark Streaming.<\/p>\n<\/div>\n\nThe architecture is composed of several building blocks namely<\/p>\n<\/div>\n
\n\n- The Feature Store API – For reading\/writing to\/from the feature store<\/li>\n
- The Feature Store Registry – User-Interface to discover features<\/li>\n
- Feature Metadata – Documentation, Analysis and Versioning<\/li>\n
- Feature Engineering Job – For computation<\/li>\n
- Storage Layer – For feature storage<\/li>\n<\/ul>\n<\/div>\n\n
Netflix Feature Store<\/strong><\/a><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe feature store developed by Netflix supports both online and offline model training and development. The online micro-services enables the framework to collect the data elements required by the feature encoders in a model. It further passes this downstream for future use by offline predictions. The Fact Logging<\/strong> service of Netflix logs user-related, video-related, and computation specific features<\/strong> in a serialized<\/strong> format in appropriate storage units (S3<\/strong>).<\/p>\n<\/div>\n\nThe unique point of this architecture is the presence of components that help to:<\/p>\n<\/div>\n
\n\n- Develop\/Create contexts to snapshot<\/li>\n
- Snapshot data of various micro-services for the selected context<\/li>\n
- Build APIs to serve this data for a given time coordinate in the past<\/li>\n<\/ul>\n<\/div>\n\n
As snapshotting data for all contexts (e.g all member profiles, devices, times of day) would incur overhead and cost, Netflix relies on selecting samples of contexts to snapshot<\/strong> periodically (at regular intervals – daily\/twice daily), though different algorithms. It achieves this through Spark, by training data on different distributions, and by using stratified samples<\/strong> based on properties such as viewing patterns, devices, time spent on the service, region, etc.<\/p>\n<\/div>\n\nNetflix embraces a fine-grained Service Oriented Architecture for cloud-based deployment model.<\/p>\n<\/div>\n
\nFBLearner from Facebook<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe FBLearner designed by Facebook is a framework for AI WorkFlow<\/strong> with Model Management and Deployment<\/strong>. It is mainly composed of 3 components – FB Learner Feature Store (runs on CPU), FB Learner Flow (runs on CPU +GPU), and FB Learner Predictor (runs on CPU). I<\/strong>t supports building all kinds of deep learning models (Caffe2, Pytorch, Tensorflow, MxNet, CNTK)<\/strong> and models can be stored in ONNX format (standardizes portability<\/a> across converters, runtimes, compilers, and visualizers. supports and to) across different hardware\/software platforms.<\/p>\n<\/div>\n\nThe above broad categories can be seen as creating logical units from hardware to application software.<\/p>\n<\/div>\n
\n\n- Frameworks<\/strong> (FB Learner Feature Store)<\/strong> needed to create, migrate and train models<\/li>\n
- Platforms<\/strong>
\n<\/div>\n
Introduction & Motivation – Why Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\nFeature store has become an important unit of organizations developing predictive services across any industry domain. Some of the earlier challenges in deploying ML solutions at scale involves :<\/p>\n<\/div>\n
\n\n- Developing<\/strong> and maintaining customized systems<\/strong> by individual teams with little or no coordination.<\/li>\n
- No collaborative<\/strong> system for sharing features for similar type ML models (models from a similar domain or models addressing. same business use-cases or customer domains).<\/li>\n
- Increased cognitive burden<\/strong> without the proper scope of scalability<\/li>\n
- Limited integration<\/strong> with big-data ecosystems.<\/li>\n
- Limited scope for model retraining, comparison, model governance, and traceability<\/strong>, limiting agile development life-cycle.<\/li>\n
- Difficult to track and retrain model<\/strong> which exhibits seasonality<\/li>\n<\/ul>\n<\/div>\n\n
To overcome the above limitations, Architects. Data scientists, Big Data, and Analytics professionals have felt the necessity to walk under one roof with one unified framework to facilitate easier collaboration, sharing of data, results, reports.<\/p>\n<\/div>\n
\nDepartments, teams and organizations shared some of the similar notions of Feature Engineering:<\/p>\n<\/div>\n
\n\n- Feature Engineering is expensive<\/strong> and amortization<\/strong> happens over time and across models.<\/li>\n
- The increase in cost is non-linear\/exponential with the increase<\/strong> in the number of features.<\/li>\n
- Triggers\/Alerts due to addition\/removal of feature is high.<\/li>\n
- Most often dependencies are not documented\/tracked<\/strong> which results in an increase<\/strong> of implicit<\/strong> and explicit dependencies<\/strong> getting added over time.<\/li>\n<\/ul>\n<\/div>\n\n
While sharing a similar opinion, it became easier to come together and create a Unified Framework<\/strong> called Feature Store.<\/strong> This would enhance the speed of ML model deployment life-cycle<\/strong> along with the creation of proper documents, required version analysis,<\/strong> and model performance<\/strong> in order to save time and effort.<\/p>\n<\/div>\n\nIn this blog, we highlight on the features supported by different Feature Store frameworks, that are primarily developed by different leading industry giants.<\/p>\n
\n<\/div>\n
\nAdvantages of Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n- Ability to re-use and discover features<\/strong> between teams across the organization.<\/li>\n
- Features should be governed<\/strong> by adding features like access control and versioning<\/strong>.<\/li>\n
- Ability to precompute<\/strong> and automatically backfill features<\/strong> — including online computation and offline aggregation<\/li>\n
- Helping to create a collaborative environment<\/strong> between data scientists and big data engineers<\/li>\n
- Save effort<\/strong> and cost<\/strong> by sharing not only features but also related artifacts<\/strong>, documents, marketing insights of models developed from these features.<\/li>\n
- Enable consistency<\/strong> between training and serving.<\/li>\n<\/ul>\n<\/div>\n\n
Michaelengelo From Uber<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nMichaelangelo<\/strong> – a framework developed by Uber<\/strong> that allows feature integration\/joining in both offline and online pipelines. Here Hive (Offline)<\/strong> and Cassandra (Online)<\/strong> acts as the main storage unit for raw\/transformed features. It provides a horizontally scalable multi-tenant<\/strong> architecture for multiple models with suitable scaling and monitoring. Training jobs can be configured and managed through a web UI or an API, via Jupyter notebook.<\/a><\/p>\n<\/div>\n\nIt further provides options to define hierarchical partitioning schema to train models per partition, that can be deployed as a single logical model. This provides easy bootstrapping<\/strong> and helps to overcome challenges when several models need to be trained based on the hierarchical structure of the data.<\/p>\n<\/div>\n\nAt runtime during serving, it finds root to the best model for each node. Further its best known for its ability to support continuous learning, providing integration with AutoML<\/strong>, along with its support for distributed deep learning<\/strong>.<\/p>\n\n<\/div>\n
\nFeast Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nGoogle released Feast<\/strong> which is primarily built around Google Cloud services: Big Query (offline) and Big Table (online) and Redis (low-latency)<\/strong>, with Apache Beam<\/strong> for feature engineering. It allows a clear separation between big data and model development. This online predictive service allows feature sharing among teams with strong consistency<\/strong> between model training and serving.<\/p>\n<\/div>\n\nFurther Feast comes with centralized feature management, discovery, feature validation,<\/strong> and feature aggregation. The feature columns reside inside wide-entity tables. In addition, the composite entities separate individual features.<\/p>\n<\/div>\n\n<\/h2>\nWix Feature Store<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nWix<\/strong> provides a platform for feature-sharing across different ML models for both batch<\/strong> and real-time<\/strong> datasets. It supports a pre-configured set of feature families on the site and user-level for both training and serving models. The different stages of data management, model training and deployment are marked and show in the figure above. It further uses S3<\/strong> to store real-time extracted features.<\/p>\n\n<\/div>\n
\nFeatureStore from Comcast<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe Feature Store<\/strong> developed by Comcast<\/strong> helps data scientists to reuse versioned features, upload online (real-time)\/streaming data, and review feature metrics<\/strong> by models. The product is available in multiple pluggable feature store components. The built-in model repository contains artifacts related to data pre-processing (normalization, scaling)<\/strong> displaying the required mapping to the features needed to execute the model. Further, the architecture is built using Spark on Alluxio (open source data orchestration layer that brings data close to compute for big data and AI\/ML workloads in the cloud), S3, HDFS, RDBMS, Kafka, Kinesis<\/strong>. The Model deployment with Kubeflow<\/strong> helps to build a resilient, highly available distributed systems with support for rate-limiting, shadow deployments, and auto-scaling<\/strong>.<\/p>\n<\/div>\n\nThe integration with Data Lake with suitable API s helps data scientists to use SQL and create training\/validation\/test datasets<\/strong> that can be versioned<\/strong> and integrated into the full model pipeline. In addition, the framework comes with the support of Seldon Inference Graphs for A\/B Testing, Ensembles, Multi-armed bandits, Custom combinations<\/strong>. The end to end system not only provides traceability from use-case, models, features, model to features mapping, versioned datasets, model training codebase, model deployment containers, and prediction\/outcome sinks, it is also known for integration with Feature-Store, Container Repository, and Git to integrate data, code and run-time artifacts for CI\/CD integration<\/strong>.<\/p>\n<\/div>\n\nJust like any other architecture, it has continuous Feature Aggregation on streaming data + on-demand<\/strong> features. The Online Feature Store uses the following sequences before giving a prediction:<\/p>\n<\/div>\n\n\n- Payload only contains Model Name & Account Number<\/li>\n
- Model Metadata informs which features are needed for the model<\/li>\n
- Pull required by features by Account Number<\/li>\n
- Pass a full set of assembled features for model execution<\/li>\n<\/ul>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\n
Source<\/a><\/p>\n<\/div>\n<\/div>\n\nHopWorks Enterprise Edition<\/a> is a multi-tenant architecture that integrates AWS Sagemaker, Databricks, Kubernetes, and Jupyter Notebook. It also supports integration with Authentication frameworks like LDAP, Kerberos, and Oauth2<\/strong>.<\/p>\n<\/div>\n\nThe Batch \/ Live Streaming functionality is facilitated by Apache Beam, Apache Flink, and Apache Spark,<\/strong> whereas the model governance and monitoring pipeline are built using Kafka and Spark Streaming.<\/p>\n<\/div>\n\nThe architecture is composed of several building blocks namely<\/p>\n<\/div>\n
\n\n- The Feature Store API – For reading\/writing to\/from the feature store<\/li>\n
- The Feature Store Registry – User-Interface to discover features<\/li>\n
- Feature Metadata – Documentation, Analysis and Versioning<\/li>\n
- Feature Engineering Job – For computation<\/li>\n
- Storage Layer – For feature storage<\/li>\n<\/ul>\n<\/div>\n\n
Netflix Feature Store<\/strong><\/a><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe feature store developed by Netflix supports both online and offline model training and development. The online micro-services enables the framework to collect the data elements required by the feature encoders in a model. It further passes this downstream for future use by offline predictions. The Fact Logging<\/strong> service of Netflix logs user-related, video-related, and computation specific features<\/strong> in a serialized<\/strong> format in appropriate storage units (S3<\/strong>).<\/p>\n<\/div>\n\nThe unique point of this architecture is the presence of components that help to:<\/p>\n<\/div>\n
\n\n- Develop\/Create contexts to snapshot<\/li>\n
- Snapshot data of various micro-services for the selected context<\/li>\n
- Build APIs to serve this data for a given time coordinate in the past<\/li>\n<\/ul>\n<\/div>\n\n
As snapshotting data for all contexts (e.g all member profiles, devices, times of day) would incur overhead and cost, Netflix relies on selecting samples of contexts to snapshot<\/strong> periodically (at regular intervals – daily\/twice daily), though different algorithms. It achieves this through Spark, by training data on different distributions, and by using stratified samples<\/strong> based on properties such as viewing patterns, devices, time spent on the service, region, etc.<\/p>\n<\/div>\n\nNetflix embraces a fine-grained Service Oriented Architecture for cloud-based deployment model.<\/p>\n<\/div>\n
\nFBLearner from Facebook<\/strong><\/span><\/h2>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n<\/div>\n<\/div>\n\nThe FBLearner designed by Facebook is a framework for AI WorkFlow<\/strong> with Model Management and Deployment<\/strong>. It is mainly composed of 3 components – FB Learner Feature Store (runs on CPU), FB Learner Flow (runs on CPU +GPU), and FB Learner Predictor (runs on CPU). I<\/strong>t supports building all kinds of deep learning models (Caffe2, Pytorch, Tensorflow, MxNet, CNTK)<\/strong> and models can be stored in ONNX format (standardizes portability<\/a> across converters, runtimes, compilers, and visualizers. supports and to) across different hardware\/software platforms.<\/p>\n<\/div>\n\nThe above broad categories can be seen as creating logical units from hardware to application software.<\/p>\n<\/div>\n
\n\n- Frameworks<\/strong> (FB Learner Feature Store)<\/strong> needed to create, migrate and train models<\/li>\n
- Platforms<\/strong>
Feature store has become an important unit of organizations developing predictive services across any industry domain. Some of the earlier challenges in deploying ML solutions at scale involves :<\/p>\n<\/div>\n
- \n
- Developing<\/strong> and maintaining customized systems<\/strong> by individual teams with little or no coordination.<\/li>\n
- No collaborative<\/strong> system for sharing features for similar type ML models (models from a similar domain or models addressing. same business use-cases or customer domains).<\/li>\n
- Increased cognitive burden<\/strong> without the proper scope of scalability<\/li>\n
- Limited integration<\/strong> with big-data ecosystems.<\/li>\n
- Limited scope for model retraining, comparison, model governance, and traceability<\/strong>, limiting agile development life-cycle.<\/li>\n
- Difficult to track and retrain model<\/strong> which exhibits seasonality<\/li>\n<\/ul>\n<\/div>\n
\nTo overcome the above limitations, Architects. Data scientists, Big Data, and Analytics professionals have felt the necessity to walk under one roof with one unified framework to facilitate easier collaboration, sharing of data, results, reports.<\/p>\n<\/div>\n
\nDepartments, teams and organizations shared some of the similar notions of Feature Engineering:<\/p>\n<\/div>\n
\n- \n
- Feature Engineering is expensive<\/strong> and amortization<\/strong> happens over time and across models.<\/li>\n
- The increase in cost is non-linear\/exponential with the increase<\/strong> in the number of features.<\/li>\n
- Triggers\/Alerts due to addition\/removal of feature is high.<\/li>\n
- Most often dependencies are not documented\/tracked<\/strong> which results in an increase<\/strong> of implicit<\/strong> and explicit dependencies<\/strong> getting added over time.<\/li>\n<\/ul>\n<\/div>\n
\nWhile sharing a similar opinion, it became easier to come together and create a Unified Framework<\/strong> called Feature Store.<\/strong> This would enhance the speed of ML model deployment life-cycle<\/strong> along with the creation of proper documents, required version analysis,<\/strong> and model performance<\/strong> in order to save time and effort.<\/p>\n<\/div>\n
\nIn this blog, we highlight on the features supported by different Feature Store frameworks, that are primarily developed by different leading industry giants.<\/p>\n
\n<\/div>\n
\nAdvantages of Feature Store<\/strong><\/span><\/h2>\n<\/div>\n
\n- \n
- Ability to re-use and discover features<\/strong> between teams across the organization.<\/li>\n
- Features should be governed<\/strong> by adding features like access control and versioning<\/strong>.<\/li>\n
- Ability to precompute<\/strong> and automatically backfill features<\/strong> — including online computation and offline aggregation<\/li>\n
- Helping to create a collaborative environment<\/strong> between data scientists and big data engineers<\/li>\n
- Save effort<\/strong> and cost<\/strong> by sharing not only features but also related artifacts<\/strong>, documents, marketing insights of models developed from these features.<\/li>\n
- Enable consistency<\/strong> between training and serving.<\/li>\n<\/ul>\n<\/div>\n
\nMichaelengelo From Uber<\/strong><\/span><\/h2>\n<\/div>\n
\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n
<\/div>\n<\/div>\n\nMichaelangelo<\/strong> – a framework developed by Uber<\/strong> that allows feature integration\/joining in both offline and online pipelines. Here Hive (Offline)<\/strong> and Cassandra (Online)<\/strong> acts as the main storage unit for raw\/transformed features. It provides a horizontally scalable multi-tenant<\/strong> architecture for multiple models with suitable scaling and monitoring. Training jobs can be configured and managed through a web UI or an API, via Jupyter notebook.<\/a><\/p>\n<\/div>\n
\nIt further provides options to define hierarchical partitioning schema to train models per partition, that can be deployed as a single logical model. This provides easy bootstrapping<\/strong> and helps to overcome challenges when several models need to be trained based on the hierarchical structure of the data.<\/p>\n<\/div>\n
\nAt runtime during serving, it finds root to the best model for each node. Further its best known for its ability to support continuous learning, providing integration with AutoML<\/strong>, along with its support for distributed deep learning<\/strong>.<\/p>\n
\n<\/div>\n
\nFeast Feature Store<\/strong><\/span><\/h2>\n<\/div>\n
\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n
<\/div>\n<\/div>\n\nGoogle released Feast<\/strong> which is primarily built around Google Cloud services: Big Query (offline) and Big Table (online) and Redis (low-latency)<\/strong>, with Apache Beam<\/strong> for feature engineering. It allows a clear separation between big data and model development. This online predictive service allows feature sharing among teams with strong consistency<\/strong> between model training and serving.<\/p>\n<\/div>\n
\nFurther Feast comes with centralized feature management, discovery, feature validation,<\/strong> and feature aggregation. The feature columns reside inside wide-entity tables. In addition, the composite entities separate individual features.<\/p>\n<\/div>\n
\n<\/h2>\n
Wix Feature Store<\/strong><\/span><\/h2>\n<\/div>\n
\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n
<\/div>\n<\/div>\n\nWix<\/strong> provides a platform for feature-sharing across different ML models for both batch<\/strong> and real-time<\/strong> datasets. It supports a pre-configured set of feature families on the site and user-level for both training and serving models. The different stages of data management, model training and deployment are marked and show in the figure above. It further uses S3<\/strong> to store real-time extracted features.<\/p>\n
\n<\/div>\n
\nFeatureStore from Comcast<\/strong><\/span><\/h2>\n<\/div>\n
\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n
<\/div>\n<\/div>\n\nThe Feature Store<\/strong> developed by Comcast<\/strong> helps data scientists to reuse versioned features, upload online (real-time)\/streaming data, and review feature metrics<\/strong> by models. The product is available in multiple pluggable feature store components. The built-in model repository contains artifacts related to data pre-processing (normalization, scaling)<\/strong> displaying the required mapping to the features needed to execute the model. Further, the architecture is built using Spark on Alluxio (open source data orchestration layer that brings data close to compute for big data and AI\/ML workloads in the cloud), S3, HDFS, RDBMS, Kafka, Kinesis<\/strong>. The Model deployment with Kubeflow<\/strong> helps to build a resilient, highly available distributed systems with support for rate-limiting, shadow deployments, and auto-scaling<\/strong>.<\/p>\n<\/div>\n
\nThe integration with Data Lake with suitable API s helps data scientists to use SQL and create training\/validation\/test datasets<\/strong> that can be versioned<\/strong> and integrated into the full model pipeline. In addition, the framework comes with the support of Seldon Inference Graphs for A\/B Testing, Ensembles, Multi-armed bandits, Custom combinations<\/strong>. The end to end system not only provides traceability from use-case, models, features, model to features mapping, versioned datasets, model training codebase, model deployment containers, and prediction\/outcome sinks, it is also known for integration with Feature-Store, Container Repository, and Git to integrate data, code and run-time artifacts for CI\/CD integration<\/strong>.<\/p>\n<\/div>\n
\nJust like any other architecture, it has continuous Feature Aggregation on streaming data + on-demand<\/strong> features. The Online Feature Store uses the following sequences before giving a prediction:<\/p>\n<\/div>\n
\n- \n
- Payload only contains Model Name & Account Number<\/li>\n
- Model Metadata informs which features are needed for the model<\/li>\n
- Pull required by features by Account Number<\/li>\n
- Pass a full set of assembled features for model execution<\/li>\n<\/ul>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\n
Source<\/a><\/p>\n
<\/div>\n<\/div>\n\nHopWorks Enterprise Edition<\/a> is a multi-tenant architecture that integrates AWS Sagemaker, Databricks, Kubernetes, and Jupyter Notebook. It also supports integration with Authentication frameworks like LDAP, Kerberos, and Oauth2<\/strong>.<\/p>\n<\/div>\n
\nThe Batch \/ Live Streaming functionality is facilitated by Apache Beam, Apache Flink, and Apache Spark,<\/strong> whereas the model governance and monitoring pipeline are built using Kafka and Spark Streaming.<\/p>\n<\/div>\n
\nThe architecture is composed of several building blocks namely<\/p>\n<\/div>\n
\n- \n
- The Feature Store API – For reading\/writing to\/from the feature store<\/li>\n
- The Feature Store Registry – User-Interface to discover features<\/li>\n
- Feature Metadata – Documentation, Analysis and Versioning<\/li>\n
- Feature Engineering Job – For computation<\/li>\n
- Storage Layer – For feature storage<\/li>\n<\/ul>\n<\/div>\n\n
Netflix Feature Store<\/strong><\/a><\/span><\/h2>\n<\/div>\n
\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n
<\/div>\n<\/div>\n\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n
<\/div>\n<\/div>\n\nThe feature store developed by Netflix supports both online and offline model training and development. The online micro-services enables the framework to collect the data elements required by the feature encoders in a model. It further passes this downstream for future use by offline predictions. The Fact Logging<\/strong> service of Netflix logs user-related, video-related, and computation specific features<\/strong> in a serialized<\/strong> format in appropriate storage units (S3<\/strong>).<\/p>\n<\/div>\n
\nThe unique point of this architecture is the presence of components that help to:<\/p>\n<\/div>\n
\n- \n
- Develop\/Create contexts to snapshot<\/li>\n
- Snapshot data of various micro-services for the selected context<\/li>\n
- Build APIs to serve this data for a given time coordinate in the past<\/li>\n<\/ul>\n<\/div>\n\n
As snapshotting data for all contexts (e.g all member profiles, devices, times of day) would incur overhead and cost, Netflix relies on selecting samples of contexts to snapshot<\/strong> periodically (at regular intervals – daily\/twice daily), though different algorithms. It achieves this through Spark, by training data on different distributions, and by using stratified samples<\/strong> based on properties such as viewing patterns, devices, time spent on the service, region, etc.<\/p>\n<\/div>\n
\nNetflix embraces a fine-grained Service Oriented Architecture for cloud-based deployment model.<\/p>\n<\/div>\n
\nFBLearner from Facebook<\/strong><\/span><\/h2>\n<\/div>\n
\n\n<\/div>\n<\/div>\n<\/div>\nSource<\/a><\/p>\n
<\/div>\n<\/div>\n\nThe FBLearner designed by Facebook is a framework for AI WorkFlow<\/strong> with Model Management and Deployment<\/strong>. It is mainly composed of 3 components – FB Learner Feature Store (runs on CPU), FB Learner Flow (runs on CPU +GPU), and FB Learner Predictor (runs on CPU). I<\/strong>t supports building all kinds of deep learning models (Caffe2, Pytorch, Tensorflow, MxNet, CNTK)<\/strong> and models can be stored in ONNX format (standardizes portability<\/a> across converters, runtimes, compilers, and visualizers. supports and to) across different hardware\/software platforms.<\/p>\n<\/div>\n
\nThe above broad categories can be seen as creating logical units from hardware to application software.<\/p>\n<\/div>\n
\n- \n
- Frameworks<\/strong> (FB Learner Feature Store)<\/strong> needed to create, migrate and train models<\/li>\n
- Platforms<\/strong>
- Frameworks<\/strong> (FB Learner Feature Store)<\/strong> needed to create, migrate and train models<\/li>\n
- Features should be governed<\/strong> by adding features like access control and versioning<\/strong>.<\/li>\n
- The increase in cost is non-linear\/exponential with the increase<\/strong> in the number of features.<\/li>\n
- No collaborative<\/strong> system for sharing features for similar type ML models (models from a similar domain or models addressing. same business use-cases or customer domains).<\/li>\n