Feature Versioning
Designing for Capital One’s Machine Learning Feature Platform
What’s a machine learning (ML) feature?
A feature is a computation based on a data set.
The current user experience wasn’t designed for the feature producer or consumer user base, and didn’t have their needs accounted for
Project Profile
Position: Design Team Lead
Design team size: 5
Users: Data scientists, engineers, and product managers in the machine learning group
Value: Over $1B in potential revenue generation
KPI: Number of features reused
Stakeholders: SVPs in Product and Tech
Time: 3 months
Teamwork Processes
Research revealed another crucial persona
Design used journey maps like this to visualize the users’ complex processes to our partners
I directed my team to pivot to a systems thinking approach to the two personas
How do the actions of a producer affect the consumer, and vice-versa?
Feature producers
Need to create and maintain multiple versions of a feature
Are limited in how many versions they can maintain at once
Encouraged to deprecate (eliminate) versions to save compute power and space
No means of communicating with feature consumers
Feature consumers
Base business operations on a re-used feature version
Need to understand when and how a feature is changed
Must adopt features before deprecation to avoid business disruption
Receive no notice that features are changing
To avoid these problems, feature consumers actively avoided feature reuse, the KPI of the business.
Instead, they wasted time creating their own features even when the one they wanted already existed.
How might we allow for feature versioning from producers while addressing the needs of feature consumers?
We designed for automation, standardization, and better communication
Better Communication
Create pathways for producer-consumer communication.
Advise the producer while working with features.
Automation+Standardization
Feature consumers get notifications that help them monitor features and take action.
Standardize feature metadata.
How I set my teams up for success
Results
The monthly sharing KPI increased 11% in the first month, and continued to increase between 9-32% in the next three months.
The change log was the most requested feature, allowing for greater feature reuse because consumers could now understand what new versions had to offer.
The feature details page was revised to display information aligned with users’ needs.
A guided Feature Registration experience made it easier for Producers and provided standardized information for Consumers.
Feature Metadata was made available at the request of the Consumers, who use the information to select what features they want to consume.
The deprecation confirmation popup warned producers before execution, preventing accidental deletions and unintended consequences for consumers.
If the system detected that a shared feature was in use, a deprecation warning was sent to producers to prevent any disruption caused by deprecation.
A deprecation notice was sent to feature consumers automatically
Once the feature was deprecated, it was reflected on the feature lifecycle page.
Results
The change log gave consumers precise descriptions of each feature version, allowing them to understand if a feature no longer fit their needs, or seeing if it did, encouraging their reuse.
A new feature details page allowed consumers to browse for features so they could avoid building their own.
A new user flow for producers changing the state of their feature walked them through the process, providing guardrails via modals and sending automated messages to consumers when changes were made.
The change log was the most requested feature, allowing for greater feature reuse.
Feature Registration
The deprecation confirmation popup warned producers before execution.
A deprecation notice was sent to feature consumers automatically
The feature details page was revised to display information aligned with users’ needs.
Feature Metadata
The deprecation warning was sent to producers to minimize disruption caused by deprecation.
Once the feature was deprecated, it was reflected on the feature lifecycle page.