“The heterogeneity in their performance and price make it challenging to decide which API to use.”
Machine learning as a service (MLaaS) is estimated at $1 billion in 2019, and it is expected to grow to $8.4 billion by 2025. At the heart of these services, sits customised APIs from companies like Google and Amazon, who have been spearheading the ‘democratisation of ML’ movement.
For example, one could use Google prediction API (Goo) to classify an image for $0.0015 or to classify the sentiment of a text passage for $0.00025 only!
The race to democratise has made MLaaS a lucrative business model. The result is, today, there are multiple APIs offering similar services. This again, can be challenging. Addressing this issue and to establish a hassle-free ML ecosystem, a group of researchers from Stanford University, introduced a predictive framework called FrugalML that assists the users in switching between APIs in a smart manner. The researchers have detailed about their new framework in a paper titled, ‘To Call or Not to Call?’
Overview Of FrugalML
via paper by Stanford University
FrugalML framework is designed to first learn the strength and weakness of each API jointly, and then perform an efficient optimisation to identify the best adaptive strategy given the user’s budget constraint.
The above illustration compares different approaches that use ML APIs.
- Naively calling a fixed API in (a) results in a fixed cost and accuracy.
- The simple cascade in (b) uses the quality score (QS) from a low-cost open-source model to decide whether to call an additional service.
- Proposed FrugalML approach, in (c), exploits both the quality score and predicted label to select APIs.
FrugalML leverages the modular nature of APIs by designing adaptive strategies that call APIs sequentially.
Adaptive strategies are hard to learn, as the choice of the one predictor could depend on the prediction and confidence of the previous API. FrugalML may need to allocate different fractions of its budget to predictions for different classes. FrugalML optimises adaptive strategies to substantially improve prediction performance over simpler approaches such as model cascades with a fixed quality threshold.
Here, the master problem decides which two services can be the base service, how often should it be invoked, and how large budgets are assigned, while for a fixed base service and budget, the subproblem maximises the expected reward.
First, conditional accuracy is estimated from the training data. Next for each service, the subproblem is solved for some value of a budget, and then an optimal strategy is constructed. Finally, the master problem is transformed into a few linear programmings and solved efficiently.
For experiments, the researchers evaluated the performance of FrugalML as well as those APIs on three datasets, namely, FER+, RAFDB, and AFFECTNET, which contain images with basic emotions and component emotions.
The results show that FrugalML leads to more than 50% cost reduction when using APIs from Google, Microsoft and Face++ for a facial emotion recognition task. Whereas, experiments on FER+ dataset showed that only 33% cost is needed to achieve accuracies that match those of Microsoft API.
The authors posit that the performance of Frugal ML is likely because the base service’s quality score is highly correlated to its prediction accuracy, and their framework only needs to call expensive services for a few difficult data points and relies on the cheaper base services for the relatively easy data points.
Key Takeaways
APIs are designed to do the mundane developer chores for the ML outsiders. However, introducing a flurry of APIs come with their own challenges. This novel framework by the Stanford researchers tried to address a few of them. According to the researchers, the contribution of their work can be summarised as follows:
- A new framework FrugalML is proposed that jointly learns the strength and weakness of each API on different data and performs an efficient optimisation to automatically identify the best sequential strategy to adaptively use the available APIs within a budget constraint.
- Compared to the simple cascade approach, FrugalML consistently reaches a higher accuracy while using the same budget.
- FrugalML aims at finding strategies to select an apt API from the MLaaS market to reduce costs and increase accuracy.
With the successful demonstration of Frugal ML, the researchers are hopeful that the future work will lead to extending FrugalML to produce calling strategies for ML tasks beyond classification as well as comprehensive empirical evaluation.
