The OpenAI and Moderna collaboration is making the rounds on the internet. But a little-known fact is that Indian fitness and lifestyle startup Healthify (formerly HealthifyMe) is also using OpenAI’s GPTs for real-time nutritional analysis, healthy suggestions and much more.

Interestingly, during the release of GPT-4 Turbo, OpenAI recognised Healthify for using the model to perform real-time nutritional analysis. It did that through its image recognition feature, Snap 2.0, leading to a 50% increase in user engagement, which correlates with better diet management and fitness wins.  

According to OpenAI’s recently published customer story, this new tool lets you upload meal photos as they are, and Healthify takes care of the rest. It combines in-house models with GPT-4 Vision to understand user-specific contexts accurately while maintaining data privacy. 

After image analysis, it employs custom heuristic models for precise food recommendations, achieving human-like accuracy in dietary tracking.

Similarly, RIA, Healthify’s generative AI-powered virtual nutritionist, and Coach Co-pilot—its coach-facing assistant—are now using OpenAI’s GPT 3.5, GPT-4 Turbo, and the machine learning model for speech called Whisper, respectively.

This upgrade has doubled the interactions with Ria. Users can now have extended conversations and receive complex health insights, such as the impact of glucose levels on sleep, analysed from various integrated data sources. AI-supported coaching responses have become twice as fast, increasing client interaction by 18%. 

The blog further stated that the Khosla Ventures backed startup used the San Francisco-based AI startup’s Embeddings model to tackle a key challenge of accurately matching food names generated by GPT-4 with its internal database. This challenge stemmed from the fact that GPT-4 and its own system have different food name dictionaries. 

Under the Hood

Launched as the world’s first AI-powered virtual nutritionist in 2018, RIA represented a major step forward in personalised health tech. 

“Before adopting GPT models, RIA employed hierarchical LSTM networks and custom NLU systems to interpret and respond to user queries. Initially, as a primarily rules-based system, RIA had difficulty handling complex and less frequent queries, struggling to provide personalised feedback as the platform and user demands grew,” read the blog post.

Similarly, early iterations of Snap were based on convolutional neural networks, allowing users to take a photo of their meal, which the AI would analyse and log the caloric content tailored specifically for Indian cuisine. 

Additionally, the system was designed to learn from each interaction, gradually personalising recommendations and improving accuracy based on individual dietary patterns and preferences.

However, Snap struggled to identify foods in complex dishes like salads and mixed Indian meals, leading to inaccuracies in the calorie count and nutritional assessments, which reduced user trust and engagement. As noted by Vashisht, this led to a limited usage rate of only 10-20% of the time. 

Behind Healthify’s success is its compact 30-member tech team, supported by 15 analysts and data scientists collaborating closely with seven product managers focused on AI-driven offerings. The company serves over 700 health coaches and around 240,000 clients globally.

“The team uses multiple AI models, including generative and non-generative types, to enhance user interaction and data analysis,” Tushar Vashisht, CEO of Healthify, told AIM. 

Further, he said that the platform is also experimenting with statistical models from AWS, Meta, OpenAI, Anthropic, Google, and open-source platforms, prioritising accuracy, reliability, and cost in its assessments. 

We integrated with everybody who’s out there. OpenAI was the best,” said Vashisht in the blog post.

What Next for Healthify?

“Over the next year, we want to improve autonomous and agent-like functionalities in health applications. These will not only provide suggestions, but will actively facilitate necessary actions to support health management,” Abhijit Khasnis, vice president, technology, at Healthify, told AIM during a recent conversation. 

These agents will proactively manage health by analysing data to recommend optimal dietary, sleep, and exercise routines. For instance, users can schedule medical consultations automatically or order specific nutritional foods tailored to an individual’s health needs based on data monitored by devices tracking sleep, exercise, and dietary habits. 

As per the customer story, initially, the fitness platform faced difficulties scaling its services internationally. It took two years to expand to Southeast Asia using traditional machine learning technologies. Now, leveraging OpenAI’s capabilities, they anticipate launching in 20 countries within the year, advancing towards their goal of impacting a billion lives globally.

“The focus is on expanding into the US and UK markets first, with other markets on the radar. The versatility of generative AI aids in adapting to new markets by facilitating on-the-fly data generation and cultural translations. However, the long-term goal is to be relevant in every market globally, harnessing AI,” Vashisht said. 

Read more: Data Science Hiring Process at Healthify