Why AI is Not Working for You

Introduction

Today AI is everywhere - well almost; at least it’s everywhere in conversations but is it making a real-world impact? Impact on everyone’s daily lives? As we enter 2025 after the amazing AI ride of the last 2 years, let's answer this fundamental question: is AI in 2025 going to be for Everyone, Everywhere? And, if yes, what do we need to do to get there?

The AI boom has been surreal to watch and is hard to ignore, from the amazing demos released by the OpenAI team catching everyone’s imagination of the possibilities to the 3.66 Billion visits per month (yes, billion with a B) recorded by ChatGPT. But we are all still waiting for the moment AI truly, fundamentally changes the lives of 7 Billion+ people in the world. While this seems like an inevitability, we have all seen how the tech world often gets too absorbed in the innovation bubble (remember Web3?) leaving behind the key ingredients for success - improving the experiences and enhancing productivity for not just the technically capable but for each and every one of us. Today, our phones are the primary means through which we are interfacing with technology. On our phones over 80% of our time is spent on our favourite apps (Facebook, Instagram, Netflix, Spotify, Instacart, etc.) instead of browsing or searching via Google or web browsers. Has AI reached these apps (and you) in a way that has fundamentally changed your experience or made you smarter/faster/better?

We all know the answer to that question is “not yet”. So let’s dig into why that is and how we can change that.

The AI journey so far

In 2024, enterprises building these apps have been quick to adopt AI wherever productivity could be increased for their internal teams (coding, data mining, meeting summarization, etc.) or better customer support for their users, as aptly captured in this Menlo Ventures blog.

‍

These low-hanging fruits of adoption in enterprises along with the rise of net new vertical AI applications warrant us to ask the broader question—will the rise of AI require 7B+ people to learn new ways of interfacing with AI via new apps (i.e. leaving behind their existing favourite apps) or does AI need to be seamlessly integrated in our apps so that AI adapts to you instead of the other way around? 

The current trend in AI adoption very aptly fits the launch of Apple Intelligence with the tagline “AI for the rest of us”, which cheekily implies that today "AI is for the best of us".

AI powering the “Why before the What”

We at NimbleEdge believe that 2025 will see a fundamental shift in consumer apps (as mentioned in these Sapphire Ventures & Lightspeed blog posts) where users will be empowered with AI-driven experiences ensuring these smarter apps understand the “Why before the What”.

These experiences will be powered by on-device AI to capture your intent adapting to every interaction on the app while preserving privacy. So, let’s explore why we believe this is needed and how we can make this future possible. 

To understand the current limitations in today’s app experiences, let’s do a quick experiment. Pick up your phone and go to your favourite streaming or OTT app (Netflix, Amazon Prime, HBO, etc.). Try searching for a movie scene that you remember, instead of the movie’s name. For example, I was recently searching for “Tango dance sequence” on my favourite app, and then searched for “Al Pacino Tango”, but got no useful results.

‍

Frustrated, I just searched for “Al Pacino” and started passively browsing the recommendations (not remotely relevant to my previous interactions) and saw the trailer of Godfather (another great movie!) which was the first recommendation the app gave. Now, as you can imagine, I really intended to watch “Scent of a Woman” but could only remember this dance sequence. In the end, I just closed the app and searched the same query on Google, which instantly understood I was talking about the scent of a woman, and showed YouTube snippets of the scene to me. 

What really happened was that the app never captured my real intent in order to surface the best recommendations. Next, to learn how AI can help here, I gave the same sequence of user in-session events and interactions to ChatGPT, and voila, its recommendations were far better with fairly accurate reasoning. Try it for yourself here.

If you repeat this experiment across apps in any category—be it your favourite shopping or travel app—the results are disappointingly similar. When you’re planning to host a party and adding large quantities of cheese, tortilla chips and browsing for Mexican rice recipes in your grocery shopping app, the intent is pretty clear: “hosting a party with Mexican food”. But most apps will end up recommending unrelated items (like Tide Pods, really?). On the other hand, a smart AI agent like ChatGPT is able to accurately reason over user behaviour and provide much better recommendations.

The examples are endless but you get the gist. We can vastly improve the user experience and retention/conversions for the enterprises building these leading consumer apps which are currently suffering from over 40% bounce rates by providing AI powered reasoning capabilities in a privacy aware manner. Hopefully, by this time you’re convinced that this brave new world is far better than going to Google to search, then going to an app to “manually” add the outputs of the first query, then rinsing and repeating many, many times depending upon the task at hand.

‍

How can you “Think before you Recommend”

We at NimbleEdge call it “Think before you Recommend.” This approach has already shown amazing results in organizations like Spotify where they have observed 4x more user engagement when leveraging GenAI’s world knowledge and reasoning capabilities:

So how do we achieve this? We believe there are three key capabilities needed in order to vastly improve everyone’s in-app experiences and truly make AI the fuel for growth in the next decade:

• Scalable user session interaction ingestion platform - In all the examples above, there’s a key difference between AI being able to capture a user’s intent versus the most advanced apps today; the ability to capture in-session user interactions and events at scale. Most of the apps interface with their backend cloud infrastructure through APIs which are used to fetch data needed to power the app. This does include responses like “best results for your search keywords” (not intent!) and recommendations tailored to you based on your past preferences (not real-time session based). Often, these systems act on stale cached data, and any machine learning based recommendations or personalization is based on user cohort data running at fixed batch intervals. However, they fail to capture in-session interactions like reading the description of a movie or adding then removing an item (hinting indecision) or the broader intent of the user (like shopping for a kid’s birthday party). Capturing each and every user interaction at the scale of millions of concurrent users is prohibitively expensive on the cloud—which means you need to capture and process these events on-device. This is exactly what NimbleEdge does. Our platform provides a complete on-device data warehouse and processing engine allowing apps to adapt to user interactions in real-time without incurring massive costs and latencies. Additionally, we enable on-device filtering of events (using Python) to selectively send them to your cloud for fine-tuning and model experimentation as described here.‍
• Scalable and Privacy Aware AI platform - Being able to capture user intent in real-time is a huge unlock for app companies, but this inherently requires them to run AI at scale with millions of concurrent users. The current state-of-the-art LLM providers (like OpenAI or Anthropic) all struggle with this scale, leading to high latencies and degraded user experiences. This can be very expensive given the current token-based pricing models. On top of that, this kind of user interaction data is highly sensitive information that enterprises will be very wary of sharing with a third party. These headwinds coupled with rapidly growing on-device compute power and distillation of models that require far less compute presents a unique opportunity to leverage on-device infrastructure to run AI for performing in-session reasoning and capturing user intent. NimbleEdge solves this by providing a complete on-device AI stack with optimized runtimes for local inference, provisions for RAG/LoRA adapters and pre-shipped optimized foundational models tailored for consumer app use cases. We partner with the leading AI teams in the world from Meta, Microsoft, Sarvam and Inflection to create the best models optimized for on-device execution.
• ‍Analytics and Rapid Iteration - As these enterprises start to roll out capabilities built on AI, a key piece of the puzzle is allowing them to rapidly iterate and enhance their customer’s user experience. It is critical for them to understand the impact of AI reasoning by capturing analytics from devices and rapidly iterating by updating Prompt templates, RAGs or even models on-the-fly without requiring app updates. For reference, Spotify’s team achieved 14% better results by tailoring and adapting Llama modes to their tasks.

Additionally, these new capabilities need to be progressively delivered in a controlled manner (such as via cohorting) to understand the impact and return on investment (ROI). NimbleEdge’s SaaS platform is built to provide enterprises with these capabilities enabling the full lifecycle management of AI on-device.

We believe 2025 will be the year of “Building experiences and not just AI” and NimbleEdge is at the forefront of leading this transformation by working with the leading food and grocery delivery, horizontal e-commerce, media, finance and gaming companies to empower users with experiences and bring AI benefits for everyone, everywhere!

Please reach out to us at contact@nimbleedge.com to learn more and get your access to the NimbleEdge platform.