Cursor, an AI-powered platform, recently earned the title of the ‘fastest growing SaaS platform’ of all time, reaching from $1 million to $100 million in annual recurring revenue in just 12 months.

When a product achieves that level of prominence, all eyes are on it. When users hit a snag, it doesn’t take long for the word to spread. It is even more concerning when they begin to drift away from the product.

These are the kinds of developments that have recently surrounded Cursor. Initially, several users reported that the app uninstalled itself from their devices. More recently, it was observed that Cursor automatically logged users out when they switched to another device. 

When users reached out for assistance from Cursor, the AI-powered support agent fabricated a usage policy that did not exist. According to reports, the AI agent, called Sam, claimed that a policy was in place within Cursor that restricted usage to a single user and on a single device. This false claim enraged many users, prompting some to cancel their subscriptions. 

“Multi-device workflows are table stakes for developers, and if you’re going to pull something that disruptive, you’d at least expect a changelog entry,” said a user on Hacker News, who also cancelled their Cursor subscription.

Moreover, many questioned how Cursor, a startup that has championed an AI use case, could fall victim to an AI hallucination. This, in turn, raises the question of whether fully automated AI customer support systems are even worth it. 

Building Reliable AI Customer Experience Systems

What AI models excel at is engaging in conversation—there is a reason they are called large language models. Over time, this has encouraged teams from large, small, and medium-sized organisations to interact with their customers and automate the process. 

Hallucinations like the above can have a deterministic effect on both user experience and the company, especially if operating at or above the scale of Cursor. 

At AIM’s Machine Learning Developers’ Summit (MLDS) 2025, Kruthika Kumar Muralidharan, director of analytics at Razorpay, spoke about how to address the above problems. 

He mentioned that the very first aspect to consider is capturing the core problems faced by customers, identifying where support teams are truly struggling and where AI should come into play before adopting LLMs. 

Defining a precise objective and scope is critical, he stated, indicating that businesses need to know exactly which problem they’re solving and how they’ll measure success. For instance, if the chatbot’s goal is to solve complex queries but it is only trained on simple, low-hanging queries, it will inevitably fall short in real-world situations. 

Hence, companies should first implement low-risk AI interactions in customer support systems. “For example, you can start with the FAQ section, where a large amount of data is already documented. It’s low risk and carries a small impact as well,” he said. 

On the other hand, he also stated that businesses should avoid adopting complex and time-consuming training methods while building a pilot program. Instead, he suggested it is better to evaluate the AI customer service chatbot with easier problems to test, understand, and validate the results. 

“Don’t expose anything directly to customers yet. Have some human testing and validation for a while before it gets released,” Muralidharan said. 

Besides, he stressed that a core mission team must regularly validate, test, and measure the chatbot’s success, make necessary corrections, and iterate. Muralidharan added that the core mission team should consist of users and professionals from all facets of the product. 

“You train, you deploy, you get weird outputs, you panic, then you make changes, and go back to step one, and start doing it again. I pretty much call this a standard operating procedure,” he said. 

Once the pilot has been validated and the solution needs to be scaled, companies must delve into specifics, such as selecting the appropriate AI model or framework to use. 

Furthermore, he stated that a key factor to ensure a reliable experience is to limit the use cases to specific questions only. This likely ensures that AI will only answer queries for which it is confident. 

“We keep experimenting, tweaking, and fine-tuning to ensure we get it right before we release it to customers,” he added. 

All things considered, evading errors, deviations and building a reliable AI chatbot requires a solid testing and validation pipeline. Without it, delivering a strong and trustworthy customer experience becomes highly doubtful.

The post Cursor’s Mishap Becomes a Cautionary Tale for AI Customer Service appeared first on Analytics India Magazine.