Silvio Savarese

"If you want to build a service agent that does returns, you cannot afford for the returns to be wrong one out of ten times. It has to be 99.9% correct. If you have an agent that comes up with a loan, you cannot have an agent make the wrong recommendation about the loan. These are scenarios where we need to have high consistency."

"These are very similar to what DeepMind did when they were training AlphaGo. They have instructions on how to play Go, along with some trajectories on how the game should be played. However, to achieve a breakthrough in performance, they had it interact with another AI. Then they began discovering new strategies, architectures, and approaches for winning the games, and eventually ended up with something that actually beat the gold masters at some point. The same concept can be applied here in the realm of enterprise."

"Generating thousands or millions of trajectories will help us find the right mechanisms for incorporating feedback and re-training those LLMs [Large Language Models]. There are numerous opportunities for developing new training engines, learning engines, and ultimately creating new benchmarks. We cannot improve what we cannot measure. The benchmarks are critical for assessing our current position."

"This doesn't cover all of the objects, fields, and complexity of the orgs that use Salesforce. But even within this simplified scenario, we can go towards something that improves performance for those use cases being tested. The next step is to increase the scope. We need to do more work, add more objects, and more complexity. That's what we want to do in the next phase."

"Prompt engineering is a bit of an art. So, there's a lot of work that goes into that, but finding an automatic way to engineer the prompt could be very useful, because it eliminates the need for humans to do the engineering. You have a system that automatically comes up with the right way of engineering the prompt."

"Our goal is to scale it up and make it as close as possible to real-world use cases. I think it's actually the next stage of the arms race for agentic AI. I have seen these other cases of discussion about simulation bubbling up in this area of the enterprise."

"This kind of scenario can be implemented by plugging it into the right structure, such as CRM-Arena, where we have started simulating all these different business objects and their dependencies. You start training agents that replicate what a customer will do and what a human agent will do to create a lot of interesting use cases. You can actually cover a lot of ground, which is somewhat difficult to predict beforehand."