Morgan Stanley is shaping the future of financial services

Rolling out AI in financial services required confidence that the technology will deliver outsized value while meeting the firm’s strict standards for quality and reliability. 

Morgan Stanley met this challenge by implementing an evaluation (eval) framework to test every AI use case before deployment. Evals measure how models perform against real-world use cases and guide improvements, with expert feedback, at every step.

The team began with three targeted goals for their first AI use cases:

• Faster information retrieval to save advisors hours of document searching.

• Automation of repetitive tasks like summarizing research reports.

• Enhanced insights tailored to client needs.

To evaluate GPT‑4’s performance against their experts, Morgan Stanley ran summarization evals to test how effectively the model condensed vast amounts of intellectual capital and process-driven content into concise summaries. Advisors and prompt engineers graded AI responses for accuracy and coherence, allowing the team to refine prompts and improve output quality.

The eval framework wasn’t static; it evolved as the team learned. They next introduced translation evals for multilingual clients and worked closely with OpenAI to fine-tune retrieval methods, ensuring AI could handle an ever-expanding document library. 

“We went from being able to answer 7,000 questions to a place where we can now effectively answer any question from a corpus of 100,000 documents,” says David Wu, Head of Firmwide AI Product & Architecture Strategy at Morgan Stanley.

McMillan notes the impact that fast and reliable answers from AI @ Morgan Stanley Assistant has had on advisors’ conversations. “Now, advisors can engage clients on topics they haven’t discussed before because the friction between knowledge and communication has gone to zero.”