Dr. Nicholas Gavin, an emergency room physician at Mount Sinai in New York, was working nights last summer when a patient arrived with a confusing set of symptoms. Within seconds, his three young colleagues – two medical students and a resident – were consulting a free artificial intelligence-based application for doctors, OpenEvidence.
Dr. Gavin quickly realized that they were far from outliers. A third of Mount Sinai’s 9,000 doctors were already regular users of OpenEvidence, health system executives discovered during a meeting last year with the startup’s executives.
“This has been an ‘aha’ moment for our leadership,” said Dr. Gavin, who also serves as the system’s director of clinical innovation.
OpenEvidence’s AI application, essentially a chatbot for medicine, has become a viral hit with doctors. Talk to a doctor and they’re likely to use the app to ask specific medical questions or to exchange ideas in a diagnostic dialogue.
More than half of the country’s doctors are regular users. Last month, they used it for 30 million questions and consultations, almost double the volume six months earlier, according to the start-up. A separate survey of 1,000 doctors last year found that 45% used the app, nearly triple the percentage of ChatGPT users, according to Offcall, a career information service for doctors.
This growth propelled the start-up to a valuation of $12 billion in January, up from $3.5 billion last July.
But rapid adoption of the app by doctors since its introduction in 2024 – one of the few AI-enhanced programs on the market seeking to win over doctors – has increased concerns about how and when the technology should be used in life-or-death situations. In a high-stakes field like medicine, health systems face thorny questions related to patient privacy, security and trust, as well as the limitations of the technology itself.
“It’s not an oracle, it’s a tool,” said Daniel Nadler, founder and CEO of OpenEvidence. “Knowledge and knowledge workers still matter. »
The medical office has been the target of computer-assisted decision-making for decades, with very limited success until recent advances in AI.
The first wave of AI in medicine aimed to alleviate the heavy documentation burden that contributes to physician burnout through transcriptions and summaries of patient visits, called AI scribe software. The second wave, which has just started, aims to use AI to help doctors by providing reliable information and advice to guide diagnosis and treatment at a patient’s bedside.
Competition has intensified in recent months. UpToDate, a traditional electronic reference popular with doctors, has given its AI service a makeover with a chatbot interface. Doximity, an online professional network for doctors, has acquired an AI startup that mines medical literature and generates summaries. Abridge, a fast-growing AI scribe maker, adds decision support tools. And last month, OpenAI introduced ChatGPT for clinicians.
OpenEvidence became a leader in part because it exclusively used medical journals and other high-quality research as data to train its AI models. Doctors can ask app-specific questions or enter a patient’s characteristics and symptoms and ask for potential explanations. The app complies with federal law that protects patient health information, and doctors are asked not to enter any personally identifying information.
OpenEvidence responds with a summary of the most likely diagnoses, then offers other “most important diagnoses not to miss.” Each contains links to the research articles that inform the summaries.
“AI solves some of the problems that have long plagued the practice of medicine,” said Dr. Raja-Elie Abdulnour, director of clinical innovation at NEJM Group, which publishes the New England Journal of Medicine. “These tools just didn’t exist before, and that’s why people are so excited about them today.”
Still, initial enthusiasm must be tempered with a large dose of caution, medical experts agree. So far, research on the pros and cons of AI in medicine is decidedly mixed.
The AI passed standard licensing exams and outperformed human doctors in diagnosing some cases. But the AI also stumbled, failing to accurately summarize research papers or giving wrong answers to diagnostic questions. And it won’t replace humans anytime soon.
“The potential for AI is enormous, but we’re not there yet,” said Dr. Eric Topol, a cardiologist and executive vice president at Scripps Research in San Diego. “It hasn’t really been tested and demonstrated in the real, messy world of medicine.”
Dr. Topol co-authored a recent paper, “The Illusion of Readiness in Health AI,” which revealed “significant skills gaps” in the capability of large AI systems when applied to healthcare.
Evaluations so far have largely focused on the performance of large language models from big tech companies like OpenAI and Google, which are trained on data from the open Internet.
OpenEvidence, founded in 2022, has taken a more targeted approach. He’s betting that smaller AI software models, trained on highly specialized data, could outperform giant models in a specific, information-rich field like medicine. The startup initially trained its software on publicly available medical data from sources such as the government’s National Library of Medicine.
Next, the company entered into content licensing agreements with the New England Journal of Medicine, the Journal of the American Medical Association and other publishers of peer-reviewed medical literature.
OpenEvidence is available to any government-verified physician in America as a free downloadable app.
“We treat doctors like consumers,” Mr. Nadler said. Users see ads, many from pharmaceutical companies, for about five seconds where they wait for the AI to respond. Doctors receive ads on only 5 percent of their questions, the company said.
Bypassing the traditional gatekeepers of hospital technology services has raised some issues. OpenEvidence relies on workplace behavior known as “shadow AI,” with workers using such tools without the knowledge or oversight of their employers.
Some health systems are now working to integrate OpenEvidence into the institutional fold. Mount Sinai announced in March that it would provide a link to OpenEvidence directly from a patient’s electronic health record.
But the agreement does not give the start-up access to the medical center’s patient data. This integration could come later, Dr Gavin said, but only after rigorous testing and monitoring.
Protecting patient privacy and safety will be “paramount,” he said, adding that “we’re not just going to throw a patient’s data over the wall in favor of a private company.”
Doctors in small practices across the country, particularly in rural areas, say the technology has won them over.
In Corinth, Mississippi, Dr. Ben Long considers himself an AI skeptic. But he was reassured that OpenEvidence generates answers based only on high-quality, peer-reviewed information.
At first, Dr. Long used it primarily as a reference tool, asking factual questions. But now he sees the app more as “a consultant, a thought partner” with whom he dialogues, he says.
“AI forces you to think more deeply about your own thinking, question your assumptions and understand why you might be wrong,” Dr. Long said.
AI can also allow doctors to tap into expertise that would normally be reserved for specialists.
Dr. Barbara Creighton often diagnoses and treats complex cases at a community hospital in Fairbanks, Alaska. They can involve multiple conditions and failing organs. In a large medical center, a team of specialists may be consulted: an infectious disease expert, a pulmonologist and a gastroenterologist, for example.
Dr. Creighton’s small hospital is not as well staffed. He has an agreement with a large medical center to pay for specialist consultation sessions. She now increasingly relies on OpenEvidence to answer many questions, saving her time and her hospital’s money.
“It’s like having a group of specialists in your pocket,” Dr. Creighton said.
At Mount Sinai, Dr. Gavin said he sees AI technology as a powerful tool to help realize the promises of precision medicine with treatments tailored to individuals.
Progress will require a “patchwork of solutions” from hospitals, medical schools and private companies, he said. Whether OpenEvidence thrives and plays a role in this long-term future remains to be seen.
“But this is a step in that direction,” Dr. Gavin said.
