Results of the study were announced at the 62nd Annual Meeting of the Association for Computational Linguistics
K Health, the leading AI Primary Care platform, is pleased to announce at the Association for Computational Linguistics (ACL) Conference the results of an AI Medical QA Benchmark Study that demonstrates the accuracy of its Knowledge Agent’s responses to medical questions. The study shows that K Health’s trained LLM Knowledge Agent results are up to 36% more comprehensive and are 41% less hallucination-prone than LLM counterparts.
Primary care is the backbone of U.S. healthcare, vital for better health outcomes and lower costs. Each year, Americans make a staggering one billion visits to primary and urgent care facilities. K Health stands alone as the pioneer in clinical-grade AI for primary care, it is the first patient-facing LLM that is trusted by health systems like Cedars Sinai and other leaders across the country.
For over seven years, K Health has been refining its AI, using unique data to deliver highly personalized care. Their latest innovation K-QA, is a dataset of 1,000+ real patient questions from their platform. With K-QA, the company set out to prove its mettle, pitting its large language model (LLM) against top-tier LLMs and their own expert physicians.
“We are constantly iterating on our technology in order to make sure that our partner health systems are providing their doctors and patients with the best, most accurate and comprehensive medical responses,” said Ran Shaul, Co-Founder and Chief Product Officer, K Health. “Ensuring the accuracy and trustworthiness of K Health’s outcomes is at the center of all we do, and we are pleased to see that our results outpace other AI resources and provide additional support for doctors using our technology to respond to patient questions and concerns.”
K Health’s study addresses the challenges of providing accurate AI-generated responses in clinical settings. K Health first constructed the K-QA dataset consisting of 1,212 patient questions spanning about 100 different medical conditions, which originated from real-world conversations held on K Health’s AI-driven clinical platform, which has over 8 million unique users.
The results were evaluated by a team of 12 in-house medical doctors, who adjudicated the responses by consulting credible medical sources (including UptoDate and PubMed) to provide accurate and scientifically-backed answers; these were then evaluated by an experienced overseeing physician. The decomposed answers were assessed in comparison with the results found in a variety of leading LLM-based architectures, including MedAlpaca, Mistral 7B, Bing Chat, BARD, and GPT3.5 - 4.
Despite the immense potential of AI in healthcare, many solutions have so far been unable to address the underlying problem of providing high-quality, truly personalized care at scale. The positive results from this benchmark support K Health’s groundbreaking offering edify the company’s partnerships with the best health systems in the U.S., including Cedars-Sinai and Mayo Clinic.
About K Health:
K Health is the AI-driven Primary Care company on a mission to provide access to high-quality medical care at scale. Its 24/7 virtual primary care solution is available to millions of people in the U.S. through leading health systems and insurers, as well as through a direct-to-consumer mobile app. K Health’s platform includes a medical chat and AI predictive models based on large datasets to help physicians optimize diagnosis and treatment, and focus on patient care. K Health is backed by leading investors including Valor Equity Partners, Mangrove Capital Partners, GGV Capital, 14W, Comcast Ventures, Lerer Hippeau, Primary Venture Partners, and PICO Capital Partners. K Health is headquartered in New York. For more information, please visit www.khealth.com.
View source version on businesswire.com: https://www.businesswire.com/news/home/20240819504768/en/
