What I heard at CTA’s fourth annual Health AI+ Conference
On September 9, I spent the day at the Consumer Technology Association’s Health AI+ Conference in Washington, D.C., including joining Jesse Ehrenfeld, Laura Adams, and John Whyte for a panel moderated by Julie Barnes with the deliberately provocative title: Should AI Be Allowed to Practice Medicine?
It was a lively discussion. But what stayed with me afterward was not whether anyone could produce a satisfying yes-or-no answer. It was how often the conversation across the day moved beyond that question into the conditions under which AI would actually become useful in care. What happens after the AI produces an answer? What evidence should allow it to act more independently? When does human oversight improve safety, and who remains accountable when the system acts?
I did not hear consensus on those questions. I would have been suspicious if I had.
Patients are not waiting for the healthcare system
René Quashie opened the conference with CTA research on how consumers use AI and social media for health information. The research described people using these tools before and after clinical visits, with many verifying the information they find. The question is no longer whether consumers will bring AI into their healthcare. They already are.
That reality kept resurfacing throughout the day. Patients are using general-purpose AI to understand symptoms, interpret records, prepare for appointments, make sense of medications, and ask questions they may be embarrassed or reluctant to ask another person. The chatbot panel described people using AI to interpret discharge instructions while obtaining medical supplies, or disclosing information to an AI that they had not disclosed to their physician. These were accounts of how people are already using the tools, not proposed features for a future product.
Laura Adams, drawing on a formulation from Sue Sheridan, described patients adopting AI at the speed of desperation. Her point was that access to care is itself a patient-safety problem. A person waiting weeks for an appointment may have a question that needs an answer now, regardless of whether the healthcare system is ready to accommodate the tool they turn to.
We are not deciding whether to introduce AI into a pristine clinical environment. It is already entering care through health systems, clinicians, devices, vendors, and increasingly through patients themselves. The clinical relationship may no longer be just patient and clinician, or even patient, clinician, and institutional AI. The patient may bring an independently chosen AI into that relationship as well. That has implications I do not think healthcare has begun to absorb.
Assume the AI works. What happens next?
Katie Drasser of RockHealth.org asked us to assume the AI works and consider what happens after it does. Can the patient get the specialist appointment the AI says they need? Is the intervention covered? Is there somewhere nearby to receive it? Is the necessary data available? Her concern was that the democratization of intelligence is moving faster than innovation in the care model around it. Amit Phull of Doximity made a closely related point: intelligence alone will not be enough; the system around that intelligence determines whether better outcomes actually result.
I think this is an important correction to the way we talk about AI. We spend enormous amounts of time debating models as though the model were the intervention. It usually is not.
The clinical value of a diabetic-retinopathy system depends on a patient being identified, screened, appropriately referred, treated, and followed—not merely on an algorithm correctly classifying an image. A medication-management agent succeeds not just because it knows the appropriate dose, but because the patient, data, drug, workflow, monitoring, exceptions, and accountability surrounding that decision work together.
Patients experience pathways, not models.
The afternoon discussion of the AI divide extended this point. Some patients who historically have poor access to healthcare may now have extraordinary access to information. That does not mean they have insurance, transportation, a specialist, or a healthcare system capable of acting on what they learn. Nor are the tools evenly distributed: the panel discussed clinicians in small or rural practices lacking the resources available at major medical centers, and patients unable to afford premium wearables or subscription services.
Katie described an infrastructure divide between what people can now do for themselves and the system’s ability to support what comes next. That is a different problem from simply getting technology into more people’s hands.
Democratizing intelligence is not the same thing as democratizing healthcare. But neither should we ignore the possibility that inexpensive intelligence can meaningfully reduce longstanding asymmetries between patients and institutions. The question is whether the care available to those patients can make use of it.
“Human in the loop” needs a job description
Our panel spent considerable time on human oversight. John Whyte emphasized clinical context that may not be present in the data. Jesse Ehrenfeld described exception-based oversight, in which routine functions could operate with greater autonomy while defined changes trigger escalation to an accountable clinician or care team. Laura Adams challenged the assumption that adding a human necessarily makes a pathway safer, particularly when review introduces delay or when clinicians override a system that is performing correctly.
All three observations can be true. Which is why I increasingly find “human in the loop” to be an inadequate endpoint for the discussion. The better question is: what is the human there to contribute?
The contribution may be contextual clinical judgment or exception handling. It may be relationship, empathy, values, informed consent, or accountability. A human may genuinely catch what the AI misses. But inserting a human into every transaction may also add delay, cost, inconsistency, or false reassurance without improving the result.
Human presence is not the same thing as human contribution. If a human role is required, we should be able to explain why.
Compared to what?
This was one place where the conference reinforced rather than changed my thinking. I have written and spoken repeatedly about what I think is one of the missing questions in healthcare-AI safety: compared to what?
When an AI error becomes visible, too often the implicit comparator is an imaginary clinician with unlimited time, perfect knowledge, flawless records, immediate availability, no cognitive bias, and no diagnostic error. That clinician does not exist.
The real alternative might be excellent specialist care. It might also be an overworked primary-care clinician, a six-week wait, fragmented information, Google, a family member, or no care at all. On our panel, Jesse made essentially this point when discussing consumer-facing AI: the comparison cannot always be an ideal clinician who is not actually available to the patient.
That does not mean poor existing care should become the safety standard for AI. It means the actual alternative belongs in the analysis. The harms of deploying an AI system matter. So do the harms of not deploying one. Patient safety requires us to count both.
The policy conversation is becoming more concrete
The government sessions brought these questions into discussions of evaluation, payment, and oversight. Stephanie Carlton of CMS distinguished benchmarking from clinical confirmation, using the analogy of a medical degree followed by residency. She asked practical questions about what should be monitored in real time, who should monitor it, which outcomes matter, and how CMS’s reporting capabilities could contribute. She described work still being developed and sought input, rather than presenting a finished answer.
Representatives from FDA, CMS, ONC, and ARPA-H also described their collaboration. Rick Abramson explained that ARPA-H research had influenced FDA’s regulatory thinking, that FDA’s discussion paper was influencing CMS policy proposals, and that the relationships among the agencies ran in multiple directions.
That is not the same thing as having resolved the regulatory questions. Systems that change rapidly, interact directly with patients, and operate with varying degrees of autonomy do not fit neatly into a discussion about a product alone. Their use also involves payment, health-information infrastructure, organizational governance, and the practice of medicine.
The questions being discussed were correspondingly specific: what clinical function is being performed, what evidence demonstrates competency, and under what conditions should greater independence be permitted? How should performance be monitored after deployment, and what happens when the system changes? Who remains accountable for the care pathway in which it operates?
Those questions connect the technology to the decisions that patients, clinicians, institutions, and regulators will actually have to make.
Better questions are progress
One of the recurring temptations in healthcare AI is to make either the technology or the regulation the whole story. Health AI+ reminded me why neither is sufficient. The model matters enormously, but so do the evidence supporting it, the workflow surrounding it, the actual alternatives available to the patient, and the infrastructure that turns its output into care.
We will need to determine where people add distinctive value—and where requiring people to perform tasks that machines or deterministic systems can perform better simply creates another bottleneck. We will also need to remain accountable for what happens when those tasks are redistributed. The technology is moving quickly enough that we will not get to answer those questions one at a time.
Health AI+ did not provide all the answers. What it provided was a room full of people willing to ask the harder questions, disagree seriously about the answers, and keep working on them.
My thanks to René Quashie and the Consumer Technology Association team for bringing that room together, to Julie Barnes for moderating our panel, and to Jesse Ehrenfeld, Laura Adams, John Whyte, and the many other speakers and participants who made the conversation worth continuing.
-Marc d. Paradis
About the Author: Marc d. Paradis’ professional journey is a fusion of academic rigor with real-world impact. He began his career over 30 years ago as an academic molecular neurobiologist, instilling in him a deep respect for critical thinking and the scientific method.
Transitioning into industry, he held leadership roles that bridged data and healthcare: as Vice President of Data Strategy at Northwell Health, Marc leveraged one of the world’s most diverse clinical data sets to drive patient-centered innovation via a $100M partnership with Aegis Ventures, launching multiple AI-centered startups; and as Vice President & Dean of Data Science University at Optum, he spearheaded the training of thousands of professionals in practical, product-centric AI, data-driven decision making, and ethical data practices. In each role, he fostered cultures of curiosity, critical thinking, and collaboration – precursors to the Constructive Inquiry ethos.
About SIYOM Consulting: Founded by Marc d. Paradis, SIYOM Consulting is a boutique advisory specializing in Data and AI Strategy for Healthcare and Life Sciences. We help health-system executives, pharma innovators and investors identify, evaluate and execute on high-value data and AI opportunities.
Source note for review: Speaker remarks are paraphrased from personal notes, not presented as speaker- and/or audio-verified quotations. My notes attribute the “speed of desperation” formulation to Sue Sheridan, as relayed by Laura Adams.