When AI Gives Bad Health Advice, Who Regulates the Regulators?

When AI Gives Bad Health Advice, Who Regulates the Regulators?

https://kaboompics.com/ Pexels
Your smartwatch tells you that your recovery score is terrible. An AI fitness coach recommends changing your training. A symptom checker suggests that the pain in your chest is probably harmless. Another chatbot confidently explains which supplements you should take, how much weight you should lose or whether a medical treatment is really necessary. Artificial intelligence is rapidly moving from answering trivia questions to giving advice about our bodies. That changes the stakes. A bad restaurant recommendation may ruin dinner.

Bad health advice can do considerably more damage. American lawmakers have noticed, and states are beginning to build rules around AI systems that influence health decisions. That sounds sensible. But it also raises an uncomfortable question: if artificial intelligence must meet increasingly strict standards when discussing health, what standards should apply when questionable medical claims come from humans in positions of authority?


Your fitness tracker is becoming a health adviser

The border between fitness technology and medical technology has become increasingly difficult to see. Apple Watch can record an ECG and issue irregular-rhythm notifications. Oura tracks heart rate variability, temperature trends and sleep. Garmin, Whoop and countless other devices turn physiological measurements into readiness, recovery and training recommendations. Add generative AI and the relationship changes again.

Instead of simply displaying numbers, software can interpret them for us. Ask why your resting heart rate increased, whether your HRV should worry you or whether you should train with a poor recovery score, and an AI system can produce an authoritative-sounding explanation within seconds. Sometimes that explanation may be excellent. Sometimes it may be wrong. The dangerous part is that both versions can sound equally confident.


Health informationWhen AI says itWhen a human says it
Medical claimShould be supported by reliable evidenceShould be supported by reliable evidence
UncertaintyShould be communicated instead of inventing certaintyShould be communicated instead of presenting opinion as fact
Serious health decisionHuman professional review may be necessaryQualified medical expertise remains necessary
Wrong informationAn AI hallucination is not evidenceAuthority or political office does not turn a claim into evidence
AccountabilityDevelopers, providers and regulatorsProfessionals, institutions and public authorities
SAME MEDICAL STANDARD. WHETHER IT’S AI OR HUMAN.

America is building an AI rulebook one state at a time

The United States does not currently have one simple national rulebook governing every use of artificial intelligence. States have moved in different directions, creating an increasingly complicated regulatory landscape. California has introduced requirements for certain chatbot operators, including disclosure when users could reasonably believe they are interacting with a human. Colorado has also introduced protections around conversational AI and professional services. For companies building national fitness and health platforms, this creates an unusual situation: the software may be identical, but the legal environment surrounding it can change when the user crosses a state line.


Colorado shows what happens when AI enters real medicine

Some regulation goes considerably further than telling a chatbot to admit that it is a machine. Colorado's HB26-1139, signed in June 2026 and taking effect in January 2027, addresses artificial intelligence used in health-care utilization review. If an AI-assisted system contributes to denying coverage for health-care services, that denial must be reviewed by an appropriately qualified licensed clinician or other competent regulated professional.

The underlying principle is difficult to argue with: algorithms can assist a decision, but consequential medical decisions should not simply disappear into a black box. In other words, when an automated system may affect someone's treatment, lawmakers want competent humans somewhere in the loop.


That sounds reasonable — until humans start giving the advice

And here the story becomes much more interesting. In August 2026, President Donald Trump signed an executive order calling for a reduced routine childhood vaccination schedule and advocating separate measles, mumps and rubella vaccines rather than the established combined MMR vaccine once separate products become available. The policy advances goals associated with Health Secretary Robert F. Kennedy Jr. and arrived amid renewed suggestions from the administration of a relationship between vaccination and autism.

That proposed relationship is not supported by decades of scientific evidence. Major medical organizations rejected the changes, saying there was no new scientific evidence that justified them. Even Republican Senator Bill Cassidy, a physician and chairman of the Senate health committee, publicly said the executive order was wrong.


The strange case of splitting the MMR vaccine

The MMR example demonstrates why evidence matters more than who is speaking. Separating the measles, mumps and rubella vaccine might sound intuitively safer to someone worried about giving several vaccines together. But intuition is not clinical evidence. Separate single-disease versions are not currently licensed and available in the United States. Manufacturers have said there is no published scientific evidence showing a benefit from separating the vaccines, while decades of data support the combined vaccine.

Developing, testing, licensing and manufacturing separate versions could take years. This is precisely the type of situation in which we would criticize an AI chatbot if it confidently recommended a medical intervention without evidence. The uncomfortable question is why the standard should change when the recommendation comes from a human being.


A hallucination is still a hallucination when a human says it

Much of the debate surrounding generative AI focuses on hallucinations: answers that sound convincing but are unsupported or simply invented. It is a genuine problem, particularly in medicine. But humans have been producing the biological equivalent for centuries. We believe anecdotes, confuse correlation with causation, selectively accept evidence that supports our existing beliefs and sometimes continue repeating claims long after better evidence has arrived. Artificial intelligence did not invent misinformation. It merely found an extraordinarily efficient way to produce and distribute it. A hallucination does not become evidence because it comes from a chatbot. But misinformation does not become evidence because it comes from a podium in Washington, either.


The answer cannot be to trust AI instead of people

None of this means that AI should be allowed to say whatever it wants about health. Quite the opposite. A chatbot telling someone to ignore chest pain, discontinue prescribed medication or take an unsafe dose of a supplement could cause genuine harm. Health applications need safeguards, transparent limitations and clear boundaries between general information and individualized medical care. Systems making consequential decisions should be independently tested, and users deserve to know when an algorithm rather than a clinician is influencing those decisions. The mistake would be assuming that inserting a human automatically solves the problem. Human review is valuable only when the reviewer applies evidence, professional standards and appropriate expertise.


Fitness sits directly on the fault line

This matters particularly to the fitness industry because fitness has always occupied the fuzzy territory between lifestyle advice and medicine. Telling someone to perform three sets instead of four is unlikely to become a regulatory crisis. Interpreting an abnormal ECG is different. So is advising someone with diabetes about fasting, telling a pregnant athlete which supplements are safe or interpreting repeated oxygen-saturation abnormalities during sleep.

Modern fitness platforms increasingly possess information that previous generations would have encountered only in a clinic: resting heart rate, ECG traces, blood oxygen estimates, temperature changes, sleep patterns and sometimes continuous glucose data. AI can combine those measurements and identify patterns no person would notice by scrolling through months of graphs. That potential is enormous. So is the temptation to turn probabilities into diagnoses.


The most dangerous sentence may be “AI says…”

We already have a cultural tendency to treat computer-generated numbers as more objective than human judgment. A recovery score of 38 feels strangely authoritative even though it is the product of sensors, assumptions and an algorithm. Generative AI adds language to that authority. Instead of showing an ambiguous measurement, it can explain exactly what supposedly caused it and what you should do next. The user may never see the uncertainty behind the answer. Good health AI therefore needs something humans often dislike admitting as well: “We don't know.” Sometimes the scientifically correct answer is not a recommendation but a statement that the available information is insufficient.


AI vs. Human Health Advice: Should the Standard Be Different?
The questionAI health adviceHuman health advice
Is there evidence?The claim should be supported by reliable scientific evidence.The claim should be supported by reliable scientific evidence.
Is uncertainty acknowledged?AI should not turn incomplete information into confident medical certainty.People should not turn opinion, intuition or political conviction into medical certainty.
Can the claim be challenged?Sources, limitations and consequential decisions should be open to scrutiny.Public authority should not place a medical claim beyond scientific scrutiny.
What if it is wrong?A confident AI hallucination is still misinformation.A confident statement from an influential human can still be misinformation.
What should matter most?Evidence, transparency, appropriate expertise and accountability.
SAME MEDICAL STANDARD. WHETHER IT’S AI OR HUMAN.

Maybe we need the same rule for machines and humans

America's emerging AI laws are trying to answer extraordinarily complicated questions: Who is responsible when an algorithm causes harm? When must a human review its decision? When should users be told they are interacting with AI? Those debates are necessary. But health information presents a deeper problem that legislation alone cannot solve. The real dividing line should not be human versus artificial intelligence.

It should be evidence versus assertion. A medical claim should face scrutiny because of what it says and the evidence supporting it, not because of whether the words were generated by silicon or spoken by a person. If we eventually demand transparency, evidence and accountability from AI health systems, that would be progress. It might be even greater progress if we remembered to demand the same things from ourselves.

0 Comments