The growing inclusion of social robots in therapeutic situations raises a variety of unresolved legal and ethical issues, including risks to patient autonomy, human dignity and trust, the potentially life-threatening effects of inaccurate or malfunctioning technology, diminished privacy due to the reliance on enormous amounts of personal (sensitive health) data, new challenges to data security due to the cyber-physical nature of robots, and the problem of how to obtain informed consent to medical treatments based on opaque AI decision making. From this broad spectrum, the article deals with the protection of the health and safety of patients and care recipients under EU law. A more detailed analysis shows that neither the EU’s Medical Device Regulation nor its Artificial Intelligence Act adequately address the risks to patient health and safety that arise in many situations where AI systems are used in the healthcare and nursing sector. Against this backdrop, the article provides recommendations as to which aspects should be regulated in the future and argues for a public discussion about the extent to which we, as a society, should replace human therapists with AI-enabled technology.