I don't think we can assume that it was NOT some of those things the co-worker is worried about that caused the miscarriage of the pregnancy.
For most people, living normally will not hurt their pregnancy, for others with certain conditions, it may be an issue. For example, my aunt had to lie in bed for 9 months twice, to have her two children, because just doing normal housework would bring on bleeding and miscarriage. Her daughter inherited the same condition.
Regardless, IMO, the OP should go with the "my doctor is OK with what I am doing", and drop the subject.