When we were in Italy and I tried speaking Italian, every single person would immediately start speaking English to me. I would laugh because often their English was MUCH worse than my Italian.
My maternal grandmother was the first of her FOO to be born in the states, as she's the youngest of 5 and her mother was pregnant with her when they immigrated from Sicily in '21. So she grew up with a lot of Italian, I imagine, as she told me once that when she went to visit Italy she decided she was going to use that Italian.
She laughed as she told me that they said "Speak English, we understand you better!"
As for the question? No, you're not rude. I do think the idea to converse in French with people you know well is a good one.