No, it's not rude.
If anything, it might be awkward depending on how much you interacted with this person in the group settings you usually meet in. But, if you broke the ice in the group setting to the point where you are comfortable inviting this person out for coffee on your own, then there's nothing wrong with it.