Not rude per say; I've gotten friend requests from friends of friends, who I don't even know, and I've just ignored them. I also defriended someone else I knew of, from college, when I realized I really didn't know her, and wasn't interested in seeing all her stuff on FB. I was new to FB and hadn't figured out that I wanted to keep my friend list to those I knew well.
Personally, I'd just leave it up to the person being friended whether or not they wanted to accept the request. And I don't know if I would have de-friended someone over that, but its a personal choice, if you felt like you had to, then its your choice.