I'll admit - the president/owner of my company has interviewed in a baseball cap before. That's his normal "work uniform" though. We're in a very laid back office - jeans and t-shirts (clean and neat) are the normal clothing worn here. Several people have visible tattoos and my own hair is bright magenta.
I think it really just depends on your office environment. We work in music so our clients are normally a laid back, eccentric bunch. In other office environments I understand that the office dress code is more professional and baseball caps, t-shirts, and jeans (not to mention non-natural hair colors and visible tattoos) wouldn't be acceptable.
When it comes down to it, if you're uncomfortable with an office culture that's casual enough that the owner/interviewer wears a baseball cap (has visible tattoos, wears jeans and a t-shirt, etc.) then you can turn down the job. If the interviewer was allowed to come into work that day wearing what they're wearing and interview you, then obviously it's an appropriate work uniform for that particular job/office. I wouldn't call it rude, it's just a different, more relaxed office culture.
Honestly, I love my casual office. During the summer I wear flip-flops everyday, over the past 5 months I'd dyed my hair various colors and, as I mentioned before, currently have bright magenta hair. My boss loves it. BUT - our office is extremely casual. If I ever changed jobs, I'd go back to my natural hair color and wear more professional, conservative clothing until I got a handle on what the new office's culture was like.