It depends on how offensive the jokes were to you and how comfortable you are being direct. I would start off by just responding with body language - when he makes a racist joke or a joke you're not comfortable with, don't smile or respond positively. Just raise an eyebrow and give him a look that conveys: "Really? You just said that?". Then change the subject or make a harmless joke.
If he persists, and doesn't pick up on your cues, you might say something like, "That's really not appropriate for work." If he really crosses the line and won't stop, then you pull out the big guns and be very direct, "That's racist, offensive, and completely inappropriate. Do not speak like that around me. The next time I will go to HR (or the boss or whomever)."
But it sounds like that kind of directness won't be necessary, and the guy has gotten the hint already. Some people just don't know their audience, and I bet all you'll need to do is just not laugh at the jokes you don't find funny.