The words you want to use really depend on your relationship with him. For the sake of my answer I am going to assume you have a warm, good one, one that is open enough and trusting enough that you can be honest. If that is the case I would see myself saying something like "Friend, I love your nonfiction pieces. They are charming, carry a hint of nostalgia along with fascinating history and wonderful insights. Your talent really lies in this direction! I don't think anyone else here could do it better. However, and in all honesty, you don't have that same talent for fiction. It's just not your thing. It's like me; I write too as you know but I know I can't write humor. I've tried, and believe me when I tell you mine is beyond awful. It comes across as so angry, and I don't know why. So I have given that up, and stick with the format that is best for me. I don't want to discourage you if you feel this strongly about it, but I have to say that I have a strong preference for your nonfiction and would prefer to read that. As for fiction and publishing, have you ever looked at the Backspace or Absolute Write forums for excellent advice? They are honest and outstanding, and you can't go wrong seeking advice there."
ETA: OP, I do not mean what I said to sound harsh or worse. But from your original post it sounds like he has little experience navigating today's publishing technology if he honestly believe vanity publishers look for good writing. They look only for good money as you know. If you feel this is the case, then my priority for my friend would be to be honest, to direct him to good writing sites, and to avoid being too gentle (as in "oh, I don't read much fiction to be able to comment") replies, and yes, to discourage his forays into areas where he can get ripped off and eventually very hurt when his books don't sell.