Depending on the wording it could be easily misconstrued. Would you like to go out to dinner with us? VS. Would you like to have dinner with us?
The later form makes it less obvious what the plans are and if they aren't suggesting/getting suggestions on a restaurant in the email I would probably assume it was at their home as well.