While you can always decline invitations, there are going to be some that you are going to want to accept. So you need to work on techniques that will allow you to visit with people and at the same time eat enough to stay healthy.
One thing you might consider is using the term "medical condition" a lot. "High metabolism" isn't getting the message across to people, so an alternative might carry more weight.
Another thing is that even though you have explained the condition to people, they will still need further explanations as to how this affects your day to day life.
We had lunch at 11:00AM and at 4:00 I asked my aunt for a snack. She gave a tuna sandwich, an apple, and a pear, and said "don't eat too much we have a big dinner ready in a few hours".
Clearly, your aunt doesn't understand. This is the time to look her in the eye and explain, "Don't worry about that, Aunt. I've explained about my medical condition. Even though this looks like a lot of food, I will most certainly be very hungry for dinner."
Sometimes people don't always plan everything they are going to do so Sunday afternoon we go for a drive and then my uncle suggests we stop in and visit a friend. Well, OK but I need to eat in half an hour. "Can you wait a while? Maybe my friend can give you a snack." Well, OK I can wait, but I am really supposed to eat, I'm already starting to feel hungry, and it is embarrassing to explain to someone I don't know why I need to walk into his house and put away a 1500 to 2000 calorie meal.
If you agree to the surprise visit without eating beforehand, what you are doing is sending mixed signals. First, you say you need to eat a lot and often. Then, when you mention eating before the visit, you don't press the point and seem to be able to go longer without food. You still need to educate these people.
"Uncle, I'm sorry, but I really do need to get something to eat before we visit Friend. I thought we would be home by 4 o'clock, so I only packed a few protein bars. I'm going to need a meal in the next hour. It would be really embarrassing for me to walk into a stranger's home and demand a meal. Could we please stop at [insert restaurant/deli/fast food place] on the way so that I can get some food?" "Uncle, a snack is not going to be enough. I need to have some protein and carbs and about 1500 calories. A turkey sub would fit the bill--do we pass by a sandwich shop on the way? I can eat it in the car."
Stop telling yourself, and others, that you can wait when really you cannot wait. They aren't going to take your needs seriously until you, yourself, take them seriously.
Be as proactive as possible. Bring extra food. Try to find out the day's schedule if possible. "Auntie, you know I need to plan my meals out ahead of time to make sure I get enough to eat. When were you planning on having lunch and dinner? Oh, okay. If we all have lunch at 11, I'll have another meal at 3:30--don't worry, I brought the food with me-- and then I'll be hungry for dinner at 6."
Try to keep a sense of humor about it all. "You know me, I'm like a hobbit. I have First Breakfast at 7, and Second Breakfast at 11."