You were using the spot for the purpose for which it was intended-parking. Whether you were parking to eat, check a map, comb your hair, call home to make sure the cats hadn't set the place on fire, or walk around the block really doesn't matter, IMO.
Sure-it is very nice of you to move from a popular spot to a less popular spot if it is an option. But it isn't rude to use a parking spot to park in-even if you don't 'need' to be in that particular spot. Truthfully, outside of handicapped spaces, noone needs to be in a particular space as a general rule (I am sure that there are exceptions and special cases that I am not thinking of).
If I was in your shoes, and I often am as I travel and drive for work a lot, I would have stayed in the spot I found until I was done eating unless there was a nearby spot that I preferred because it was shady/had a nicer view/was quieter/etc. I eat a lot of fast food in my car when I am doing field work, and I park where it is legal/convenient/free without a single worry about whether someone else would like to park where I am.