Yeah, I know exactly what you mean. I'm in a very similar situation. All the work I do is initiated by someone else's request. So if they want it fast, they have to be available to answer questions. If I need further clarification on something and can't proceed until I get it, it's in their best interest to be there to answer. And I've even had people tell me they'll be in a meeting and to come get them out if I need something. So I figure, it's their project, and they know the urgency of it.
If I ask them to come over and they feel that something else takes precedence (even answering emails) then, well, that's their call. In my case I've pretty much always got something else to work on if they can't come right away so I just divert my attention as needed.
Maybe it's the particular dynamic of the office I work in. Because everyone is so crazy busy, and they all know that I am too, no one ever bats an eye that I've moved on. I just get back to whatever it is I was asking about when they do come over, sometimes taking a few seconds to refresh in my mind what the question was. Or if I'm away from my desk, they just come back or IM me to say they came by or whatever.
Are you worried about a backlash if you aren't at your desk when they arrive, even if it's been more than just a couple of minutes?