Wow. People who says those kinds of things really have their own issues. As an adopted child myself, I only really consider that I have one set of parents - those who adopted me. Is your mother still alive? If she is, and these are her brothers and sisters saying this to her/you, I would be very, very hurt - and I am very sorry anyone is insensitive enough to voice something like that ... hugs. Back in the day I would have given them a piece of my mind, and then the cut-direct. Who are they to tell you how much you love someone - blood or not.
Now, I, at the very least, would give a very stern and very audible "Excuse Me?" When approached with the insane-thoughtless question. And then etiquettehell would remind me to breathe and say something along the lines of "I know you are in a lot of pain from the loss, so maybe your diminishing of my relationship with Grandma and Grandpa, the times/years we spent together and the love and moments we shared is not really something you meant to say since I know they loved me and my mother just as much as we loved them and any of their children and grandchildren." If they persist, I would say something along the lines of "I'm sorry you feel I am not really a part of the family. I have always loved and cherished this family, and will keep that in my memories." Then I would turn on my heels and remind myself to never speak with that member of the family again. And I would be honest with anyone who asked why ... "Well, they said since my mother was adopted that I am not really a member of the family, as such I don't feel I have to interact with them or submit myself to those hurtful words anymore."
My sympathies for your loss, and for having to endure the boorish comment from a person you considered a family member.