Honestly, I don't think I would have mentioned that you already had the book. I would have just said thanks and plan on giving it away/exchanging it later. I wouldn't have handed it back to them. Given your rela-tionship with your brother, what you said is in all good humor and not at all horribly bad, but I wouldn't have shared that it was a duplicate at all. I wouldn't have given it back either.
If I duplicated a gift, I wouldn't expect that gift back, surely they can exchange it or do something with it. Bummer, though, but these things happen.
Same here. I've gotten some doozies from family over the years. I just smile, say thanks, and donate the item later.