If I was wanting to catch up with a friend, I wouldn't want to do it at a young childs child's bday party. When the kids are under 5, the hostess and parents of the guests will be busy paying attention to the kids, especially if there is organized games.
But if the purpose is so your old friends get to meet your child and you get to meet theirs, then I think it is fine to do a kid friendly party, I just wouldn't do it with my child being the GoH.
If I was an old friend childess adult, I wouldn't want to attend. But if I had suggested we get together and this is what you invited me to, then I'd feel obligated to go.
Your son is 3 and probably has less of a concept of time than you. You said he was having a party at home. On his birthday, I would plan an outing with your family and BFF to celebrate his actual birthday.