Is it wrong to be upset because of this?

My boyfriend and my brother go on little golfing trips and are gone for at least 5 hours when they do. My boyfriend works two jobs so I hardly get to see him, and when he chooses to hangout with my brother rather than me, I get upset. Is that acceptable? :/