What do you when?

So, I am at a luncheon meeting this week. Another faculty member I know (but do not consider a 'friend') says, "Hey Dr. Awesome, there's room at our table." So, I felt obligated to sit down at his table when I would have rather sat with others (who hadnt arrived yet). There is no way to say, "No, don't want to sit with you. Want to sit with my friends."