"The gentlemen retired to the smoking room for port and cigars"
Cuban Cigars in a Humidor
Creative Commons AttributionCuban Cigars in a Humidor - Credit: Alexbrn

Port and cigars were the traditional British way of finishing dinner: a rich, sweet alcohol to aid digestion after a heavy meal, and a pungent cigar to complement the taste of the port.  Tradition dictates that the port bottle is always passed to the left, and never put down.

Men and women would separate after dinner, at least for a short while, with the women withdrawing to the drawing room and the men remaining at the table or, as here, moving to the smoking room.  It would be up to the men to decide when to rejoin the women.