"among the riff-raff of Bohemia"

This is 'Bohemia' in the cultural sense, as in the movement towards a freer, glamorously impoverished life focused on the arts. Madame Verdurin is known as a supporter of young musicians and artists, and this lifestyle is at odds with the hierarchical, conservative atmosphere of the Faubourg Saint-Germain.