This is rather ambiguous both in terms of who is speaking and what is being referred to. The most likely explanation is that Quentin is being mocked for his incestuous desire for Caddy (something he is apparently poor at concealing). Amongst the many scandalous affairs on which Lord Byron embarked was an alliance with his half-sister, Augusta Leigh. Controversy has long raged around the physicality of their relationship, but the current scholarly consensus is that it was consummated and that the poet is likely to have fathered Elizabeth Medora Leigh. This is evidently not the view of the speaker (Shreve or Sydney) who implies that if a notorious Lothario such as Byron failed to seduce his sister, Quentin stands no chance.
‘Epistle to Augusta’ (1816) testifies to Byron’s desire for his half-sister.