"I had always held a little towards Gulliver's opinion"

This refers to an episode in 'Gulliver's Travels', written in 1726 by Jonathan Swift.  It occurs in the fourth and final part of the story, set from 1710 to 1715.  Gulliver has been abandoned on a distant island by his mutinous crew, which transpires to be the Country of the Houyhnhnms.  He first encounters a savage group of people called the Yahoos, who are essentially humanlike, but in all the worst ways.  They are ruled by the Houyhnhnm, a race of intelligent horses.  These creatures exhibit far better characteristics, such as wisdom, calmness and rationality.  They have no religion, so - fascinatingly - their state is entirely secular.  They do demonstrate a lack of emotion, but Gulliver comes to admire them greatly, and love their country.  When he is eventually forced to return home to England, he becomes reclusive - preferring to spend time with his horses. 

Gulliver's Travels on Book Drum