This comes from the book Walden by American author and philosopher Henry David Thoreau. Thoreau had spent some time living in an isolated cabin near Walden Pond, Massachusetts, as a social experiment and he wrote a book about his experiences.
If a man does not keep pace with his companions, perhaps it is because he hears a different drummer. Let him step to the music which he hears, however measured or far away.
The book was published in 1854, making it quite possible that Wigram in The Far Pavilions had read it and was quoting from it.