"he squeezed it so hard between his Hoof and his Pastern"

Pastern
GNU Free Documentation LicensePastern - Credit: BS Thurner Hof
The rear of a horse's foot just above the hoof is the pastern. The Houyhnhnms have apparently a much more maneuverable hoof than ordinary horses do. They can so bend them that they can grip something between the hoof and pastern.