"The Children of Green Knowe" is the first in a series of six books about a house in Hemingford Abbots, Cambridgeshire, called Green Knowe, and the spirits who live there. Most of the books concern a boy named Toseland (Tolly for short) and the spirits of his ancestors, who also appear as children. Through the ghosts of the house, Tolly learns about his own heritage and the history of Green Knowe itself.
The Children of Green Knowe was adapted for television in 1986- here's a clip from the show. At the time of writing, the entire series was available on YouTube, but it's probably best not to get distracted this early on!