"what was he doing, teaching those young orphans?"
Ospidale della Pieta
Vivaldi, although ordained as a Roman Catholic priest, did not practise as a priest for long. For thirty years he worked at the Ospedale della Pieta,
which sounds like a very enlighted institution for the time. He taught music to orphaned boys and especially girls, and composed for them. Many of the children were deformed, and Vivaldi had instruments adapted for them. The girls performed for the local gentry, and gained an excellent reputation, even abroad.
The music was performed in the church of Santa Maria della Pieta, no.8 on the map at the back of the book (Afterword p.7).