"they must be swallows who had come back from the south."

Swallows are a common type of bird, which travel in flocks and can feed whilst flying. They migrate south during the winter and north during the summer months, so their arrival is seen as a sign of oncoming summer. Their peripatetic lifestyle is similar to that which Stephen will experience after his self-imposed exile.