Folks at the Empire need to just stop messing with things they don't understand...
Take the Black Gem to the Priest in the basement. Then leave Empire, and come back in.
Return to the castle and - everyone is gone.
The few remaining people inside the castle don't give very good clues - but if you talk to the guard out front, he says the monsters broke through the door and took off to the east.
The rest of the townspeople aren't much help, either, except for the little kid on the west side of town who says he saw a monster fly off... so, apparently, you need to be able to fly after them.
A quick trip to the Mysterious Hut says you need the help of the Wind Spirit in order to climb the Wind Tower - so head to the Antarctic Plateau and talk to the Wind Spirit. She'll break the seal on the Wind Tower, allowing you to go beyond the second floor.
So, your next destination is the Wind Tower, which is west of the Small Village.