In the Small Village, you learn a little about the nearby Wind Tower. The Wind Tower is sealed and you can't get past the second floor (yet).
You also learn that the Empire is south of the village, and an old man also says they're not supposed to enter the graveyard in the south of town.
In the graveyard, there's a set of stairs leading down to a prison, which is currently holding a fairy. You need a special key to unlock the prison. That key is in West Cave, which is to the southwest, and is your next destination (after a short detour to the Traveler's Fort).
When you're done here, exit to the World Map and head due south until you cross a small bridge. Now go around the mountains, then head west to reach the Traveler's Fort.