The man in the Mysterious Hut will give you clues any time you get stuck.
Sometimes they'll be hard to figure out though, so you might still have to do some exploring.
Anyway, you wanna come here immediately after Port City, so you can get the Archer's Deed and Happy Balance.
The Archer's Deed will now let you change to the Archer job class! (For everyone except Bruce, who's already an Archer.)
And Happy Balance will let everyone in your party earn EXP - not just the ones who participate in battle.
Once you've got that, sail a bit further south to reach Dragonlord Castle.