Talk to tour guide in Kalay's Inn. Not the one in purple, but the one in yellow on the stage.
Leave the town to the side and you will see them leave the town before you.
Buy a ticket.
Talk to everyone on the ship, the charm is [spoiler]on the mast[/spoiler] after a lot of talking the ship will set sail.