That is a great question.
I believe that the economic times have hit Hassan much harder than even he himself would admit. He had to let go of most of his crew, except of course his beloved Frederick.
Did you know Hassan found Frederick by the dock one day while cleaning barnackles off of his boat? Yeah. He found him in a clamshell bassinet with a note pinned to his chest:
"Take care of my boy, his name is Frederick. We will return for him in time and you will know us because we will be carrying half of a seasweed and squid ink locket that will be the perfect mate for the one around the boy's neck."
So Hassan raised him as his own.
So he couldn't fire him. When he is referring to "the guys" I believe he is referring to all that remains of his crew, which is himself, Frederick, Cookie (their galley chief) and Swarthy McStab, their enforcer.
Cookie and Swarthy were sleeping off a bit of the drink in the cabin near the dock and didn't make the voyage because in keeping costs down, Hassan only went with himself and loyal Frederick.
Which is a shame. Swarthy McStab comes in REALLY handy in a fight and Cookie makes a mean Squid Pie.