It is possible to skip the peahats near the stamp chest by doing boomerang LJAs over the void.
The trigger that loads the path to Arbiter's Grounds doesn't exist until you defeat King Bulblin and break through the fences with a boar.
Instead of making the whole trip through the Bulblin camp while fighting you can lja on top of the Camp to trigger the cutscene and skip the small key saving ~8 seconds.
The LJA is rather difficult because of the rather small frame window for a successful jump.
Note that its still faster to Jumpattack + Quickspins on King Bulblin.