It is possible to skip the peahats near the stamp chest by doing boomerang LJAs over the void.
The trigger that loads the path to Arbiter's Grounds doesn't exist until you defeat King Bulblin and break through the fences with a boar.
Instead of making the whole trip through the Bulblin camp while fighting you can lja on top of the Camp to trigger the cutscene and skip the small key saving ~8 seconds.
The LJA is rather difficult because of the rather small frame window for a successful jump.
Note that its still faster to Jumpattack + Quickspins on King Bulblin.
It's possible to skip talking to the fairies on the 10th, 20th, and 30th floors by clipping through the door that opens after talking to the fairies. When completing Cave of Ordeals, skipping the 40th floor fairy won't work as then the final fairy on the 50th floor won't spawn in.
Using a Look Through Clip on top of the opened chest near the door, it's possible to clip through past the door and into the hallway. Target walking with the Ball and Chain is recommended as it makes getting onto the edge easier without falling off. Be sure to not look too far left when looking down into the wall, otherwise you may clip out of bounds in the wrong direction by touching the left side of the door.