This bug has been around for awhile, but I just found it out.
This works with any ability, I'm just using HoJ as an example.
Let's say you have Hammer of Justice bound to your "C" key. After you die, spam the C key, then leave the arena.
If I wanted to HoJ someone after I died in an arena match, I would spam C (my HoJ bind) and then leave the arena while continuing to spam C.