The best thing to do is set yourself about 2 hours of free time, turn off the power to any wire near that box (use one of those electric sensors), make sure you have a flash light, around.
First, pull as much of the wires free as you can, restrip everything and get new caps and tape everything new. If you think or even suspect another shitty splice is outside the box then yes, break through the drywall and make sure you tape and replace any splices there as well.
edit: ha, just saw that's not drywall...well tile is not as easy to fix, but honestly it's not that bad either...just a bigger project.
edit: am I seeing water damage (the green on the copper and debris in the box?) Just make sure there's no water leaking in your wall....
source: own several very old houses - I do this everytime I change a ceiling light - they are just masses of cloth tape and garbage...