This is kinda funny cause this doesn't come up often, but I've answered this exact question to 3 different people in the past month. The problem is with the way your bridge pickup is made. Pickups like tele neck, tele bridge, P90s, 2-cable humbuckers with covers are problematic for this. The reason is that they generally only come with a coil-start and coil-end wire. Annoyingly, they don't expect people to want to phase switch so they multi-purpose one of the wires as a GROUND as well. They figure you're just going to solder it to ground anyways.
What I mean by this is they are running a jumper from one wire to the cover/plate to ground that too. It works fine when in normal. However, when you swap the wires, your 'hot' now is a ground hahahaha.
Solution: Unwire the jumper that goes to the cover/base-plate and run that independently to ground (either with a single wire or by upgrading the entire pickup to have a 3-conductor cable out).
Best of luck!
Chris