The only wood I've ever had issues with was Cocobolo. It stings my nose after awhile, however, I love working with it, particularly on the lathe making pens.
Another wood I once had was Ipe. An extremely dense, stiff wood I thought would make good neck laminations, however, when I read on the net it causes dermatitis, some kind of skin reaction, it went in the trash.
It's interesting to read some of my favorite woods like Mahogany and Pau Ferro could cause reactions.