I have inlayed a 7.25" radius fretboard and didn't run in to much trouble. Since most shell products are cut to .05" thick there is usually plenty of surface to sand down when the inlay is complete so you shouldn't "sand through" your shell. Now the grain of the shell may change slightly due to the slightly different depth that you may sand into your shell, but it should still look good. I do suggest radius sanding blocks that can be found at most luthier supply stores (I bought some from stew mac). The 8" work the best for me since they give you more sanding surface area at once and give you a truer radius. The sanding blocks even work well when changing fretboard radius. I turned a 9.5" to a 12" radius without any problem. Once you've cut or routed your cavities, just glue/epoxy in your inlays as normal adding slightly more glue to the outer edges of the inlay if your pattern covers a large area of the fret (since shell doesn't bend well). Your router should follow your fretboard radius fairly well keeping the cavity depth even. Once dried, your radius sanding block will sand down all your high spot and radius your inlays as well as any high spots on your fretboard. As far as a compound radius, I've never done one. Since your radius changes throughout the length of the fretboard I imagine you will need several sanding blocks and a lot of patience and care in ensuring there are even transitions. But all in all I think the inlay process will remain the same.
Hope this helps.