If it's in an ebony neck I would take as fine as you can generate ebony dust and fill the spaces (not cracks!) with that DRY. Then take Stew-Mac liquid CA and glue in the inlay as normal, letting it fill in underneath the inlay and soak in the dust.
If it's in any other wood type (depending on how wide the gap is) either leave it, and the natural wood will show through, or you just need to rout more carefully.
If you decide to "fill" the gaps, and it's rosewood, or someting as dark, just use black. It will most likely look less obvious than trying to match the wood color type.
If you choose to leave it clear so you see the wood through the clear glue, it will make a "window" of sorts. The wood underneath wil look a little darker due to the glue, but keeping the fretboard oiled will help blend it in.
Also if using the "window" idea- take a permanent black marker and blacken the sides of your inlays before installing them, then the bright pearl (if it is) will not show the sides through the gaps making the inlays look off. It helps it look more perfect.
These ideas only work well for ebony and darker rosewood, or darker walnut.
If it's maple then you can just do it over again..
Craig