I have had this exact same problem with frets. In my case, I think the prob was my hammer was so forceful in the middle it was popping the edges out a bit (not saying that's your prob - I have no idea). My most recent one I switched to a tiny hammer, just lightly tapped each edge, then used a drill press/caul to go all the way, and also used super glue. def listen to prostheta - he's a real guru.
really like where this build is going.