i am going to take a guess as i dont know what you are using, and advise you invest in better/the correct sandpaper.
it can seem expensive, but good quality sandpaper clogs less and stays sharp for longer. i have had some cheapo stuff before where the grit literally fell away on anything more than the softest of woods. you wont get anywhere like this
also, the correct type helps. you are still sanding finish rather than wood so use the appropriate type
i am currently favouring emery cloth intended for polishing metals. It can clog on some woods, but eats through most hard finishes in no time. it looks like its clogging, but a little brush/vacuum off and its fresh as a daisy. i never have to go coarser than 100 grit with it, and that is a nice place to be when you hit wood