I see no reason to stop the stove to brush the ashes off the damper plate. I generally just hold it open by hand for a few seconds, use a toothbrush to brush the ashes off, and chimney draft pulls most of them in and right up the chimney. Try not to drop the toothbrush down the stovepipe.
As to WHEN you should clean the plate, whenever you feel like it. I hardly ever do it. In the pictures, yours doesn’t look bad. Compare the ash weight to the weight of that big counter-balance disk, and the ashes are probably insignificant.