Backing away only encourages someone to pursue the matter. You tried being gracious, but now it’s time to speak your mind.