No wonder you have contact with the board. A 95 E is low enough tension and dropping it to C is 40% less tension.
Some people have claimed that some of the old 4001s had softer frets. I couldn't say one way or the other. But a 4003 should have the same hardness nickel silver frets as the competition.
The outer windings of stainless rounds have surfaces like little files, and the windings of Rotos are coarser than most of their competition. Some people who play aggressively do have problems with them eating frets, others don't.
Rick fretboards are finished. I don't see how a refret would be done without disturbing it, and if you're playing with stainless rounds and your strings hit the board, you're going to damage that finish too. Nickel rounds shouldn't be a wear problem for frets, but again, if you hit the surface of the board with them, it will eventually wear.