I don't get too bent out of shape about "per say."
The original phrase is Latin, and few people read it anywhere. They hear it.
And "per" meaning "according to" fits with "say" in the meaning of "per se."
The true meaning of "per se" is "by, of, or in itself."
But let's look at a sentence:
This candidate is not a pacifist per se, but he is in favor of peaceful solutions when practicable.
Substitute "according to how that is normally said" in there:
"this candidate is not a pacifist according to how that term is normally said, but he is..."