I realized something as I was thinking about my next post. Pull your armor penalty out of the skill total and put it in parenthesis on your sheet, since these skills are much more broad and the penalty will not apply to all situations.
Examples: "Listen" and "Spot" roll into Subterfuge, and your armor won't affect those, breaking a door down is Physical, etc.