Your vote on this answer has already been received
Even when we humans were in an earlier anthropological state....there was nothing else to attract but the scent of a human body between members of the opposite sex.
However...over a period of evaluation...today...the odour is masked in deoderants and perfumes.
So the feel which filters down to mind is the way we like or dislike the attitude and presentation...which is now term as sex appeal...and its true...its our differing perceptions of the person today...which heightens the appeal or negates it...alongwith the God gifted power of attracting the opposite sex...
which percolates down to the term ' sex appeal ' as we know it now .