Holding ones fist up to their ear, as if holding a rotary phone handset, may come across as overtly aggressive.

Or sexual depending on the angle and tightness of fist.