the bottom of the yard and jaws twist around the mast allowing the yard to be inclined to one side.
I used to have the same problem, which had several causes that I eliminated one by one:
1. My gaff jaws were too tight (they needed pushing apart a bit), which meant that the yard tended to twist around the mast and then get stuck in one less than aerodynamic position
2. However you attach the halyard to the yard, the angle of pull up towards the wooden block on the back of the mast must be absolutely central. If the halyard attachment is slightly squint on the yard, the yard will not go up square. Claus's halyard through a hole suggestion eliminates this problem
3. Make sure that the yard is properly tight against the mast, which is difficult to achieve if you get the first two wrong
4. If your halyard is stretchy, the yard may fall away from the mast a bit when it is windy. I now use Dyneema for the halyard