The AI is not "picking" a target. There are rules re priority targets which apply to player and AI units alike, and can be taken into account when manoeuvring troops.
In a game that is attempting to simulate realistic battlefield behaviour, in the context of an IGOUGO system, such rules are necessary to prevent units from behaving in farcically unrealistic ways. In a real battle, the enemy would not go into suspended animation while your troops move. Your unit would not ignore cavalry to its front to attack an enemy unit that currently poses no threat to it - however tempting a target it might seem for the wargamer. If they did so in a real battle, they would be charged by the cavalry and caught out of a defensive formation, so the unit commander knows not to do it.
Essentially, the positioning of those cavalry protects that flank. That, after all, is the cavalry's job.
There is no random element in the application of the priority target rules. To quote the manual:
Units may have priority charge target(s) – these are (a) enemy within two map squares directly to their front (one square if the chargers are facing diagonally) and facing within 45 degrees of directly towards them, or (b) enemy ZOCing them. Priority charge targets apply if the charge path starts in or passes through a square in which they would apply. If a unit has a priority target, it cannot charge a non-priority target. This stops units from ganging up on the front units of a chequerboard formation, or adjacent units in a line. Restrictions apply equally to player and AI units. Priority targets do not affect shooting (except reaction fire), but the arc of fire rules ensure that a unit which shoots at a different target will usually only shoot at half effect.
Note that it is possible for a unit to have more than one priority target, in which case it can attack any of these but not some other unit which is not priority.
I can't attack the unit I worked so hard to get on it's flank and I can't attack the cavalry.
