Did you remove the 4-player-only cards from the deck (and use 6 face-up cards)? I didn't count so I don't know if it comes out evenly then, but the setup is definitely different for each number of players.
As for the military, the decision is made is order of descending number of military tokens. Player order is only used when people are tied on military tokens.
Yes we did everything the way you describe