Theodotus wrote:
The fact that in less than two months you've been able to improve the AI as much as you have just crystallizes for me in my mind something I've pondered before on several occasions. Which is this: Why can't the designers of a particular game themselves create an AI which plays the game like a reasonably good Human player? I mean, it's their game -- they designed it.
In my opinion it is all a matter of how much time and effort you want and are allowed to put at it.
In order to be motivated to put that effort in, you have to consider it fun. This is extremely important in everything you do. If something is no fun for you, you'll never be dedicated enough to do it as good as your ability allows.
And for me AI-programming is a lot of fun. The challenge to teach a machine how to properly play a game has something very fascinating for me.
Also: There's no real pressure from the outside. I can spend days at experimenting with stuff and then scrap it.
I think the absence of pressure is extremely helpful.
Take tile-improvement-management for example. From all things I've worked with, it's the one that I've spent the most time on. And I'm still not satisfied.
Had I been pressured and told I have 5 days and not more, then a lot of tests could not have been conducted.
It's really tough to take everything there is into consideration.
Last sunday I redid the mechanism how the need for mines/fungi is determined because the AI was planting way too many fungi for my taste.
Fungi-planting looked fine then but I sometimes would see suburbs and purifiers on mountaintops because my new evaluation formula valued the base-mineral value of the mountains in addition to the value of the other improvements.
In order to find out what went wrong, I needed more time and patience.
Both things that I imagine are limited when you are under pressure.
I eventually found out it was a copy&paste error, where I mixed up food and minerals at one point, which meant mineral-need was linked to food need and the more food there was the less eager it was to increase mineral-income.
Theodotus wrote:
Clearly, you have significant experience playing the game, and you've put significant thought into how to play it well. But I would anticipate (hope?) that the designers themselves have also done this as they've worked on the game all this time. So why you, rather than them, being the one to finally bring this level of improvement?
We've actually played with the Devs in Multiplayer. They are not bad at the game. No Zak0r level but quite decent.
So I can say that lack of experience and understanding about their game is not the reason.
Theodotus wrote:
I think it boils down to this: you possess a rare cognitive ability to analyze complex systems and interactions and then perceive what's necessary for improvement and how to put it into effect. And I think it's an ability that few people in the game "industry" possess.
That is a nice way to put it.

I personally would have said that what I'm good at is inventing algorithms that recreate my behaviour.
Like: What considerations go into choosing how I would distribute my population to the different jobs and how do I get to my result. A lot of this is done subconsciously from experience. So I have to think about what impregnated my subconscious and be able to word it.
Do I need Units? Do I have a lot of unbuilt buildings/advancements? Am I ahead or behind in science? Does this city have special boni? How do I prevent getting too many or too few miners in comparison to workers, when I don't want to use scientists anymore? Should I use mineral-sources that produce only one mineral? If yes, under what circumstances and how many? Should I consider pollution when doing these assignements, if yes how?
Once again it took many iterations to get it where it is now. Checking that the way the thoughts were implemented actually does what the thought intended for any of the imaginable scenarios takes time and patience.
TL;DR: Lack of ability imho is not the main-reason why AI seems such a neglected field in the gaming-industry. The requirements in order to write a good AI besides the obvious ability are:
time, freedom from pressure and most importantly fun in doing it
I blame capitalism. When games are made to make a living and not because you consider doing it as a great way to spend your free-time, you'll never have the time and dedication to make a game as good as you can imagine.