Non-existent users of the future

Need to talk more about my favourite topic because I am too excited and want to share the joy.

This morning I did a thing I had wanted to do for months and it is finally here. The reasonable question is why I waited for months, and the answer is that user experience matters more than developers' fun, doesn't it?

Let's zoom out for a bit and talk about types of testing. You are probably all familiar with unit, integration, and e2e, because most of my friends here are in engineering. If you are engineering-adjacent (product, design), you know about usability testing, accessibility testing and overall user testing done with real people using your product. It is the most important testing outside of engineering (because if your integration tests let an error bring down production, user testing will not save you).

User testing matters because, however hard you try, you can't get rid of all your assumptions and biases when building a product. I'm sure everyone has seen the picture of the intuitive design of three cat bowls, where instead of each cat using the one in front of them, they contort as much as possible to use random ones? That is pretty much what you want to catch at user testing.

But in very complex products like games, user testing gets much harder. Players make different choices, those choices affect the game differently, and you end up with a huge tree of possibilities instead of one user journey.

There is a fallback so the tree of possibilities doesn't drive you crazy. It's the game loop. The core loop of player behaviour that the designer wants the player to repeat again and again. Larithea's is very simple: fight to get materials → craft spells → use them in fights to get more materials → craft better spells. Simple, and intentionally simplified - every player goes through it.

Adding numbers to the pot. Larithea chapter 1 has around 90 scenes. About 40 of them are on the main quest; the rest are split between 7 side quests. There are 45 sigils (the parts spells are built from) to be combined into about 100 possible spell variations. So is there a way to test all of that without sitting 100 people down for 7 hours?

YES! That's why I'm so excited. Larithea is turn-based and event-sourced, and fights are seeded — same seed, same actions, same result. That makes it possible to build quite effective player simulators, because the number of "correct" actions is limited and the outcome is predictable. So I built my synthetic player to go through fights (it enumerates every legal move-and-cast combination, runs each one through the real game code, and takes the best-scoring board) and then added 3 simulations:

Economy - fight all the scenes. Open sigils. Validate the shortage or excess of materials for sigil crafting. This tells me what the player can afford at the moment each sigil opens.

Difficulty - fight all the scenes, opening sigils as intended, crafting spells when possible, and check where the heroes hit the wall.

Arena - with a predefined set of spells and enemies, how often do you win or lose specially crafted fights.

It's a good start (I think), but it's a mechanical simulation, not a BEHAVIOURAL one. How people interact with the game loop changes based on who they are. But do you know what you can do with AI now pretty easily? SIMULATE BEHAVIOUR.

It is the best thing in the world, honestly. With those 3 sims ready, I had a long conversation with Claude about player personalities, and we developed 3 personas, then joined all 3 sims into a global one.

Default - builds every sigil the moment it opens, in the cheapest version it can, and goes and earns whatever it's short of. The impatient completionist: never leaves a sigil unbuilt, never pays more than it has to.

Strong - refuses the cheap version. It builds the best spell it has the right to build and grinds for however much more that costs, so it fights with better spells but spends far longer getting them.

Rusher - wants to see the end of the chapter. It crafts only what it can pay for on the spot, never replays a battle for materials, and buys a potion when something beats it. When it can't get past a fight, that's where its run ends.

With these 3 personalities, we added levers (weights on choices) to customise the global sim per persona. And then I ran it on different seeds for different personas. It gave me A LOT of JSON output (recommendation: save it per run, not globally at the end so you don’t have to wait 50 mins before asking questions). With AI magic, Claude and I could look through all of it and start finding patterns. Where does Default fail but Strong succeed? Why? Can we make it an equal experience?

An example above is the difficulty run for the default player with the difficulty lever "match the heroes' spell strength enemy side" on and enemy heavy at the end. You can see everybody died on iteration A (lever on + end heavy enemies) and got into the right spot with the change on iteration C (lever off + start heavy enemies). Can you imagine how much user testing I needed to find and confirm it manually?

I know it's funny (look at the solo developer going crazy excited about a conversation with Claude), but the conversation part is not the most important thing. The most important thing is that now I can simulate user BEHAVIOURS, and if I want to go really crazy I can feed those personas to Claude-in-Chrome and WATCH them play.

It goes without saying, but similar experiments can (and maybe should) be run on any web project. Create an unconfident user who reads everything they can before making a product choice. Create a non-native speaker who gets confused by labels. Create a user with a visual impairment. Create a user who doesn't care about the price. And let them use your product in a synthetic way to tell you things.

Above is an example of the report on the story progression by the default player. It shows the scene that was a wall at first 2 builds and how far wall moved with fight structure changes.

Meanwhile, I will go and create a casual user who builds no sigils at all and see how far they get in my game. This is so cool!