Create High Level Test Framework for StandardRuleset.ResolveMoves(...)

[bvanvugt]

We're hoping to build a very thorough and thoughtful test suite for high level move resolution (using the Standard Ruleset for now).

Ideally this consumes input board state, input moves, and expected output board state from test files (probably json?).

Example:
tests/standard/1.in.json
---> contains a JSON representation of BoardState to start with + SnakeMoves to execute
tests/standard/1.out.json
---> contains a JSON representation of the resulting BoardState after StandardRuleset is applied

[xtagon]

I'd love to collaborate on this. Ideally the test data/fixtures could be made public so that the same tests can run for our snakes as the tests for the official rules.

What's the next step and how can I help?

[bvanvugt]

Yes - exactly!

Next step I think would be a single test case, maybe in standard_test.go, that loads a board state from a json file and checks the results against a second json file. Once we have one test working, we can worry about adding others.

[xtagon]

Interesting, so in theory when it's time to build the whole canonical dataset we would run the rulesets for lots of scenarios, save the results as JSON, and then store it for history? So after that exists, it's literally just a content comparison to test it. That means testing it wouldn't even have to load up the full example JSON, it could go by a hash or file checksum

[bvanvugt]

Yeah, regression testing becomes super easy. New rulesets can be implemented using test cases first. In theory yeah you could use a checksum but ordering in JSON isn't deterministic so that would need additional manipulation. Easier to just do JSON object dumps of the structs, load and run compares (which go handles well).

[bvanvugt]

(closed by accident ;))

[xtagon]

Haven't really started yet, but saving this here for reference: https://www.npmjs.com/package/json-diff (might be useful)

[coreyja]

I opend a PR for this on the repo! BattlesnakeOfficial/rules#43

[andrewdmaclean]

Thanks for opening the PR @coreyja - I have passed it over to @chris-bsnake and the team so they can give it a review and merge if possible.

[bvanvugt]

Moved this to an issue, since this is work we'd like to get done. #73 for tracking.

[xtagon]

One important part of this to consider is how the ruleset interface handles random chance. For tests you want it to be deterministic, which it is, but it's only deterministic if you pass a seed, and this is not ideal for testing because you end up hard coding examples to verify for example that a food was placed in X,Y location, which is linked to the seed and not the intent that it was randomly placed there by chance.

My suggestion is to change the ruleset interface just slightly, so that there is a separate function/method for generating events by chance than applying them, both for the initial state and for move applications. This would make it so that you could test the fully deterministic functions purely, because they would be separate steps.

Let me know if this makes sense.

(This also relates to #76)

[coreyja]

Great call out @xtagon!

In terms of code, I believe this is the only non-deterministic call in the Standard ruleset: https://github.com/BattlesnakeOfficial/rules/blob/main/standard.go#L214-L218

This is where food is potentially placed on the board, and includes the RNG. In other game modes, like Royale, there are other non-deterministic events like the 'sauce' spawns

Question (that can be answered by a close reading of the rules/code): Do all the 'random' events happen in the same 'order' in all the game modes?
For instance can there be a consistent interface where all non-deterministic events happen in a single method call/section of code?

In standard mode the food spawn currently happens as the second to last step, right before snakes are eliminated. Is all randomness confined to this spot? Or alternatively could all randomness happen at the 'end' of the deterministic move? For standard this would mean generating food AFTER snakes have been eliminated.

[xtagon]

Because randomly spawning food, moving hazards, etc. all happens during the turn, it helps to think of it as a "chance player" or "nature player". It's still simultaneous with player moves, from a game theory perspective. I think modelling "spawn new food at (x, y)" and "close in hazard from top" as moves being played by nature, just like "right" is a move by a snake, would be a great way to design this. That way the createInitialBoardState and createNextBoardState functions would simply take a set of any chance moves as well as a set of player moves, and it would be a pure function, no need for a seed or RNG. Two new functions would need to be added to the interface, something like decideInitialChanceMoves and decideNextChanceMoves which takes a seeded RNG (and the previous state for the latter), returning only the chosen moves but no state changes. The engine on each turn would first gather chance moves while gathering player moves, then apply them to the next state function like normal.

It adds some indirection but I think it is worth considering. Test cases would very easily be able to either mock out chance completely when testing state chances, or to control the seed when testing the randomness itself.