Simulations and Models

For those who haven’t clicked on my about page, my day job is a PhD student in electrical engineering.  When doing a PhD one of the things you start thinking about a lot are tools, what the most appropriate tool to measure something with, what are the trade offs to hacking functionality into one tool vs. the other.  With that in mind, and because questions about tools come up very often, this inaugural post will be about theorycrafting tools.  Specifically SimCraft, ShadowCraft and Ask Mr. Robot.

SimCraft, ShadowCraft and AMR are interesting because they are all different kinds of tools, SimCraft is a simulator, ShadowCraft is a model and AMR is an optimizer.  To begin with lets draw a distinction between simulation and model because they are the most directly comparable.

A simulation is focused on events, it moves forward through time just like the phenomenon it is simulating.  If the player needs to make a decision at this point in time a simulation makes a decision.  If there is randomness, does this ability proc crit for instance, then the simulation generates a random number.

A model by contrast is focused on mathematical relationships.  Where a simulation says a mutilate generates 2 or 3 cps by generating its own random number a model says, mutilate generates 2.3 cps (assuming 30% crit).  A model is a set of mathematical relationships that compute something about some phenomenon.

To put things in more concrete terms if I wanted to simulate how often I get 3 heads from 3 coin flips I would write a program that generated three randoms numbers, 1 or 2, a few thousand times and average up how many times I got 3 1s.  A model would simply say, the chance of 1 head is ½ so the chance of 3 is (1/2)^3 or ⅛.

Now that we understand the difference between a model and simulation the next, and obvious, question is, which is better?  The answer as is often the case is, it depends on what you want to do.

The coin toss example above shows the two weaknesses of simulations, speed and precision.  Since a simulation simulates the actual RNG the character experiences it is susceptible to RNG.  To minimize the effect of RNG you have to do a lot of simulations, for example SimCraft recommends 10K iterations for low error EP value calculations.  Simulations can be very precise given a long time or they can be quite fast but with limited guaranteed precision.

On the upside simulations are easier to write and debug.*  If a phenomenon can be described algorithmically writing a simulation can be pretty easy.  Simulations can also be very adaptable, questions about rotation issues are very well handled by simulation because simulations map close to directly to player experience.  The last thing simulations can do very well is reason about variability.  This is the least obvious advantage but a very important one, consistency is important in high end raiding, only meeting a dps check when the stars align isn’t a sustainable solution.  Obviously there will be some dps variance no matter what but being able to see the range of expected dps given correct play is very valuable.

Looking at the coin example you can see the advantage of a model, a model is deterministic, it will give the same result every time, and a model has a constant execution time, a single run gives you maximum available precision.  The downside is models are hard to write, most models make simplifying assumptions to make the model easier to write however these simplifying assumptions can mask potentially relevant behaviors.  For example the cata combat model in ShadowCraft used a time average of energy, the energy regen of AR was distributed over the entire fight time.  In practice this was incorrect and made the combat unable to properly reason about energy capping.  For many complex physical phenomenons it simply isn’t feasible to write a model because of the complexity of the phenomenon.

Another consequence of models being hard to write is models work in rotational abstractions that don’t necessarily map to how players play the game.  Whereas a simulation can allow you to simulate arbitrary rotations and encounter types models can generally only support a small number of rotation options and rotation changes may require substantial changes to a model.  Similarly SimCraft has support for more complex encounter types while ShadowCraft probably never will.

The goal of a model is to make simplifying assumptions minimally impactful to the overall accuracy.  It may be the case that you miss certain 2nd or 3rd order effects but if those effects are small enough it shouldn’t impact accuracy too much.  Of course it is hard with just a model to determine if those effects are sufficiently small.

To summarize, models are best used for rapid iteration on gear sets or gear optimization and simulations are best for exploring new rotations and effects that models do not currently support.  We in the rogue community are lucky to have both a model and a simulation tool available to us, this gives us the best of both worlds.  SimCraft can help us determine what effects are small enough to discount, drive exploration into new rotations and help verify accuracy.  ShadowCraft meanwhile can benefit from this exploration and create a more accurate tool for rapid gear iteration.

I was planning on talking about optimization and the challenges that poses but this post is long enough already so that will have to wait till next time.  Tomorrow, Ask Mr. Robot, WoW optimization tools and their weaknesses and limitations.

*The fact that simulations are easy to write does not take away from the work of the folks who work on SimCraft.  A trivial simulation is easy to write but writing a simulation with good performance can be very difficult.  The tricks and techniques for writing an efficient simulation are beyond the scope of this post but suffice it to say they can make things very complex.