1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

State of the Meta - ITS Season 12

Discussion in 'Access Guide to the Human Sphere' started by inane.imp, Apr 24, 2021.

  1. inane.imp

    inane.imp Well-Known Member

    Joined:
    Jan 28, 2018
    Messages:
    6,040
    Likes Received:
    7,177
    So we're 6 months into N4 and half way through Season 12. I'm curious to see how people judge the current state of the meta.

    To that end I've put together a survey that's designed to capture a snap-shot of people's view of what factions are strong and which ones aren't at this point of things: https://forms.gle/JBGx9wNEStJrd2UB9

    Feel free to share it widely across all the Infinity communities.

    If there's a lot of uptake, I'll post the results - both in the raw and with some analysis - and re-do the survey at regular intervals to see how people perceive the changes.

    //

    So I’ve spent the better part of May looking at the responses to my survey on the perceived state of the meta. The last response I received that’s included in the analysis was on 11 May.

    My broad intent with the survey was to first get a solid idea of what people think is good and what we think is bad: this is an common conversation with comments like “Look, <X> isn’t really as bad as people think.” or “<y> is still over-rated by the community based on N3 experience” pretty much an expected and standard part of the discourse. So what I wanted to do was provided some data to back up “well how exactly DOES the community rate X or Y?”

    That’s effectively the first question: what is the perceived strength of each faction?

    The second main question is really a follow up: ok, well what factors affect those perceptions?

    Finally, the third question I had was: what else can we see in the data?

    Unfortunately, while I have an OK answer for the first but the second question has proved much harder. Largely this issue has been one of survey design.

    Overall I had 2 main errors in designing the survey:

    First, as MANY people identified, the fact that you HAD to provide an opinion of every faction meant that the data has a significant amount of “dumping” – where respondents used “3” in lieu of “No opinion”. This is clear in the overall shape of the responses.

    upload_2021-5-24_22-4-8.png

    If you assume that the overall shape of all responses should look similar to the shape of people’s responses about their own factions it means that ~1/3rd of all “3” responses were the result of dumping. If you correct for that (reduce the count of 3s by 1/3rd) you end up with a set of responses that looks like this.

    upload_2021-5-24_22-5-18.png

    By and large I’ve resisted using the “corrected” figures (because the correction is extremely agricultural), and where I do I provide the uncorrected figures as well. This is because without further surveys I can’t really tell whether or not the overall shape of “own faction data” is similar to “all player” data and, more importantly, the rate of dumping is not consistent across the dataset. I do, however, use it later to examine when a difference between “all player” and “own faction” responses may be explained by dumping alone.

    The other significant error I made in survey design was using a 5-point Likert scale. The result of this was it SIGNIFICANTLY reduced the ability to discriminate between the MASS of factions Infinity has. In practice factions have come out at 4 different ratings, whereas with a 7-point scale I’d expect to have seen them distributed over 5 or 6 different ratings. This has had less effect on the outliers, but – as we shall see – it’s made discriminating between the mass of factions that are “Neither Strong nor Weak” essentially impossible (a fact compounded by my first error).

    I made the decision to use the 5-point scale to reduce the cognitive burden on respondents – but allowing people to choose not to answer questions would have achieved the same outcome while preserving an increased ability to discriminate between factions. The diagram below illustrates the expected distribution of answers, which should see most of the “Strong” and “Neither Strong nor Weak” factions (median score of 3 or 4) split between “Neither Strong nor Weak / Slightly Strong / Strong” (median score of 4, 5 or 6). As you’ll see below, most of the factions fall between “Strong” or “Neither Strong nor Weak” so adding increased discrimination would be useful.

    upload_2021-5-24_22-6-44.png

    All that being said, I still think the exercise has shown some interesting outcomes. Thanks to everyone who contributed, and if this is well received I’ll look at correcting my errors and doing another survey at the end of the season.

    Raw data - and my horribly excel formulas can be found here. https://docs.google.com/spreadsheets/d/1EGP1CU1jMLvDIHpg3ER2xNB9vZPYyg4S1MxVzvn_S8s/edit?usp=sharing

    DEMOGRAPHICS

    I've just taken these straight from the forms page, so you can see what the raw data looks like.

    upload_2021-5-24_22-8-3.png
    upload_2021-5-24_22-8-16.png
    upload_2021-5-24_22-8-25.png
    upload_2021-5-24_22-8-39.png
    upload_2021-5-24_22-8-52.png
    upload_2021-5-24_22-9-0.png

    upload_2021-5-24_22-9-40.png
    So what: the answers skew online, skew experienced, skew Aussie, and lean competitive. Basically, exactly the sort of people you'd expect to answer a survey about the State of the Meta in a year where there's a pandemic and Australia and New Zealand are some of the only places running in-person games.

    Note, in total I had 226 answers. 1 of those answers came in late after I'd taken the data away to play with, and 1 I threw out from later analysis. The 1 answer I discarded stated they played in Antartica, played FRRM - but what convinced me to discount it was that the answers were all 1s and 5s, but in obviously trolly ways (IE FRRM was a 5 as was ISS, but Nomads, CA, Aleph, Haqq, YJ etc were all 1s).

    OVERALL PERCEPTIONS

    One of the issues with using a Likert-Scale (and in particular one with a small range) is that the distance between 2 points on the scale is not consistent. That is one players assessment of how large a difference there needs to be between two factions to label one strong and the other weak is potentially different to another players. This means treating the answers to a Likert-Scale numerically and determining things like mean or running regression analysis needs to be carefully considered. By and large I’ve tried to resist doing this; however, there’s a couple of times where the interest of the answer has outweighed concerns with pure accuracy (the consequences of making an error essentially don’t exist).

    For the overall “Tier list, in faction order” I’ve assigned them to the different ratings based on the median of that faction’s own players’ responses and used the mean of those responses to assign order. I’ve used “own faction” median rating as it accounts for dumping effects and “own faction” mean as it allows me to order factions within ratings. However, I caution people to be take the order with a grain of salt: for many, if not most, comparisons within a rating differences will not be significant and – as I explain above – is based on tenuous logic, I include it simply as a point of interest.

    Very Strong - Nomads, CA, HB

    Strong - CJC, ASA, Aleph, YJ, Shas, Haqq, Tohaa, OSS, VIRD, WhiteCo, Kosmo, Ariadna, O-12, TJC, Spiral, JSA, MO, NCA, TAK, PanO

    Neither Strong nor Weak -
    IA, White Banner, BJC, SWF, FoCo, Dahshat, MAF, Onyx, StarCo, CHA, SP, Ikari, Ramah, QK, Starmada, ISS

    Weak - Druze, USARF, FRRM

    Very Weak - None

    Below I’ve included the player’s assessment of their own faction as compared to all players and players who don’t play that faction.
    upload_2021-5-24_22-43-48.png


    There's a few interesting things on this graph - but one of the things it does show is that Nomad players rate their faction significantly higher than non-Nomad players: this is what puts it into top spot over CA.

    But, overall, the assessment of all players and that faction’s players are reasonably congruent – there are however some notable exceptions. The nature of the using a median of responses means that if there is a difference between two subsets it will almost always be significant – the issue tends to be the other way, significant but more subtle distinctions can easily be missed.

    Drilling down into the data does show a few interesting things.

    First, let’s look at the only two factions where the median of players’ responses was LOWER than the median of non-players’ responses: Druze and USARF.
    upload_2021-5-24_22-17-10.png
    upload_2021-5-24_22-17-18.png


    My first thought was the difference was likely to be due to dumping effects by non-players raising the assessment of “weak” factions, and while that’s certainly part of the story it doesn’t provide all of the answer. Even once we apply a correction of dumping (discarding 1/3rd of all “3” responses) we can see that the shape of the data remains significantly different between all players’ scores and Druze/USARF scores. Fundamentally, it appears that the experience of playing these factions correlates with assessing them as being weaker.

    We see almost the opposite with the three PanO armies that saw a difference between players’ perceptions and non-player’s perceptions.

    upload_2021-5-24_22-18-16.png
    upload_2021-5-24_22-18-23.png
    upload_2021-5-24_22-18-29.png

    Even accounting for dumping, the difference between PanO player’s assessments of their factions and non-PanO’s players’ assessments is striking. Personally, I find this interesting because I’ve long held that PanO style builds can be strong in the hands of someone who knows how to use them well but lack the tools that players more used to other factions view as basic requirements: such a theory could explain the discrepancy between PanO players’ thinking and non-PanO players’. However, the fact both Nomads and White Company fit this same pattern undermines the argument. Nomads in particular the trend for players to rate their faction higher than non-players is quite visible: notably it’s in a faction that has very little in the way of dumping effects, so it must be caused by something else.

    upload_2021-5-24_22-18-45.png
    upload_2021-5-24_22-18-50.png

    TAK, Kosmoflot, Spiral, TJC and JSA are different again. By and large they appear of have suffered from dumping effects suppressing the non-player scores. Removing those effects results in a non-player median score equal to the players’ median score.
    upload_2021-5-24_22-19-7.png
    upload_2021-5-24_22-19-12.png
    upload_2021-5-24_22-19-18.png
    upload_2021-5-24_22-19-25.png
    upload_2021-5-24_22-19-30.png

    O-12 also follows the same pattern as the other factions that have suffered from dumping effects; however, the most popular faction I would expect to see lower dumping effects. If the dumping rate is 10% or less than (ie 10% of all “3” answers for O-12 should be “No response”) then we’d see a difference between the all-player median and the median of O-12 players’ responses.

    upload_2021-5-24_22-19-43.png

    The takeaway – most notable in PanO, where perceptions for 3/5 of its sectorial see significant disparity between PanO players’ opinions and respondents as a whole, – is that there is significant difference in players vs non-players’ responses and, generally, players have tended to rate their own faction higher than factions that they don’t play: the mean of all responses is 3.39 (Neither Strong nor Weak) whereas the mean of own faction responses is 3.5 (Strong). The only factions that buck this trend are: VIRD (3.79 vs 3.79), ISS (2.81 vs 2.61), FRRM (2.28 vs 2.15), USARF (2.58 vs 2.39), Ramah (3.04 vs 2.86), BJC (3.4 vs 3.38), Onyx (3.23 vs 3.22), Steel Phalanx (3.26 vs 3.07), Druze (2.67 vs 2.53) and Starmada (2.74 vs 2.63). This difference is only partially explained by dumping, so there’s something else going on.

    Fortunately, the answer may be in the demographics of respondents. As I pointed out earlier, respondents skewed heavily “competitive” – both in terms of what they enjoy out of the game, but also in terms of frequency of play and in particular frequency of tournament play. If people play what they perceive as strong, then we’d naturally expect “own player” ratings to bias high: the responses of players who tend to think “that faction is weak so I won’t play it” are artificially excluded.

    This is exactly what we see in the data.

    upload_2021-5-24_22-20-3.png

    There is a correlation between the amount of people who play a faction and how strong that faction is perceived to be. Amongst the respondents, approximately 45% of the difference in faction popularity can be ascribed to a correlation with perceived faction strength. This correlation is strongest amongst players to answered that they “like honing lists and practicing for competitive play” (~51%). So that data supports the statement “factions perceived to be strong attract more competitive players.”

    A question this raises is whether the perception of factions perceived as being stronger is due to having more competitive players or do competitive players flock to factions perceived as stronger? Unfortunately, I don’t have the data that can answer this question. Intuitively I’d suggest it’s both: competitive players join factions perceived as strong and factions with a large amount of competitive players are perceived as being strong.

    Conclusion

    There's about a thousand other ways I could cut the data - and if you did through the excel spreadsheet you'll see some of it. But ultimately, I got stimied by the 5-pt Likert scale. With 42 factions it's very difficult to spot significant differences between factions with so few points of difference.

    If you all like this and find it interesting, I'll look at re-doing it at the end of the Season, albeit with better survey techniques.
     

    Attached Files:

    #1 inane.imp, Apr 24, 2021
    Last edited: May 24, 2021
  2. Sergei Simonov

    Sergei Simonov Well-Known Member

    Joined:
    Dec 2, 2017
    Messages:
    107
    Likes Received:
    650
    It's probably better if you don't require grading all factions, and set the survey to only require grading factions that a respondent actually plays, otherwise you have people like me with no experience with many factions being forced to say whether it's weak or strong based on whatever the hearsay on the forums is
     
    WiT?, Elric of Grans and dhellfox like this.
  3. inane.imp

    inane.imp Well-Known Member

    Joined:
    Jan 28, 2018
    Messages:
    6,040
    Likes Received:
    7,177
    "I haven't had the chance to play N4 yet" is a valid option. Hot takes on factions you haven't played are expected.

    It's partly why I asked about "what media do you consume" - you'll have had some basis for those opinions, whether it's from gameplay or not it's interesting to me.
     
    #3 inane.imp, Apr 24, 2021
    Last edited: Apr 24, 2021
    Hachiman Taro likes this.
  4. Hachiman Taro

    Hachiman Taro Inverted gadfly

    Joined:
    Jan 16, 2018
    Messages:
    1,089
    Likes Received:
    1,991
    Also your opinion on what youve played against.
     
    inane.imp likes this.
  5. inane.imp

    inane.imp Well-Known Member

    Joined:
    Jan 28, 2018
    Messages:
    6,040
    Likes Received:
    7,177
    With enough responses it's possible to do things like "players who play Vanilla Nomads or a Nomad sectorial, in general, rate Ariadna higher than players who do not play Vanilla Nomads or a Nomad sectorial". That's interesting.
     
    Elric of Grans and Hachiman Taro like this.
  6. Kiwi Steve

    Kiwi Steve Well-Known Member

    Joined:
    Mar 20, 2018
    Messages:
    117
    Likes Received:
    185
    There were quite a few factions on that list I haven't heard anything about nor played, so in the absence of any option to say I hadn't played I rated them a 3.
     
  7. chromedog

    chromedog Less than significant minion

    Joined:
    Nov 24, 2017
    Messages:
    1,364
    Likes Received:
    2,642
    I've played 4 games of N4.
    Not a valid enough statistical sample to use that for any kind of research.
     
    xagroth, nazroth and dhellfox like this.
  8. inane.imp

    inane.imp Well-Known Member

    Joined:
    Jan 28, 2018
    Messages:
    6,040
    Likes Received:
    7,177
    That’s why I’m not asking “based on your experience playing N4, rate them”, whatever you use to form your opinion is fine. Whether that’s games played, chatting to people online or just reading lists.

    My hope is that I get enough responses so that it’s meaningful to compare the opinion of people like you (played less than 1 game a month in N4, played few or no tournaments) with the opinion of people who’ve played a lot of N4.
     
  9. Mahtamori

    Mahtamori Well-Known Member

    Joined:
    Nov 23, 2017
    Messages:
    12,018
    Likes Received:
    15,302
    I'm really missing a "no opinion/not enough encounters". There's so many factions and it's the details that makes a faction strong or weak (usually) so it's hard to just look at a faction and form an opinion.
     
    chromedog, nazroth and Elric of Grans like this.
  10. inane.imp

    inane.imp Well-Known Member

    Joined:
    Jan 28, 2018
    Messages:
    6,040
    Likes Received:
    7,177
    Given how common that complaint is I'll certainly consider it if I do it again. I don't really want to change mid-stride though.

    Part of it is that I think the question of low-information assessments is interesting, so giving people an out for going "yeah, I don't play them so I don't have an opinion on whether they're strong or weak" means I lose that data. When has having insufficient information stopped a lot of people forming an opinion?

    Mostly I'm not too concerned as factions for which people don't have a strong opinion will see regression to the mean, so will end up "meh, somewhere in the middle". But what I'm hoping it does is force people to actually think a little about factions they've only heard about, which gives me a better data on potential group-think. This does mean that it's obscure the few highly-accurate assessments made by people with a broad experience playing lots of different factions... but honestly, I don't think I have enough data to pull out those people in a meaningful way.
     
    Mahtamori likes this.
  11. Mahtamori

    Mahtamori Well-Known Member

    Joined:
    Nov 23, 2017
    Messages:
    12,018
    Likes Received:
    15,302
    Certainly true, but I think there's like three different types of answers grouped together that's causing pollution and of those the "I have an opinion based on few or no games" is honestly mostly noise (e.g. I inputted an opinion on Acon only because I was forced to have one to submit the poll). Opinions based on many games or based on playing the faction are interesting on their own, but can probably be mashed together if that'd scope creep too much - that being said I'd love to see the difference between people who play Nomads and people who don't regarding Nomad factions.
     
  12. dhellfox

    dhellfox The keeper of the Forgotten

    Joined:
    Sep 21, 2019
    Messages:
    708
    Likes Received:
    625
    there really needs to be "i haven't faced them/ not enough games to make a fair assessment" option on the faction power scale section.
     
    chromedog likes this.
  13. QueensGambit

    QueensGambit Chickenbot herder

    Joined:
    Jan 31, 2019
    Messages:
    2,213
    Likes Received:
    3,456
    Eh, I took it in the spirit it was offered, and gleefully made uninformed guesses as needed. What's the worst that could happen?
     
  14. RobertShepherd

    RobertShepherd Antipodean midwit

    Joined:
    Jan 2, 2018
    Messages:
    2,048
    Likes Received:
    4,191
    *monkey's paw finger slowly curls closed*
     
  15. Hachiman Taro

    Hachiman Taro Inverted gadfly

    Joined:
    Jan 16, 2018
    Messages:
    1,089
    Likes Received:
    1,991
    Surely 'What do people who dont really know what they are talking about think' is valuable information to have.

    I would have liked a self assesment of skill in there too. Vaul recently asked something like 'When I attend a competive tournament I generally place Top 3 / Top third / middle third /bottom third' to sort players in a narrative league. Something like that would be interesting, giving info like 'people who self select as high ranking competitively favour these factions disproportionately'
     
    nazroth and inane.imp like this.
  16. RobertShepherd

    RobertShepherd Antipodean midwit

    Joined:
    Jan 2, 2018
    Messages:
    2,048
    Likes Received:
    4,191
    I like how the Johnny-Spike-Timmy question handles that, personally. Putting 'how do you place' feels a bit on the nose for a few different reasons and I'm sort of sceptical of it both from an engagement perspective and also whether it's the most useful measure - but I get what you're saying.
     
    inane.imp likes this.
  17. chromedog

    chromedog Less than significant minion

    Joined:
    Nov 24, 2017
    Messages:
    1,364
    Likes Received:
    2,642
    It may not stop others, but others are not me. I wait for sufficient information before I opine.

    I like or dislike entire factions based purely upon their model aesthetics. Their "power levels" isn't really a metric I use.

    That said, 3 of those games were against the same faction with me just getting used to how my army worked (at all) let alone how to maximise it against X faction.
     
    #17 chromedog, Apr 26, 2021
    Last edited: Aug 23, 2021
    Panzerschwein, nazroth and inane.imp like this.
  18. Diphoration

    Diphoration Well-Known Member
    Warcor

    Joined:
    Aug 9, 2018
    Messages:
    1,349
    Likes Received:
    2,533
    The poll will give us "What is the perceived power level of each faction by the community" as a metric.

    Which can still be a relevant thing to know.
     
  19. inane.imp

    inane.imp Well-Known Member

    Joined:
    Jan 28, 2018
    Messages:
    6,040
    Likes Received:
    7,177
    My intent with the Johnny-Spike-Timmy, "how long have you been playing" and "what frequency of games do you play" questions to act as a proxy for competitiveness without having to explicitly ask "but are you any good?"

    The way Vaul approached it is probably decent - but it suffers from big fish, small pond issues.
     
    Teslarod likes this.
  20. wes-o-matic

    wes-o-matic feeelthy casual

    Joined:
    Dec 22, 2019
    Messages:
    633
    Likes Received:
    1,051
    It absolutely is when you’re looking to poll public opinion, by definition.
     
  • About Us

    We are a company founded in 2001 in Cangas (Spain), and devoted to design and manufacture games and figures. Our main product, Infinity the Game, was born with the ambition to satisfy the most demanding audience, offering the best quality.

     

    Why are we here?

     

    Because we are, first and foremost, players.

  • Quick Navigation

    Open the Quick Navigation