Bad week for video games research

Industry stats, data, and bullshit

During his effort to eradicate pinball machines from the city, New York’s 99th mayor Fiorello LaGuardia liked to have himself photographed holding a sledgehammer. During his tenure from 1934 to 1945, he regarded pinball machines as a stepping stone to gambling and a scourge. He may as well have been talking about video game market research.

Since Nielsen shut down SuperData, a host of former clients have reached out to me in search of the services it had provided them. Turns out, there’s demand for reliable industry data. The pandemic’s momentum that drove games into the mainstream among consumers and investors, has created an enormous void in information and insight.

Slimy tinhorns

But when demand outstrips supply, quality often declines. In an already embattled industry category, one of the most prominent providers of mobile data just got its ass handed right back to itself due to “deceptive practices and making material misrepresentations.” This week the SEC ordered App Annie and its co-founder and former CEO and Chairman Bertrand Schmitt to pay $10 million to settle fraud charges.

It should have been more.

As someone who built and sold a data business previously, this is near and dear to my heart for a few reasons. First, in war, truth is the first casualty. In VC-backed data businesses we are not far behind. Growth at all costs is a challenge for any company (*cough* Theranos *cough*). But this counts double (see what I did there?) for organizations that feed data to investors and purport to provide market transparency. I’ve worked at firms before where a candid opinion often died a premature death because it would complicate some never-ending negotiation with a glass chinned industry partner. Raising a bunch of investment money has a comparable effect on a data business. Fealty to financial impetus rather than factuality dooms research firms.

Second, already do many dislike research firms, and now App Annie made that a lot worse. It is a huge disservice to the integrity of well-meaning and hard-working analysts out there that try to separate fact from fiction. Often regarded as some brown-nosing expendables that contribute little to the creative vision, market research, data analytics, and thought-leadership often live on the fringes. Especially in creative industries, there is little love for industry insight. 

“People don't know what they want until you show it to them. That's why I never rely on market research. Our task is to read things that are not yet on the page.”

Thank you, Steve Jobs. But even Mr. I-don’t-use-market-research proved unable to contain his excitement about mobile gaming in 2010 when his firm had not “set out to compete with Nintendo or Sony on their PSP, but we are now a significant part of that market.” How did he know? No one knows. He thought so differently.

Lying about how you got your data and its accuracy is tantamount to sabotaging an entire industry of honest analysts. In this case, App Annie and Schmitt matched confidentially obtained information with the outcome of its statistical model, which it referred to as “error-halving” in lieu of the overhaul proposed by its chief data scientist. The firm had previously promised that it would not use confidential information but did so anyway. It lied about its method and, worse, about its handling of material information.

Granted, data is often far more scarce than you’d think. Even publicly traded firms are extremely parsimonious in doling out company performance data. The games industry and entertainment at large is a massive poker game where everyone is curious what the others are holding, but no one wants to share. The intermediaries that try to fill that vacuum play a critical role. A role that just got harder.

Well dressed and living in luxury

One of my colleague at NYU generally warns his students to go beyond just Googling information when they are looking for an answer. This is especially important when it comes to valuations and market sizing. One such infraction emerged this week after Naavik partnered with Bitkraft and produced a rather unfortunate headline: “Gaming Industry Nearly Twice as Large as Reported, at $336B.”

Yeah, not so fast.

Research, in my experience, consists of establishing a primary data source (e.g., consumer survey data, retail-level transactions) and using that information to describe the market, provide insights, and prognosticate the future. It is not super complicated, really.

Contemporary market research is so seemingly abundant and accessible that making any effort becomes meaningless. Why bother to use an actual data source when you can just copy and paste this cool looking infographic that god-knows-who put together?

There is no quicker way to double-count market data than basing your estimates on someone else’s estimates which, it turns out, are also based on estimates. For example, in the Naavik/Bitkraft report, we find an estimate for $11.5 billion for the annual “trade of video game systems, games, or virtual items through unintended or unofficial methods,” or grey market sales. That is quite the assessment.

According to the footnotes, they arrived at this estimate by combining growth estimates (ok, I guess) and a 2015 report that was “Powered by Deutsche Börse” (sounds official). There we find it was really put together by one Alexander Hoeptner who was CEO of a firm called Swapster, which positioned itself as “a neutrally organised on-exchange market” and was in business for a total of six months. The 15-page report talks mostly about Swapster and its offering.

Moving on, Mr Hoeptner (whose credibility, I should remind you, is not in question here) footnotes his finding by sourcing, among others, SuperData. I spent an hour searching all those old hard rives I keep in my basement looking for the “Virtual Goods Market Size 2014 – 2020E” but the closest I could find was this talk I gave in 2014.

I’m flattered that one would have such faith in me and the durability of my forecasts. But are we really to believe that an industry presentation from seven years ago is the *best* information available and sufficient to support the thesis that, contrary to popular belief, the games industry is really double the size? That’s the evidence?

Penny thievery

And speaking of evidence, we obviously all spent the weekend reading the 185-page ruling by judge Yvonne Gonzalez Rogers on the dispute between Apple and Epic Games. (If not, plenty of smart writers have covered the case.)

It is filled to the brim with excellent information on both Apple and Epic, an in-depth look at the the economic underpinnings of the games industry, and, finally, a clear perspective on why antitrust regulation in the platform era is going to be tougher than it looks.

Of course, the outcome had been somewhat predictable: Epic was never going to win this, but Apple was still at risk of losing. Epic proved unable to convince the judge on 9 out of its 10 claims against Apple, which promptly claimed victory. Subsequently, Epic has already paid the $6 million to Apple as ordered by the judge. Apple, on the other hand, is no longer allowed to prevent game developers from directing players to off-platform payment options. It is a significant crack in the wall. 

Part of the explanation why it played out this way was that Epic presented an overall sloppy methodology in its evidence. Dr. Susan Athey, for example, who is an otherwise highly respected micro-economist with impressive academic and industry credentials gave a largely theoretical account on switching costs. The only data presented, however, was sourced from “a news article, a European journal, and a biography of Steve Jobs.” Predictably, the Court did “not give those opinions much weight.” (see page 49) Ouch.

Next, Epic offered a seemingly hastily conducted survey by Dr. Rossi. The Court found its wording confusing, the study did not account for holiday spending, and the analysis focused on all app purchases and not just mobile games (p. 56). Judge Yvonne Gonzalez Rogers honored this effort with the accusation that Dr Rossi

“was more interested in a result which would assist his client’s case than in providing any objective ground to assist the Court.” 

Daaaang.

And finally, Dr. Evans presented results from a specific test (an economic test that looks at “Small but Significant and Non-transitory Increase in Price”, or SSNIP, if you must know). He did so against better judgement (p. 60). The SSNIP test is really designed for M&A scenarios. It was also based on Dr. Rossi’s aforementioned “flawed” data. And, last but not least, the test was limited to the US, not the global market. Says the Court: “Given the flaws in both the underlying survey and Dr. Evans’ calculations thereon, the Court finds this evidence wholly unpersuasive of substitution.” If you keep in mind that experts of this caliber generally charge well over $1,000 per hour, the only credible conclusion you can draw is that Epic spent a lot of money that yielded a measly 1 out of 10 result.

It is important to remember that Apple is the only one in this whole conversation who has access to relevant information. By and large, even the judge had to make some assumptions about market sizing and Apple’s share of the overall mobile game pie. That is precisely the problem, of course: how to regulate a platform economy without having access to all the relevant data? How can we ensure the health of an industry ecosystem without a having a read on what happens around us? Game design doesn’t happen in a vacuum.

Between App Annie, Naavik/Bitkraft, and Epic, you really do get a sense that due diligence doesn’t matter. Who cares what the numbers say, as long as they say what we want them to say?

Don’t be lazy. Do the work.