Anomalies in Steam Community data

In a recent post I introduced the Steam Community API, and showed how to retrieve gamer data and perform a few simple but fun analyses.

While writing the posting, I came across several problems associated with the data that's returned. If you're thinking about using Steam Community data, it's worth bearing these anomalies in mind because of the impact they'll have on downstream processing and further analysis.

Frustratingly, the quality of the data available through the Steam Community API is quite variable - in particular there are many discrepancies between global achievement data compared to achievement data for individual players. I also came across several global achievement rates that were clearly invalid, and in some cases found that global achievement records for games were totally missing.

The net result: it's hard to trust that the data that's returned. It is still possible to analyze returned data, but you're going to need strong validation, normalization (e.g. of player achievements against a 'gold standard'), and potentially multiple attempts to retrieve equivalent data to ensure what you have is accurate.

Below, you'll find a (non-exhaustive) list of data quality issues I came across, along with some examples, and a little discussion about the problems they introduced and workarounds I used.

Disclaimer: the issues described here reflect my experience while using the Steam API to retrieve data in bulk, to allow me to analyse data for a large number of gamers. Your experience may differ - if so, please let me know.

"Test" achievements

One of the most obvious problems you might find are spurious achievements associated with games. These include a few that can easily be filtered ("null" or empty names) as well as others that are more problematic - such as many 'test' achievements. For example:

TEST_ACHIEVEMENT
TestAchievement
Achievement_Test

Those readers familiar with the Valve games Portal and Portal 2 will realize why these can't be easily filtered - many valid achievement names include some variation of the word "test", e.g. portal_escape_testchambers.

I only spotted these when querying global achievement statistics, so it's possible that problem may just be a filtering issue on that particular API endpoint.

Another example can be seen when compare my (woeful) achievements for AaAaAA!!! - A Reckless Disregard for Gravity with the global achievement data. You should see one extra achievement testo2 which a vanishingly small number of people have achieved - more than likely because it's a leftover artifact from when the game was integrated into Steam.

Out of date achievement lists

Another closely related issue is that personal achievement lists can get out of step with global achievement lists, casting doubt on the reliability of any comparisons made between player achievements.

For example while processing the global record for The Legend of Grimrock, I noticed two additional achievements in comparison to my record:

FIND_ALL_TREASURES, complete_game_normal

It's worth noting that the FIND_ALL_TREASURES achievement appears in lower case in my record, but the complete_game_normal entry was missing completely. As a result, it's necessary to normalize all achievement records for players before making any comparisons, which unfortunately means making assumptions about why entries are missing (e.g. that the game hasn't been played recently) and how to fix the data.

Interestingly, since last viewing the global stats for this game, the data has become one of the ...

Missing global records

A more severe, though in some ways easier to handle issue is that global stats for some games are just missing - though this seems to be an intermittent issue.

The aforementioned Legend of Grimrock is currently one of them, as was Civilization V a couple of weeks ago. This seems to be an API specific issue, because the equivalent website for The Legend of Grimrock shows many achievements as I write this.

It seems that obtaining global stats for games is a bit of a hit-and-miss affair, so be careful with any apparently empty lists of achievements you may see and don't assume that such responses are correct when considering further processing.

Achievements with huge percentages

The final issue I've come across is that some of the global stats for achievements are simply incorrect. For example RAGE by iD Software has several achievements held by over 730,000% of players.

Thankfully this particular issue is easily detected, and offending achievements can be filtered easily.

Photo credit: wilhei55 / Foter / CC BY

Comments

Post a Comment

Popular posts from this blog

I Know Where You Were Last Summer: London's public bike data is telling everyone where you've been

Image
This article is about a publicly available dataset of bicycle journey data that contains enough information to track the movements of individual cyclists across London, for a six month period just over a year ago. I'll also explore how this dataset could be linked with other datasets to identify the actual people who made each of these journeys, and the privacy concerns this kind of linking raises. -- It probably won't surprise you to learn that there is a publicly available Transport For London dataset that contains records of bike journeys for London's bicycle hire scheme. What may surprise you is that this record includes unique customer identifiers, as well as the location and date/time for the start and end of each journey. The public dataset currently covers a period of six months between 2012 and 2013. What are the consequences of this? It means that someone who has access to the data can extract and analyse the journeys made by individual cyclists within ...
Read more

Personal Data Hacks: Visualizing Data from OpenFlights.org

Image
A friend recently told me about OpenFlights.org , a website that allows you to record, analyze, and share personal flight data. He showed me his dataset, which contained a record of every flight he'd taken over the past 10+ years. I was keen to investigate the dataset further, and my friend was happy to provide me with a copy so I could have a play (thank you Luigi!). The end result is the following collection of visualizations created in Gephi , with a little help from R . They show key transport hubs and routes for airports, countries, and continents that my friend has visited, and demonstrate some of the fun, insightful ways you can use such personal data. If you're interested in how the visualizations were created, check out the section at the end of this blog posting where I briefly describe the technologies required and steps involved. Note: OpenFlights.org is free to use, and supported by advertising and donations. You can join me in supporting OpenFlights.org ...
Read more

Visualizing The Wheel of Time: Reader Sentiment for an Epic Fantasy Series

Image
In the following blog post, I explore reader sentiment for the Epic Fantasy book series The Wheel of Time , as expressed in user-submitted ratings on Amazon and GoodReads. If you're a data scientist (or similar), you'll probably be interested in the data analysis which includes some interesting observations about the usefulness - or otherwise - of Amazon reviews. If you're a Epic Fantasy Series reader, you'll be interested in the outcome of my analysis: I've decided to go ahead and read all fourteen books. Note -   This is a spoiler free zone .  Updated 17th Sept: charts now more accurate, not quite as pretty * Recently, I was looking for a good book to read, and a friend recommended The Wheel of Time  series by Robert Jordan . But I'd heard from a few sources that the later volumes were harder going than the earlier volumes. Having struggled with later volumes of the Game of Thrones  series (sorry George ) I was wary of starting a mammoth fourteen vo...
Read more