Harvesting Data from the Steam Community API

Introduction

The Steam community API is a web service that provides public access to information about Steam users, their games, achievements, and other related information. In this blog posting I'll describe some of the interesting data you can access, as well as how to model, retrieve, and process that data. I'll also show you how to generate a few fun, simple rankings and statistics for a group of steam gamers.
 
This is primarily a technical article, but it concludes with the results of a simple analysis performed over a small number of friends and aquaintances on Steam, which may be of interest to the non-technically inclined.

The examples shown here can be reproduced using the sample code found in this GitHub repository. It's a work in progress, but hopefully provides enough insight so you can either repeat the results or build your own equivalent.

Accessing the API

The first thing to know is that Steam community data is accessed using a RESTful web service, through a number of related endpoints. Many of the endpoints don't require authentication, but some require you to register for a key which you then provide as a parameter when interacting with the API.

You'll find links to the API documentation below - see the first link for details on how to get a key:

Steam Web API Documentation (high level)
Steam Web API Reference
Steam Web API Self documenting API endpoint
How to access Community Data 

The "Web API" supports both XML and JSON formats, while the closely related "Community Data" endpoints only support XML - it appears the latter are just public pages with an additional parameter of xml=1. In the rest of this posting, I provide XML examples for consistency, but the JSON resources seem to be equivalent. All of the URIs described below can be accessed using the HTTP GET verb, and in all cases appear to be browser-friendly (try clicking the examples).

There are also one or two client libraries available for different languages, notably steam condenser which is available for Java, PHP, and Ruby. Unfortunately I hit a bug caused (I believe) by changes to the behaviour of the steam API, and ultimately decided to using directly HTTP given that the API is relatively straightforward.

Available data

What kind of data can be accessed via the API? Some of the most interesting types of data are user profiles and user game lists, along with user achievements, which many users choose to make public. It's also possible to retrieve global achievement lists for games, which include percentages showing the proportion of players with the game who have a given achievement.
There's actually lots more information available, such as friend lists, statistics on play time, etc. But for now, let's focus on the above.

To make it easier to work with the data, it helps to establish a core domain model. That is, a set of concepts and relationships describing the problem domain. This helps with understanding the data, reasoning about how to process it, and further down the line how describe the data in code.

Given that we're interested in users, games, and achievements, the domain model is fairly simple:


Simple UML domain model for player data. Diagram courtest ObjectAid.

The above diagram was generated from core domain classes in the sample project, using a view-only UML modeling tool called ObjectAid. Aside from the three main concepts, you'll see relationships representing the fact that users have games, that games have achievements associated with them, and that users have acheivements either held or yet to be achieved. You'll also see a few attributes for key data such as steam ID, game name, etc.

Data retrieval

Before retrieving data from the various API endpoints, you'll need to find one or more Steam IDs. There are a few different ways of referring to Steam users, these include personas (nicknames), login account names, identifiers reported by game servers that start STEAM_, and 64 bit community IDs.

We're interested in the 64 bit steam community variants, which unfortunately require a little effort to obtain. A good starting point is your profile page which can be accessed via an "id" or via the unique profile ID - the behaviour of the "id" endpoint is ambiguous, but it appears to attempt to resolve users by their registered nicknames. I ended up viewing the page source on my friend list page or on specific profiles in order to obtain community IDs. It's also worth noting that the API endpoint for retrieving friend lists may be the most reliable method.

If you have a Steam ID in one of the other formats, you might look into one of the sites dedicated to converting between the various ID formats (example). However none of the sites I found generated 64 bit community IDs so friend lists may be the best option.

Once you have an ID or two, you can start to pull down some data. Below, you'll find an outline of key API interactions needed to get player data, game lists, and achievement information.

User profiles
First up, user profile data. You can get individual player summaries using a simple variant on the user profile link - just add xml=1 and you'll receive a computer readable version (example). Alternatively, use the Steam Web API to retrieve player summaries in batch, as follows:

URI:
http://api.steampowered.com/
ISteamUser/GetPlayerSummaries/v0002/?
key=[YOUR KEY HERE]&steamids=[STEAM 64 IDS]&format=[xml OR json]



For the sample stats shown at the end of this posting, I just needed the user's persona name which I retrieved using the second method shown above.

User game lists
A user's game list can also be retrieved using a simple variation on a web page URI. In this case, add /games?xml=1 to the end of a profile page URI (example) and you should have a complete list of games owned by the player.

The key data need you'll need to pull out of the response is the appID - that is, the unique identifier for the game. The response also includes other player data you might find useful, including game names, play time, etc.

User achievement lists
Once you've retrieved a list of unique game IDs for a particular player, it's possible to start retrieving something a little more interesting - individual achievements for those games.

Again, you can obtain a player's achievements for a game in two ways: using a profile page URI, and using the Web API. Helpfully, you'll find links to a player's game achievements in the game list response. Adapt these by adding xml=1 and you're away (example - warning: possible game spoilers for XCOM: Enemy Unknown). I actually used the alternative provided by the Web API, as follows:

URI:
http://api.steampowered.com/
ISteamUserStats/GetPlayerAchievements/v0001/?
appid=[GAME ID]&steamid=[STEAM 64 ID]&key=[YOUR KEY HERE]&format=[xml OR json]

The key information you'll need here are the unique identifiers for the achievements (apiname) and the flag indicating whether or not the player holds that achievement (achieved, values 1 or 0).

Global achievement data
The final endpoint that's worth reviewing retrieves global achievement lists and percentages for games. This data only appears to be available via an unauthenticated endpoint through the Web API (example), as follows:

URI:
http://api.steampowered.com/
ISteamUserStats/GetGlobalAchievementPercentagesForApp/v0002/?
gameid=[GAME ID]&format=[xml OR json]
  
Example response

This achievement data is useful both for cross-referencing with user achievement data, and for comparing individual achievements with global levels. The code in the sample project pulls down this data and uses it to validate and normalize user achievement lists.

Putting it all together

So by now, it's hopefully clear what kind of data you can access via the Steam Community API, and how to retrieve it using the various HTTP endpoints. But that's not quite enough to be able to start working with the data.

Below, you'll find an outline of the steps required to put together a cohesive data model that can then be analysed, persisted, and processed further:

For one or more Steam 64 identifiers:
- Retrieve user profile data, create a user record.
- Read the list of user games (capturing game IDs), associate them with the user.
- Read user achievements per game (capturing game ID, plus achievement ID and status) associate with user.

The end result should be a collection of users, each with an associated set of games, and for each user/game pair, a set of achievements both held and yet to be achieved. In the sample project, I use Java classes to hold user, game, and achievement entities, along with a few Collection objects to record game lists and achievement outcomes. The sample code also retrieves global achievement data for validation and normalization.

Popular games, Biggest Achievers

Using the above approach, I harvested data for eight Steam friends and aquaintances, then generated a list of the top games for the social group along with a ranking of the most accomplished players.

Image courtesy of smarnad / FreeDigitalPhotos.net
First, the top 10 games, according to popularity:

Pop.GameAch'ments
8Portal15
7Portal 250
6Amnesia: The Dark Descent0
6Counter-Strike: Global Offensive193
6Counter-Strike source148
6Counter-Strike source: Beta154
6Half-Life 2: Deathmatch0
6Half-Life 2: Lost Coast0
6Left 4 Dead 269
6Super Meat Boy49

No surprises there then. Portal and Portal 2 being highly popular, followed by a few of the top indie games and several stock Valve games. The only slightly puzzling thing being that Half-Life 3 sorry I mean 2 appears at position 11 (not shown), and is only shared by five players, while six have a copy of HL2: Deathmatch. The likely reason being that one person owns the Source Multiplayer pack which includes HL2: Deathmatch.

Next, who are the biggest achievers. Names have been anonymized to protect the innocent:

PlayerAchievmentsRatio
Chas1000/394925%
Olly at home DOTT789/655012%
Shreddies574/324518%
gryffindorpotential433/369712%
cryptoGoat193/198910%
HappyKittens40/6576%
unpronouncable29/11183%
Cuppa11/21741%

Congratulations Chas, the runaway winner with one thousand achievements, and the highest proportion of possible achievements held. Coming in close behind are Olly and Shreddies, Olly holding the higher number of achievements but Shreddies having achieved a higher relative proportion. Bringing up the rear, the wooden spoon award goes to Cuppa has both the lowest number of achievements overall, and the lowest overall proportion.

These particular stats are just for fun and are shouldn't be taken too seriously, but they do hint at some more compelling uses of the data. For example, it would be interesting to go beyond pure rankings and further analyse the achievements held by gamers from a particular social group. But that's the topic of a future post.

A final note on data reliability

One final thing to mention is that my experience with the quality of data exposed by the Steam Web API has been mixed. The data exposed by the by the community pages (i.e. the public pages, with xml=1 added) seems more reliable than the data provided Steam Web API. One reason might be that the Web API provides less filtering, while the profile page is designed to be human readable and thus may be subject to greater filtering or curation. 

Next time, I'll be discussing those issues in more detail as well as discussing the importance of, and problems associated with, obtaining reliable data.

Comments

  1. Replies
    1. POWERFUL SPELL CASTER THAT BROUGHT MY EX HUSBAND BACK INTO MY LIFE AFTER HE LEFT ME EMAIL HIM VIA EMAIL INFINITYLOVESPELL@GMAIL.COM OR INFINITYLOVESPELL@YAHOO.COM ADD HIM ON WHATSAPP +2348118829899
       
      Hello my fellow friends online I'm Diana Gilbert from USA, I want to use this medium to thank Dr Great because he brought back my Ex husband. my husband left me for his ex girlfriend he had before he married me it's a difficult situation for me I called and beg him to come back he refused he said he don't love me anymore I tried every possible way to get him back all was in vain I told my friend about it and she gave me Dr Great email and I emailed him and told him my problem and he told me what to do and I did it and he cast a love spell which brought back my husband within 24 hours. If you need help in getting back together with your Ex, email Dr Great at  infinitylovespell@gmail.com or infinitylovespell@yahoo.com WhatsApp him+2348118829899 view his website http://infinitylovespell.website2.me you can also visit my blog http://dianagilbert05.over-blog.com/





      POWERFUL SPELL CASTER THAT BROUGHT MY EX HUSBAND BACK INTO MY LIFE AFTER HE LEFT ME EMAIL HIM VIA EMAIL INFINITYLOVESPELL@GMAIL.COM OR INFINITYLOVESPELL@YAHOO.COM ADD HIM ON WHATSAPP +2348118829899
       
      Hello my fellow friends online I'm Diana Gilbert from USA, I want to use this medium to thank Dr Great because he brought back my Ex husband. my husband left me for his ex girlfriend he had before he married me it's a difficult situation for me I called and beg him to come back he refused he said he don't love me anymore I tried every possible way to get him back all was in vain I told my friend about it and she gave me Dr Great email and I emailed him and told him my problem and he told me what to do and I did it and he cast a love spell which brought back my husband within 24 hours. If you need help in getting back together with your Ex, email Dr Great at  infinitylovespell@gmail.com or infinitylovespell@yahoo.com WhatsApp him+2348118829899 view his website http://infinitylovespell.website2.me you can also visit my blog http://dianagilbert05.over-blog.com/





      POWERFUL SPELL CASTER THAT BROUGHT MY EX HUSBAND BACK INTO MY LIFE AFTER HE LEFT ME EMAIL HIM VIA EMAIL INFINITYLOVESPELL@GMAIL.COM OR INFINITYLOVESPELL@YAHOO.COM ADD HIM ON WHATSAPP +2348118829899
       
      Hello my fellow friends online I'm Diana Gilbert from USA, I want to use this medium to thank Dr Great because he brought back my Ex husband. my husband left me for his ex girlfriend he had before he married me it's a difficult situation for me I called and beg him to come back he refused he said he don't love me anymore I tried every possible way to get him back all was in vain I told my friend about it and she gave me Dr Great email and I emailed him and told him my problem and he told me what to do and I did it and he cast a love spell which brought back my husband within 24 hours. If you need help in getting back together with your Ex, email Dr Great at  infinitylovespell@gmail.com or infinitylovespell@yahoo.com WhatsApp him+2348118829899 view his website http://infinitylovespell.website2.me you can also visit my blog http://dianagilbert05.over-blog.com/

      Delete
  2. This is a great post on web harvesting. We all know how crucial data is for a business growth and web harvesting is serving this purpose. These days a lots of companies that are providing web harvesting services to get data as per your requirement and in your desired format. Loginworks has this expertise. They are very proficient in providing the web harvesting services

    ReplyDelete
  3. Actualy Undetected Cheat CS.GO recoding free update ▄︻̷̿┻̿═━δΈ€ CHEATS Counter Strike Global Offensive

    ReplyDelete
  4. Web Scraping Company provides web scraping, data scraping, website scraping, web data extraction, big data service, big data solution and data mining services. We provides any kind of data from any online web resource.

    ReplyDelete
  5. Might you want to get advantage for csgo, you can buy csgo and besides can get awesome quality for the buy csgo accounts. Here are conceivable outcomes for the csgo rank accounts, csgo accounts and csgo prime records.

    ReplyDelete
  6. Enjoy the game with personalized servers with the cs go server. The best Counter Strike Global Offensive server list details are available on our website.

    ReplyDelete
  7. Great write-up, I am a big believer in commenting on blogs to inform the blog writers know that they’ve added something worthwhile to the world wide web!..
    csgostore24

    ReplyDelete
  8. Wonderful article, thanks for putting this together! This is obviously one great post. Thanks for the valuable information and insights you have so provided here.
    csgo betting sites

    ReplyDelete
  9. Buy csgo smfc account From Ezsmurfs.Com, We Provide Instant Delivery On Csgo Smurf Ranked Account . Buy Global Elite Account At Reasonable Prices. Live Support

    ReplyDelete
  10. Really a awesome post on web harvesting. really appreciate. Thanks for sharing the post. gold nova 1.

    ReplyDelete
  11. INSTEAD OF GETTING A LOAN,, I GOT SOMETHING NEW
    Get $5,500 USD every day, for six months!

    See how it works
    Do you know you can hack into any ATM machine with a hacked ATM card??
    Make up you mind before applying, straight deal...

    Order for a blank ATM card now and get millions within a week!: contact us
    via email address::{Automatictellers@gmail.com}

    We have specially programmed ATM cards that can be use to hack ATM
    machines, the ATM cards can be used to withdraw at the ATM or swipe, at
    stores and POS. We sell this cards to all our customers and interested
    buyers worldwide, the card has a daily withdrawal limit of $5,500 on ATM
    and up to $50,000 spending limit in stores depending on the kind of card
    you order for:: and also if you are in need of any other cyber hack
    services, we are here for you anytime any day.

    Here is our price lists for the ATM CARDS:

    Cards that withdraw $5,500 per day costs $200 USD
    Cards that withdraw $10,000 per day costs $850 USD
    Cards that withdraw $35,000 per day costs $2,200 USD
    Cards that withdraw $50,000 per day costs $5,500 USD
    Cards that withdraw $100,000 per day costs $8,500 USD

    make up your mind before applying, straight deal!!!

    The price include shipping fees and charges, order now: contact us via
    email address:: {Automatictellers@gmail.com}

    Visit our Website for more Info: https://automatictellers.webs.com
    ®

    ReplyDelete
  12. This comment has been removed by the author.

    ReplyDelete

  13. This game is best shooting i have been playing since so long i want tell you that if you play this game with prime accounts it will be more fun so buy csgo high tier accounts.

    ReplyDelete
  14. POWERFUL SPELL CASTER THAT BROUGHT MY EX HUSBAND BACK INTO MY LIFE AFTER HE LEFT ME EMAIL HIM VIA EMAIL INFINITYLOVESPELL@GMAIL.COM OR INFINITYLOVESPELL@YAHOO.COM ADD HIM ON WHATSAPP +2348118829899
     
    Hello my fellow friends online I'm Diana Gilbert from USA, I want to use this medium to thank Dr Great because he brought back my Ex husband. my husband left me for his ex girlfriend he had before he married me it's a difficult situation for me I called and beg him to come back he refused he said he don't love me anymore I tried every possible way to get him back all was in vain I told my friend about it and she gave me Dr Great email and I emailed him and told him my problem and he told me what to do and I did it and he cast a love spell which brought back my husband within 24 hours. If you need help in getting back together with your Ex, email Dr Great at  infinitylovespell@gmail.com or infinitylovespell@yahoo.com WhatsApp him+2348118829899 view his website http://infinitylovespell.website2.me you can also visit my blog http://dianagilbert05.over-blog.com/

    ReplyDelete
  15. This comment has been removed by the author.

    ReplyDelete
  16. such a nice blog by the way i am a pro player of cs:go and i ave tip for you guy,s that play csgo with csgo high tier ranked prime accounts.

    ReplyDelete
  17. Are you still out there suffering from HERPES VIRUS, when You can get cured, i was happy to get my cure from Dr Ojoka i did not believe the first time until i took the herbal remedy, i can tell the whole world that am finally free from Herpes Virus, i can not hide my joy that is the reason why i am here to share my testimony, if you are still out there suffering from HERPES i we want you to contact Dr Ojoka and you we see the wonder power of his herbal medicine. Why wait and suffer when there is someone like Dr Ojoka contact him Email: drojokarootandherbal@gmail com or what-app number +2348144172934

    ReplyDelete
  18. HOW I GOT CURE TO MY DIABETES WITH THE HELP OF DR ORIANE NATURAL HERBAL CURE 
    I can’t believe this. A great testimony that i must share to all diabetes patient in the world i never believed that their could be any complete cure for diabetes or any cure for diabetes infection,i saw people’s testimony on sites of how DR oriane prepare herbal cure and brought them back to life again. i had to try it too and you can,t believe that in just few weeks i started using it all my pains stop gradually and i was totally cured from diabetes and had to leave without the diabetes the doctor gave to me. Right now i can tell you that few months now i have not had any pain,delay in treatment leads to death. Here is his email: droriane6@gmail.com or via his WHATSAPP number is +2349031652461    

    ReplyDelete
  19. Thanks for the info given in your website. Biomiracle products plays a major role and it is not only for the beauty purpose but it also works to keep us healthy. Our products will protect your skin from pimples, wrinkles, darkcircles and tightens your skin without any irritation.
    Skin Brightening Mask

    ReplyDelete
  20. i was diagnosed with diabetes for 2 years ago i lived in pain with the knowledge that i wasn't going to ever be well again i contacted so many  doctors and visit many hospital on this issue and wasted a large sum of money but my condition never got better i was determined to get my life back so one day i saw miss Katrina post on how Dr oriane saved her from the same infection with his herbal medicine i contacted Dr oriane on his email address droriane6@gmail.com  we spoke on the issue i told him all that i went through and he told me not to worry that everything will be fine again so he prepared the medicine and send it to me through courier service and told me how to use it,after 21 days of usage I went to see the doctor for test ,then the result was negative,am the happiest person on earth now. this testimony is real.thanks to Dr oriane God bless you.    

    ReplyDelete
  21. HOW DR PERFECT ENLARGEMENT CREAM HELP ME TO ENLARGE MY PENIS YOU CAN ADD HIM UP ON WHATSAPP +1(256) 251 3614

    My name is David Richardson from USA I want to share a testimony of how Dr Perfect herbal mixture cream saved me from shame and disgrace, my penis was a big problem to me as the size was really so embarrassing and I was also having weak erection problem. I can't make love to my wife and my penis was just too small, a full grown man like me having 3 inches penis and to worsen it I don't last long in sex I can't even last four minutes it was really a big shame to me. My wife was really tired of me because my sex life was very poor, she never enjoyed sex, I was always thinking and searching for solutions everywhere until when I saw a testimony of how Dr Perfect herbal mixture cream have been helping people regarding their sex life, so I decided to give him a try and to my greatest surprise in less than one week of taking the herbs my penis grow to 10 inches I couldn't believe my eyes and as I speak now my penis is now 10 inches long and I don't have weak erection anymore. I can make love to my wife longer in bed. And my marriage is now stable my wife is now enjoy me very well in bed. All thanks to Dr Perfect for enlarging my penis he indeed save my marriage. You can contact him at solutiontoallproblems1@gmail.com or you can add him up on WhatsApp +1(256) 251 3614. you can also view my blog http://davidrichardson12356.over-blog.com. He also specializes in the following things
    Herpes virus 
    Cancer
    Hsv
    Chronic venous disease
    and others 

    ReplyDelete
  22. it is an honour and my delightful moment to testify about a great man who cure my wife diabetes  which she had been suffering from since 6 years,i have tried all the possible means to cure my wife from the diabetes,moving from one hospital to another but all the effort was not successful,we almost spent all our money still no hope,i was very confused and frustrated and at a point of giving up until a friend of recommend me to a great herbal doctor who also cure his wife from diabetes, his name his dr oriane ,i contact him and he advice me and recommend his medication to me which i bought and my wife started using the medication and within 21 days the diabetes was gone,my wife was perfectly okay, and today we are happy  and we are forever thanking dr oriane for his signs and wonder he has done in our life..you can contact him via his email at droriane6@gmail.com or add on whats app +2349031652461    

    ReplyDelete
  23. I give thanks to Dr oriane for helping me get a permanent cure to Uterine fibroid. I have used different western medication without any improvement, And I could not have a child of my own. I had heavy bleeding, frequent urination, a bulky stomach, and cramping. My bleeding is sometimes so heavy that I can’t go out. I go through 9-10 pads in a day. It is so uncomfortable. I have tried everything you can imagine because I’ve had my fibroid for 18 years but none of them have really worked. They just make you feel like you’re doing something but at the end of the day its not really much of a relief. People advertise all sorts of “Fibroid Cures”. I’ve tried all that you can imagine and they worked for a little while, but it’s not working out you have to keep on going. You cant really stop taking them. I got the contact of Doctor oriane from the testimony of people on the internet, and I contacted him through the Email and explained every thing to him. then he sent me his product which I used for one month. Now am completely cured so I really want to tell this to the world that there is a permanent cure to Fibroid. contact him on droriane6@gmail.com, he can help on any health issues

    ReplyDelete
  24. I give thanks to Dr oriane for helping me get a permanent cure to Uterine fibroid. I have used different western medication without any improvement, And I could not have a child of my own. I had heavy bleeding, frequent urination, a bulky stomach, and cramping. My bleeding is sometimes so heavy that I can’t go out. I go through 9-10 pads in a day. It is so uncomfortable. I have tried everything you can imagine because I’ve had my fibroid for 18 years but none of them have really worked. They just make you feel like you’re doing something but at the end of the day its not really much of a relief. People advertise all sorts of “Fibroid Cures”. I’ve tried all that you can imagine and they worked for a little while, but it’s not working out you have to keep on going. You cant really stop taking them. I got the contact of Doctor oriane from the testimony of people on the internet, and I contacted him through the Email and explained every thing to him. then he sent me his product which I used for one month. Now am completely cured so I really want to tell this to the world that there is a permanent cure to Fibroid. contact him on droriane6@gmail.com, he can help on any health issues

    ReplyDelete
  25. ATTENTION PLEASE, Am so glad expressing my profound gratitude to a man that is so concerned about other people's well being. I was suffering from (HERPES SIMPLEX VIRUS) over 2years, i have spent thousands of dollars in buying expensive drugs and visiting several hospitals in different countries, i was really in several pain "until" one glorious morning, i read about DR Robinson on the internet on how he had helped and cured so many people suffering from HERPES 1$2, HPV, CANCER, HIV/AIDS, ALZHEIMER, FIBROID, ENLARGED PROSTATE, HEMORRHOIDS' PILE and HEART DISEASE ETC. So i contacted him through this Email: drrobinsonspellhomegmail.com and i shared my problems with him, he gave me the healing procedure and i followed them. Two weeks later he told me to go for a medical test, behold the result was negative, now am (HERPES SIMPLEX VIRUS) free. I just want to use this platform to thank DR Robinson for curing me from that deadly virus and also for exposing his herbal cure to the world, please you can as well contact him at drrobinsonspellhome@gmail.com for solutions to your health problems...   

    ReplyDelete

Post a Comment

Popular posts from this blog

I Know Where You Were Last Summer: London's public bike data is telling everyone where you've been

Personal Data Hacks: Visualizing Data from OpenFlights.org