83

I am in charge of developing a website which should be able to show statistics from both Apple's app store and Google Play Store to clients, so they can easily see what's going on.

I have figured out some ways to get App Store's data, but the Google Play Developers statistics seem way harder to get.

I've heard of scraping, but this wouldn't be a great solution, as it would probably get broken whenever the developers console gets a major update.

I'm looking for something which would work like Andlytics or App Annie do, as an example, so I could get data with AJAX or something else (JSON format maybe?) and put it into a database.

For now, I haven't found any reliable solution (besides scraping, which seems like a unstable way to go), and this question has been asked a while ago, so I allow myself to ask it again, because maybe now there are some solutions to get around this.

All I could find was Google Play APIs, which allow me to fetch data from the public page of the app, but not from developers console, with authentication.

Any hints or help will be greatly appreciated :)

w5m
  • 2,286
  • 3
  • 34
  • 46
Cécile Fecherolle
  • 1,695
  • 3
  • 15
  • 32
  • The developer console uses an undocumented JSON API to communicate with Google's servers. You can see the requests and discover the URLs using Chrome's developer tools. This isn't trivial, but it's better than scraping, and more than likely, will remain stable longer. – 323go Jan 03 '13 at 14:41
  • Thank you for this quick answer 323go! But there is a question that remains unanswered to me : how am I supposed to do the authentication which would allow me to connect to the account? I guess I am going to need this, to gain access to the webservices and fetch data... Is this about a token? I have no clue. Anyway, thank you for the hint, I think I'm gonna end up working on something approaching, if nothing is already done somewhere else in some hidden API i haven't heard of. – Cécile Fecherolle Jan 03 '13 at 14:49
  • Seems like it's using the standard https://accounts.google.com/ServiceLoginAuth More here: https://developers.google.com/accounts/docs/GettingStarted – 323go Jan 03 '13 at 14:59
  • Thank you for your help. I've looked at Oauth 2, and tried the PHP official API for Google Authentication. After a few tries though, I haven't yet found a way to ask for authorizations to get into the Google Developers account. In this API I'm trying to use (http://code.google.com/p/google-api-php-client/wiki/OAuth2) there are just some Google services implemented, such as Analytics or Adsense. When I try the examples, it's working fine for those authorizations (the tokens asked are relevant each time) but I don't know how I could do such a thing to gain access to Developers console... – Cécile Fecherolle Jan 04 '13 at 10:38
  • Now I might have gained access to the developers google play dashboard, but I'm still not sure. Anyway I wanted to test if I could retrieve some data, in order to do so I mimicked the POST requests which are used on the developers console to bring JSON stats, the POST request uses the following URL : https://play.google.com/apps/publish/v2/statistics + a dev_acc parameter, which I filled already. Surprisingly, this URL doesn't exist, and brings me a 404 error whenever I try to do a POST request with it... I'm quite lost now – Cécile Fecherolle Jan 04 '13 at 12:51
  • Did you try to post to https or http? – 323go Jan 04 '13 at 13:57
  • So, I tried another way to make this work (didn't see your comment earlier, and I'm sorry about that) by reading the Andlytics project a bit. Right now I've managed to get a mandatory value for authenticating, which I could retrieve in a Cookie (after making a GET request to the developer console's login page) and now, following the guidelines of the Andlytics app, I'm trying to make the right POST request, so I can send the auth parameters along with this specific value. Right now it's not working, but I'll keep you posted... – Cécile Fecherolle Jan 09 '13 at 10:07
  • Great progress, Selphira. If you get it working, considering placing it on GoogleCode -- this could be helpful for many. – 323go Jan 09 '13 at 14:53
  • I've made it! I could get the list of apps and some related data, from the Developer console ! Since it's a JSON, I'll need to parse it now. And yes, I'm considering, once I get a cleaner and more complete code, to make some documentation and put this online somewhere (I don't know if Google would like that, though, so I thought of GitHub, plus, I got help from here so it would be a way to contribute) I had to do a lot of tries to get this damn data, and it might help some people not to lose as much time as I did. – Cécile Fecherolle Jan 10 '13 at 11:21
  • I don't think Google would mind, as they're already hosting http://code.google.com/p/android-market-api/ without any backlash. Just as long as you disclaim that it's unofficial. If you like, I can clean up what you have and turn it into a lib. – 323go Jan 10 '13 at 17:40
  • 1
    That would be great! I'll get back to you in this post when I have a more finished version (for now I've only gathered a few data with two POST after authenticating so there's still quite a bit to do, and I'm still thinking about the right output object model I should choose) But I'm on my way! – Cécile Fecherolle Jan 10 '13 at 17:49
  • Hi Selphira, I have the same task in front of me, and and I'm wondring what to do. Could you please post some code that can inspire me. Thanks in advance – Babak Bandpay May 15 '13 at 11:32
  • 2
    Hi, I managed to get the script working, but someone told me recently that due to some major changes in Google's service, it is not working properly right now. So, it currently needs some fixes. Also, I did make a pseudo-tutorial and posted it here but my answer got removed because it was "merely an external link"... – Cécile Fecherolle May 16 '13 at 13:57
  • @Selphira, can you please tell me which scope you used in your OAuth2 to gain access to the Developer Console? Thank you so much. – Babak Bandpay May 17 '13 at 07:47
  • 1
    @BabakBandpay In fact, I'm using a whole different auth method, since Oauth2 doesn't allow us to authenticate through the developer console's service, I'm doing it from scratch, by sending 3 HTTP requests : 1) A GET to the authentication page, to gather cookies and other information we need 2) A POST using those values + my login credentials, to get the xsrftoken and developersconsoleaccounts variables 3) Then we can make POST requests, using those two data we got, and fetch for example the apps list, etc. – Cécile Fecherolle May 19 '13 at 18:08
  • 1
    @BabakBandpay I could give more details since there are quite a lot of settings to do for these requests to work properly, but I don't think the comments is the right place to do so. Also, I'm using cURL to send the requests. – Cécile Fecherolle May 19 '13 at 18:09
  • @Selphira, thank you so much for the answers. I am considering a scraping method now. – Babak Bandpay May 21 '13 at 15:36
  • @BabakBandpay you give to much bounty which will lost if no answer satisfied you.. ^^ – Ahmad Azwar Anas May 22 '13 at 08:16
  • @AhmadAzwarAnas, really? will they be gone? – Babak Bandpay May 22 '13 at 12:07
  • @Babakbandpay you can tell me later if it won't.. ^^ – Ahmad Azwar Anas May 23 '13 at 04:16
  • @Selphira Any update? Even incomplete code would help me. I have made the get to the authentication page, stored the cookies, but the Post + login credentials is giving me problems yet. What parameters do you have to post back besides the cookies and username/password? – Innova Jun 03 '13 at 18:25
  • 1
    @Innova After you have made the first GET call, you should have the following data ready : account username and password, cookies stored from the previous GET (I personnally use CuRL), the GALX value, which can be retrieved from inside your cookies storage file (it's actually one of the cookies you just fetched). – Cécile Fecherolle Jun 03 '13 at 20:50
  • 1
    @Innova The request has the following form for me : curl_setopt($ch, CURLOPT_URL, $authUrl); curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiePath); curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiePath); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, 4); curl_setopt($ch, CURLOPT_POSTFIELDS, "Email=" . $account['username'] . "&Passwd=" . $account['password'] . "&GALX=" . $galxValue . "&continue=" . $consoleUrl); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); – Cécile Fecherolle Jun 03 '13 at 20:51
  • @Innova AuthURL = same URL you used for the GET, only it is ServiceLoginAuth instead of just ServiceLogin (last time I checked it was like that) consoleUrl = URL of the developer console (it's often changing, anyway put here the URL where you'd like to continue after the request) Let me know if you get it working, don't hesitate to ask further questions if needed! – Cécile Fecherolle Jun 03 '13 at 20:52
  • 1
    Can anyone of you answer me, how to get the permutation. I am struck at the permutation step. I can see that it remains constant for the account. But when I tried to send the same with my request. I get some encrypted response. I tried to decode it using different encoding functions but all in vain. – Navigator Mar 30 '15 at 08:09
  • @Selphira Do you have any documentation or code examples about login and getting applications from Google Play developer console? – kapa89 Apr 10 '15 at 09:33
  • @Selphira I found your tutorial. Thanks a lot for your work! – kapa89 Apr 10 '15 at 13:16
  • 1
    @kapa89 I'm glad you could find it, it's a bit dated now but I hope it will help! – Cécile Fecherolle Apr 10 '15 at 21:55
  • @Selphira..could you please let me know the procedure to get AppStore datas? – Saty Mar 09 '16 at 10:21
  • @Selphira..could you please let me know the procedure to get AppStore datas? – Saty Mar 09 '16 at 10:26

6 Answers6

22

May the source be with you.

Andlytics is open sourced and actively developed, you can probably find something useful in there :)

Robert Estivill
  • 12,369
  • 8
  • 43
  • 64
  • 2
    Actually, the crew developing Andlytics inspired me and helped (on GitHub) a lot when I was creating my script, so yeah this was useful! Even if now, I've already managed to make a working php script three months ago, I did take a close look at Andlytics' source at that time, to figure what to do. – Cécile Fecherolle May 22 '13 at 11:33
  • Not as far as I know. – Robert Estivill May 01 '15 at 15:19
  • 1
    Well, something new just came out. Not exactly realtime nor API accessible, but you can now export your data to google driver and access it programatically http://android-developers.blogspot.com.ar/2015/04/integrate-play-data-into-your-workflow.html – Robert Estivill May 03 '15 at 15:17
  • @RobertEstivill That helped a lot! Thanks. – Farbod Jul 20 '17 at 14:56
9

Your Google Play stats are published to files stored in a bucket on Google Cloud Storage (https://support.google.com/googleplay/android-developer/answer/6135870?p=financial_export&rd=1#export)

You can either download the gsutil tool mentioned in the above link and set it up as a cron job to copy the files to wherever you want them every day, or you can use the Google Cloud Storage API (https://cloud.google.com/storage/docs/apis) to go and get the files yourself.

FeichengMaike
  • 438
  • 3
  • 9
5

I know this has already been answered, but this solution deserves some extra attention. There is a really good way of getting data from the Developer console. Take a look at: https://github.com/tmurakam/googleplay_dev_scraper

I have it running on my raspberry pi and it works really good. It's pretty simple to install and further automate it with a bash script. I'd really recommend this scraper to anyone interested in getting data from the developer console! With this you can get almost all data from the developer console. I regularly scrape the downloads numbers and save them to my webpage.

peterkodermac
  • 318
  • 5
  • 13
5

It seems that Google has already allowed us to get this data through their API. Here is a page with all the available metrics that you can issue requests for.

https://developers.google.com/analytics/devguides/reporting/core/dimsmets#mode=web&cats=user,lifetime_value_and_cohorts,channel_grouping,app_tracking

Jahziel Rae Arceo
  • 111
  • 1
  • 5
3

You could use Google Analytics in both IOS and Android projects to record installs and in-app purchases. You can check it out here. It should be very easy to add google analytics to both platforms.

HailZeon
  • 954
  • 9
  • 17
  • 2
    This is not quite the solution I'm looking for, because I (and my friends' accounts which I'm doing the tests with) own a total of at least 30 apps. Plus, if I added Google Analytics, I wouldn't have any history of the statistics, it would only start the day I add it to the app. Anyway, I got the authentication and fetching of the statistics ready, but I have to finish the group project associated with this so I can do a little php lib (as proposed in the comments) with a few functions to achieve this. Please everybody just be patient :) – Cécile Fecherolle Feb 14 '13 at 08:43
1

Why not use App Annie, AppFigures (and similar) API's and let them take care of scraping/interfacing with Google Play and App Store?

http://support.appannie.com/categories/20082753-Analytics-API

http://docs.appfigures.com/

Wizche
  • 893
  • 13
  • 32
  • 1
    Actually, in my case, the point for the company I was working with (I was doing an internship there) was to build this service from scratch, not using an existing API from another website. But yeah, this could help some people with less precise goals, thanks for the tip :) – Cécile Fecherolle May 22 '13 at 17:23
  • @wizche. I want only daily installation for one of my app for both (iOs and Android) platform. Does appannie has any free api to get this data? – Manish Sapkal Sep 23 '16 at 08:29