0

I want to get data with Rcurl, when I downloaded the web, the data I want are not in the file,how can I get it, any clue will be appreciated.

require(RCurl)
url="http://hk.ishares.com/product_info/fund/holdings/SEHK/2823.htm?ls=true&  l=en&periodCd=d"
data=getURL(url)
write(file="data.html",data)

the following data which you can see in the web is not in data.html, how can I get it?

% Net Assets    Name    Exchange Ticker     Trading Exchange    SEDOL   Market Price    
(HKD equivalent)    Shares Held     Market Value  
(HKD equivalent)    Sector   
9.17    PING AN INSURANCE (GROUP) CO. OF C  601318  XSHG    B1SVWB6     50.40       81,991,646  4,132,780,718   Financials  
6.25    CHINA MERCHANTS BANK CO. LTD A  600036  XSHG    6518723     12.41     227,242,375   2,819,477,954   Financials  
4.86    SHANGHAI PUDONG DEVELOPMENT BANK C  600000  XSHG    6182043     9.15    239,651,727     2,192,915,873   Financials
Julius Vainora
  • 47,421
  • 9
  • 90
  • 102
Ff Yy
  • 247
  • 2
  • 7
  • 12

2 Answers2

4

Your RCurl pull basically just downloads whatever you've connected to it. So, your output ends up just being whatever the HTML underlying that page is. I'm at a language disadvantage here in assisting you (I don't speak or read Chinese), but from what I can see in the page source, it looks like most all that data is called using JavaScript. You end up with copies of the actual JavaScript, not the data, when you do the getURL().

Your best solution would be to identify if there's an API of some kind that lets you pull actual data, instead of just copying the HTML itself. I'm not sure if they offer that. However, another totally plausible option would be getting the data as a spreadsheet. It looks like there's a link on the page to grab an CSV file with the data in it. You could grab that, and then use readlines() or something similar to parse it into data. I grab CSV data for work sometimes, and I use the following to take it into a data frame:

out <- postForm("https://the.api.com/api", token="abcd", format="csv")
mydf <- read.table(textConnection(out),header=TRUE,sep=",",quote="\"")
TARehman
  • 6,659
  • 3
  • 33
  • 60
2

Notice that on that page, there is a "Download" link that points to http://hk.ishares.com/product_info/fund/excel_holdings.htm?ls=true&periodCd=d&%20%C3%82%C2%A0l=en&ticker=2823.

Maybe you can use that to download the data?

I have a function called getHoldings.iShares (which will be called by getHoldings) in my qmao package that can download the holdings of iShares ETFs from the U.S. website.

I cannot find that exact ETF on the American iShares website, but this will get the holdings of the FTSE China 25 Index Fund. Perhaps you can look at the code and adapt it for use on the Chinese site.

[A Word of Warning: iShares.com changes the format of their ETF holdings spreadsheets more often than any other ETF website that I track, so this may break in the future.]

You will first need to install the qmao package

> library(qmao)
> getHoldings("FXI")
$iShares
[1] "FXI.h"

Warning message:
In FUN("FXI"[[1L]], ...) :
  FXI has some holdings with duplicate Symbols: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
> head(FXI.h)
           FXI.Weight         ISIN        Sedol     Symbol
X..             10.33 HK0941009539 6073556      941       
X...1            9.20 CNE1000002H1 B0LMTQ3      939       
X99B1G1QD3       8.21 CNE1000003G1 B1G1QD8      1398      
X...2            7.00 HK0883013259 B00G0S5      883       
X...3            5.96 CNE1000001Z5 B154564      3988      
X...4            4.12 CNE100000Q43 B60LZR6      1288      
                                   Name Market.Value Market             Sector
X..                    CHINA MOBILE LTD    474724703   XHKG Telecommunications
X...1         CHINA CONSTRUCTION BANK-H    422947063   XHKG         Financials
X99B1G1QD3     IND & COMM BK OF CHINA-H    377377292   XHKG         Financials
X...2                         CNOOC LTD    321489999   XHKG          Oil & Gas
X...3               BANK OF CHINA LTD-H    273700485   XHKG         Financials
X...4      AGRICULTURAL BANK OF CHINA-H    189355603   XHKG         Financials
           Exchange.Rate Market.Price Shares.Held
X..                 7.75        11.09    42794500
X...1               7.75         0.68   619833320
X99B1G1QD3          7.75         0.59   642996995
X...2               7.75         2.02   159155000
X...3               7.75         0.38   724190000
X...4               7.75         0.39   486089000
GSee
  • 48,880
  • 13
  • 125
  • 145