RCurl postForm with url encoded names

Question

I am trying to submit a postForm request using RCurl from the following website, however the form attribute names are in a format that does not seem to be readable with the function because it uses brackets and single quotations.

url = "http://www5.statcan.gc.ca/cansim/a26?id=2820001"
postForm(url, MBR['GEOGRAPHY'], style = "post")

Which results in the following error:

Error: unexpected '=' in "postForm(url, MBR['GEOGRAPHY'] ="

I can append the url with the url encoded names and associate value like so:

postForm(paste(url, "&MBR%5B%27GEOGRAPHY%27%5D=2", sep=""), style = "post")

However this only allows me to select one geography, whereas in reality I want to select multiples (i.e. values 1,2,3, and 4). There are other form names, but the same issue applies so I have simplified it using only geography. I would also include the action%3Aa47 name to submit the form.

If anyone knows how I can still use the postForm function with these [] form names or a way to use the work around but select multiple values I would be very appreciative.

Thank you.

What is `dput(MBR['GEOGRAPHY'])`? Also, the second argument in `postForm` is `...`, which means you must name that argument. — Rich Scriven, Feb 18 '17 at 16:15
There's an R package for extracting data from CANSIM that you may find helpful: https://cran.r-project.org/web/packages/CANSIM2R/index.html (read the date on this, sorry about the ancient suggestion) — Reeza, Oct 06 '17 at 02:07

score 0 · Accepted Answer · answered Feb 18 '17 at 16:42

This may help get you started:

library(httr)
library(rvest)

POST(url = "http://www5.statcan.gc.ca/cansim/a26", 
    body = list(id = "2820001", 
        lang = "eng", retrLang = "eng", 
        accessible = "false", previewOutput = "false", 
        outputFrequency = "UNCHANGED", 
        whenConvertingFrequency = "USE_CALENDAR_YEAR", 
        manipulationOption = "DATA_AS_RETRIEVED", 
        tabMode = "customize", 
        pattern = "", p1 = "-1", 
        viewId = "1", `D1-picklist-containerL1` = "1", 
        `MBR['GEOGRAPHY']` = "6", 
        `MBR['GEOGRAPHY']` = "7", 
        `MBR['CHARACTERISTICS']` = "3", 
        `D3-picklist-containerL0` = "1", 
        `MBR['SEX']` = "1", `D4-picklist-containerAll` = "1", 
        `MBR['AGEGROUP']` = "1", 
        `MBR['AGEGROUP']` = "2", 
        `MBR['AGEGROUP']` = "3", 
        `MBR['AGEGROUP']` = "4", 
        `MBR['AGEGROUP']` = "22", 
        `MBR['AGEGROUP']` = "5", 
        `MBR['AGEGROUP']` = "6", 
        `MBR['AGEGROUP']` = "7", 
        `MBR['AGEGROUP']` = "8", 
        `MBR['AGEGROUP']` = "9", 
        `MBR['AGEGROUP']` = "10", 
        `MBR['AGEGROUP']` = "11", 
        `MBR['AGEGROUP']` = "12", 
        `MBR['AGEGROUP']` = "13", 
        `MBR['AGEGROUP']` = "14", 
        `MBR['AGEGROUP']` = "15", 
        `MBR['AGEGROUP']` = "16", 
        `MBR['AGEGROUP']` = "17", 
        `MBR['AGEGROUP']` = "18", 
        `MBR['AGEGROUP']` = "19", 
        `MBR['AGEGROUP']` = "20", 
        `MBR['AGEGROUP']` = "21", 
        smonth = "9", syear = "2016", 
        emonth = "1", eyear = "2017", 
        exporterId = "TABLE_HTML_TIME_AS_COLUMN", 
        verificationOption = "NORMAL_RETRIEVAL", 
        `action:a47` = "Apply"), 
    encode = "form") -> res

Working with the output:

content(res) %>% 
  html_table(fill=TRUE)
## [[1]]
##    Geography         Age group      2016    2016     2016     2016    2017
## 1  Geography         Age group September October November December January
## 2  footnotes              <NA>      <NA>    <NA>     <NA>     <NA>    <NA>
## 3     Quebec 15 years and over   4,208.1 4,203.9  4,178.8  4,160.5 4,112.2
## 4     Quebec    15 to 24 years     540.8   545.3    531.5    544.5   518.4
## 5     Quebec 25 years and over   3,667.2 3,658.6  3,647.4  3,616.0 3,593.8
## 6     Quebec    25 to 44 years   1,852.7 1,849.4  1,842.4  1,848.5 1,837.2
## 7     Quebec    45 to 64 years   1,663.1 1,662.1  1,658.8  1,621.8 1,612.3
## 8     Quebec 45 years and over   1,814.5 1,809.2  1,805.0  1,767.5 1,756.6
## 9     Quebec    25 to 54 years   2,807.8 2,785.7  2,779.7  2,763.3 2,753.9
## 10    Quebec 55 years and over     859.4   872.9    867.7    852.7   839.9
## 11    Quebec    15 to 64 years   4,056.7 4,056.9  4,032.7  4,014.8 3,967.9
## 12    Quebec    15 to 19 years     163.6   161.9    170.8    176.5   174.4
## 13    Quebec    20 to 24 years     377.3   383.4    360.6    368.1   344.0
## 14    Quebec    25 to 29 years     445.8   429.8    429.7    432.3   435.6
## 15    Quebec    30 to 34 years     470.8   464.8    457.0    460.4   447.7
## 16    Quebec    35 to 39 years     481.6   501.8    508.1    506.3   515.5
## 17    Quebec    40 to 44 years     454.5   453.0    447.5    449.4   438.5
## 18    Quebec    45 to 49 years     449.9   441.8    441.5    435.1   430.8
## 19    Quebec    50 to 54 years     505.2   494.5    495.8    479.7   485.9
## 20    Quebec    55 to 59 years     449.4   456.8    457.2    446.4   441.4
## 21    Quebec    60 to 64 years     258.7   269.1    264.3    260.6   254.2
## 22    Quebec 65 years and over     151.4   147.0    146.2    145.7   144.3
## 23    Quebec    65 to 69 years      95.1    99.2     97.1     90.9    92.6
## 24    Quebec 70 years and over      56.3    47.8     49.0     54.8    51.7
## 25   Ontario 15 years and over   6,989.5 7,041.4  7,049.9  7,033.5 6,972.5
## 26   Ontario    15 to 24 years     874.5   902.8    905.7    890.5   856.8
## 27   Ontario 25 years and over   6,115.0 6,138.6  6,144.2  6,143.0 6,115.7
## 28   Ontario    25 to 44 years   3,007.5 3,013.2  3,040.7  3,039.5 3,024.8
## 29   Ontario    45 to 64 years   2,793.3 2,811.9  2,791.7  2,791.2 2,791.1
## 30   Ontario 45 years and over   3,107.5 3,125.4  3,103.5  3,103.5 3,090.9
## 31   Ontario    25 to 54 years   4,611.5 4,630.8  4,666.9  4,670.4 4,647.4
## 32   Ontario 55 years and over   1,503.5 1,507.8  1,477.3  1,472.6 1,468.3
## 33   Ontario    15 to 64 years   6,675.4 6,727.9  6,738.1  6,721.2 6,672.7
## 34   Ontario    15 to 19 years     265.0   279.0    271.6    264.2   258.2
## 35   Ontario    20 to 24 years     609.6   623.8    634.1    626.3   598.6
## 36   Ontario    25 to 29 years     773.7   761.1    766.0    761.6   751.9
## 37   Ontario    30 to 34 years     746.8   756.1    770.8    778.0   781.2
## 38   Ontario    35 to 39 years     759.3   757.3    754.3    756.2   739.4
## 39   Ontario    40 to 44 years     727.7   738.7    749.6    743.7   752.3
## 40   Ontario    45 to 49 years     733.2   745.6    741.9    747.8   750.0
## 41   Ontario    50 to 54 years     870.8   872.0    884.3    883.1   872.6
## 42   Ontario    55 to 59 years     727.8   732.9    728.6    723.9   727.7
## 43   Ontario    60 to 64 years     461.5   461.3    436.9    436.4   440.8
## 44   Ontario 65 years and over     314.1   313.5    311.8    312.3   299.8
## 45   Ontario    65 to 69 years     206.4   206.3    205.7    206.4   195.4
## 46   Ontario 70 years and over     107.7   107.2    106.1    105.9   104.4

Thanks hrbrmstr. This also worked with postForm, I just forgot the period in the .params = list() argument. — SteveM, Feb 22 '17 at 03:37

RCurl postForm with url encoded names

1 Answers1