0

I am running a scraping project from a headless browser using node.js and Puppeteer. I want to post the data to a Google Apps Script for further processing. I expect to see the data in my GAS project with populated parameters. But instead, I get the following result with only empty parameters.

https://script.googleusercontent.com/macros/echo?user_content_key=[key]

{"parameter":{},"contextPath":"","contentLength":-1,"queryString":"","parameters":{}}

Here is the GAS code that generates that response.

Code.gs 
function doGet(e){
  return handleResponse(e);
}

function doPost(e){
  return handleResponse(e);
}

function handleResponse(e) {
  var json = JSON.stringify(e)
  var textOutput = ContentService.createTextOutput(json);
  return textOutput
}

Here is the code I am using to send the request.

scraper.js 
const request = require('request');
request({
  method: POST,
  preambleCRLF: true,
  postambleCRLF: true,
  uri: postUrl,
  multipart: {
    chunked: false,
    data,
  },
},

I have verified using [RequestBin][1] that I am sending a valid POST request.

What am I doing wrong?

Let Me Tink About It
  • 15,156
  • 21
  • 98
  • 207
  • Can you provide a sample value of `data` in the script of Node.js? By the way, in your case, you are required to use `multipart/form-data` or `multipart/related`? – Tanaike Nov 14 '19 at 01:08

1 Answers1

1

Please review how you're publishing the web app.

  • Execute the app as: Me
  • Who has access to the app: Anyone, even anonymous

The URL for your POST & GET should be something like https://script.google.com/macros/s/IDENTIFIER/exec, not https://script.googleusercontent.com/macros/echo?user_content_key=[key]. The latter URL looks like a redirect from publishing a web app that is not accessible to "Anyone, even anonymous".

If that's set correctly, this request

curl --request POST \
  --url 'https://script.google.com/macros/s/IDENTIFIER/exec?param1=1&param2=2' \
  --header 'content-type: application/json' \
  --data '{"json": true}'

returns

{
  "parameter": {
    "param1": "1",
    "param2": "2"
  },
  "contextPath": "",
  "contentLength": 14,
  "queryString": "param1=1&param2=2",
  "parameters": {
    "param1": [
      "1"
    ],
    "param2": [
      "2"
    ]
  },
  "postData": {
    "type": "application/json",
    "length": 14,
    "contents": "{\"json\": true}",
    "name": "postData"
  }
}
Diego
  • 9,261
  • 2
  • 19
  • 33
  • You are correct that the URL I posted is a redirect. However, I did verify the permissions are correct as you described. I posted the `curl` command and got the response you described. Which was also a redirect. – Let Me Tink About It Nov 14 '19 at 00:36
  • @Mowzer Ah I understand now you were just saying it's the URL that you were redirected to, not where you were sending your request. I tested with the exact same Apps Script code that you posted. Now it's not clear to me where specifically you're experiencing the problem. – Diego Nov 14 '19 at 00:48
  • @Mowzer Please try navigating to `https://script.google.com/macros/s/IDENTIFIER/exec?param1=1&param2=2` in your browser (change the IDENTIFIER) and let me know what you get. – Diego Nov 14 '19 at 00:50
  • When I navigate to `https://script.google.com/macros/s/IDENTIFIER/exec?param1=1&param2=2` URL in the browser, I get `{"parameter":{"param1":"1","param2":"2"},"contextPath":"","contentLength":-1,"queryString":"param1=1&param2=2","parameters":{"param1":["1"],"param2":["2"]}}` – Let Me Tink About It Nov 14 '19 at 00:55
  • Here is a related question I just asked that would help me understand what's going on a little better. https://stackoverflow.com/q/58847822 – Let Me Tink About It Nov 14 '19 at 00:57
  • @Mowzer The response in the browser tells me that it's working. I suggest you use an API client like [Insomnia](https://insomnia.rest/) or Postman, instead of using curl. (Personally, I highly recommend Insomnia.) – Diego Nov 14 '19 at 01:16
  • I suspect the problem might be in how my data is formatted. I can get a successful result when I only post one object or one row. What's throwing me off is posting multiple rows in a single request. – Let Me Tink About It Nov 14 '19 at 01:18