-1

I am new to nodejs and I am trying to call an api server with url parameters stored in a text file. The text file is a large one (around 20 gb). Now how can I read that file and make urls based on the parameters and call the api server?

The api server doesn't return much information. It just returns OK or nothing. So, the response doesn't matter now. It can be a 'fire and forget' request. If we can handle response, it will be good so that I can re run failed requests (if any) based on response code or response text

Thanks

Poomalairaj
  • 4,888
  • 3
  • 23
  • 27
  • paste some sample data of the text file and URL format of the api server. Without these information, how can we guess what you are trying to process and request? – Arie Xiao Jun 27 '13 at 03:15
  • Are you trying to use node.js to read the file and submit the requests, or are you open to other ideas? Does the text file contain full URL or do they need some processing before the requests are sent? – Joe Jun 27 '13 at 03:17
  • @ArieShaw : the api server url looks like this http;//api.subbe.myserver.com/api/v3/da347987230bbacefffaaa/srt/?s=2354523402&s=344401 – Poomalairaj Jun 27 '13 at 07:03
  • @Joe : I was trying to use twisted to send multiple requests simultaneously but I was getting "too many files open" error. So increased "ulimit" and tried. Still I could not get the performance. I had to delay 3 or 4 seconds after every 5000 requests. I could not find a way to send requests without that delay (sleep). Thats why I am looking into nodejs. I am open to any technology – Poomalairaj Jun 27 '13 at 07:06
  • @ArieShaw : the query string parameters are coming from a text file. Now the file reading IO should not block sending multiple requests simultaneously. Please let me know if you need more information – Poomalairaj Jun 27 '13 at 07:08
  • The operating system may be limiting how many ports you can have open at once. If each request is sent discreetly in the traditional way, each request will need its own network port. When you hit the limit you have to wait for some of the ports to close to open new ones. Have you tried using keep-alive or pipelining to make more than one requests per connection? – Joe Jun 27 '13 at 15:09

1 Answers1

0

You want to look into FileSystem.createReadStream http://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options

Brian Lewis
  • 5,739
  • 1
  • 21
  • 28