-1

I have an API call from mule to Eloqua.

I'm using http connector to connect the API, there are around 80000 records which need to fetched in to Mule ESB. But whenever it connects to that API fetching only 1000 records that all records are random pick up.

Please suggest any modifications need to be done to Http connector to fetch all the records.

I have enabled streaming also.

Thanks.

  <sub-flow name="exported-data">
    <http:request config-ref="Eloqua_BULK_API" path="#[flowVars.'export-data-contact']" method="GET" doc:name="Get All Exported Data"/>
    <set-payload value="#[dw('payload.itm')]" mimeType="application/json" doc:name="Set Payload"/>
    <logger message="#[payload]" level="Info" doc:name="Logger"/>
</sub-flow><http:request-config name="Eloqua_REST_API" protocol="HTTPS" host="${host}" port="${port}" basePath="/API/REST/2.0" doc:name="HTTP Request Configuration">   

<http:request-config name="Eloqua_REST_API" protocol="HTTPS" host="${host}" port="${port}" basePath="/API/REST/2.0" doc:name="HTTP Request Configuration">
    <http:basic-authentication username="${username}" password="${password}" preemptive="true"/>
</http:request-config>
Thiru
  • 404
  • 1
  • 16
  • 44

2 Answers2

0

You cannot set limitation to Mule HTTP requester. It is very direct, if you give proper request, then you will get response/s as produced by host/server.

To prove that, try to give request to that API from other clients like POSTMAN or SOAPUI with same parameters. If you are getting 1000 records there, then it clearly means nothing do to with Mule configuration.

Refer API docs or /ask the API owner. There may be some additional parameter/s to be send to fetch desired records.

Gnana Guru
  • 715
  • 5
  • 11
0

The maximum number of records returned in one Application API request is 1000.

The page URL parameter would be used to retrieve the rest of the records by looping through each page. The number of pages will be the total number of records divided by count. If there are 80,000 total records and you're retrieving the default count of 1000, there would be 80 pages.

The Application API is synchronous, so it's not recommended for high volumes of data. The Bulk API would always be recommended for interacting with large volumes of data. Here is a tutorial on how to retrieve large volumes of data with the Bulk API.

loupatrick
  • 106
  • 4