0

How can I use Data Connection to download files from a large list of URLs as part of a single sync?

I want to be able to do this in parallel, since I'll be downloading 1–2,000 new files per day. I also want all the files to be stored in a single dataset.

Andrew St P
  • 524
  • 1
  • 5
  • 13

1 Answers1

-1

This is supported using the magritte-rest-plugin. Configre the source as a map of magritte rest sources. The format of the source is:

type: magritte-rest-v2
  sourceMap:
    source_name_1:
      source-object
    source_name_2:
      source-object2

which when filled out, might look like:

type: magritte-rest-v2sourceMap:
    my_api:
      type: magritte-restauthToken: "{{token}}"url: "https://some-api.com/"another_api:
      type: magritte-restauthToken: "{{other_token}}"url: "https://some-other_api.com/"
Adil B
  • 14,635
  • 11
  • 60
  • 78