Questions related to development using Palantir Foundry's Data Connection application are appropriate to use here.
Data connection is Foundry’s tool to perform and schedule large uploads of data.
Questions tagged [foundry-data-connection]
31 questions
0
votes
2 answers
Can I get the file names for synced text files in my pipeline in Foundry?
I have a bunch of text files that are synced from my raw system, I want an easy way to use the file names (in addition to the contents of the files) downstream in Foundry transforms.
I know this is possible using raw file access, but that seems…
ollie299792458
- 222
- 3
- 16
0
votes
0 answers
Data connection with SharePoint list
I am new to foundry and I am trying to set up a data connection to a SharePoint list. I know SharePoint supports REST APIs but that is about the extent of what I know. The Foundry Data connection documentation wasn't as helpful as I had hoped.
How…
Connor
- 41
- 3
0
votes
3 answers
How can I have nice file names & efficient storage usage in my Foundry Magritte dataset export?
I'm working on exporting data from Foundry datasets in parquet format using various Magritte export tasks to an ABFS system (but the same issue occurs with SFTP, S3, HDFS, and other file based exports).
The datasets I'm exporting are relatively…
ollie299792458
- 222
- 3
- 16
0
votes
1 answer
How to parse the complete set of records for a dataset through an API call?
How can I get the full dataset records through foundry API call?
I want to use the dataset in another Python application outside Foundry and using requests only first 300 rows of records are coming.
The requests API end point I have is using Contour…
Asher
- 175
- 11
0
votes
1 answer
Does Foundry Data Connection support NFS transfer?
Is it possible for a Data Connection agent to connect to NFS in order to transfer data?
Andrew St P
- 524
- 1
- 5
- 13
0
votes
1 answer
How to decrypt AES-encrypted files with Data Connection?
I'm looking for the best way to ingest AES-encrypted .csv files in Foundry. What would be the quickest way to accomplish this task?
Andrew St P
- 524
- 1
- 5
- 13
0
votes
1 answer
Data Connection - downloading files from multiple URLs in one sync
How can I use Data Connection to download files from a large list of URLs as part of a single sync?
I want to be able to do this in parallel, since I'll be downloading 1–2,000 new files per day. I also want all the files to be stored in a single…
Andrew St P
- 524
- 1
- 5
- 13
0
votes
1 answer
Does Foundry Data Connection support SFTP sources?
When using Data Connection, can I ingest files into Foundry from an SFTP (SSH File Transfer Protocol, not FTPS) source?
Adil B
- 14,635
- 11
- 60
- 78
0
votes
1 answer
What's the difference between Sources “S3 (through Hadoop)” and “S3 (Direct)” in Foundry Data Connection?
What's the difference between the two S3 source options that are available in Foundry Data Connection?
S3 (through Hadoop)
S3 (Direct)
Is one preferred for ingesting parquet files?
Adil B
- 14,635
- 11
- 60
- 78
0
votes
1 answer
Data connection - Parallel JDBC extracts failing with OutOfMemoryError
I'm trying to run a few JDBC extracts in parallel, but this fails on: java.lang.OutOfMemoryError: Java heap space.
How does Data Connection memory usage work, and how do I resolve this problem?
Andrew St P
- 524
- 1
- 5
- 13
0
votes
1 answer
How can I ingest an Excel spreadsheet with multiple tabs?
I would like to ingest Excel files in a remote folder or a SFTP. It works with CSV files but not XLS or XLSX files.
Andrew St P
- 524
- 1
- 5
- 13
0
votes
1 answer
Is it possible to sync data via Foundry Data Connection to a specific branch of a dataset?
I'd like to sync data ingested via Foundry Data Connection into a specific branch of a dataset. Is there a way to configure this kind of setup?
Adil B
- 14,635
- 11
- 60
- 78
0
votes
2 answers
What is the best way to handle zipped files in Foundry?
What's the recommended approach to integrate zipped files into Foundry? I can see 3 options:
Unzip on the box (if there is such option), and use Data Connection to ingest unzipped files
Use some Data Connection plugin (if there exists one) to unzip…
Andrew St P
- 524
- 1
- 5
- 13
0
votes
1 answer
How do I assign a Data Connection extract to a different source?
I'd like to change the source that is associated with a Data Connection extract. Is this possible, without re-creating the extract in the new source?
Andrew St P
- 524
- 1
- 5
- 13
0
votes
1 answer
In Foundry, is it possible to extract the original file names from a concatenated import?
I need to import billions of very small image files into Foundry. I understand Foundry won't handle that amount of files very well, so best practice would be to concatenate them and make the import incremental.
However, I need to be able to get the…
Adil B
- 14,635
- 11
- 60
- 78