Stack Overflow
Stack Overflow

Questions tagged [aws-datasync]

40 questions
1
vote
3 answers

AWS DataSync Lambda Automation

I am automating AWS Datasync task execution for EFS-EFS across different regions. I have written a lambda code in python and want to start the execution of datasync task through CloudWatch event cron expression. import boto3 client =…
Shoaib Mohammed
  • 19
  • 1
  • 7
0
votes
0 answers

DataSync agent ran into an error connecting to AWS

We are trying to move files from EFS to EFS but they are not in the same region. We have connected the 2 VPCs in different regions with PCX and we need to create a EC2 instance hosted in source EFS as DataSync agent. We have also created VPC…
tom10271
  • 4,222
  • 5
  • 33
  • 62
0
votes
1 answer

How to use TaskToken feature with Datasync task in a step function?

I am executing a datasync task from step function using sdk and appending waitforTaskExecution at the end. But the state keeps on running even if the datasync task is completed. I came to know about callback feature $$.Task.Token which will throw a…
PPK
  • 55
  • 7
0
votes
1 answer

Execute AWS Datasync task using step function

We have a requirement to create a state machine which includes to datasync tasks followed by each other. These datasync tasks are already created and now I want to execute it using step functions one after another. What is the resource to execute a…
PPK
  • 55
  • 7
0
votes
0 answers

Task timed out when creating DataSync agent through boto3 in aws lambda

I'm trying to create a DataSync agent. When I use the console with the same parameters, it works fine and very quickly (a few seconds). However, if I want to create it by using the lambda function, the script returns a Task Timed Out Error of 900…
Ktos
  • 47
  • 8
0
votes
0 answers

Unable to establish airbyte connection from MongoDB to S3 using Octavia CLI

I used octavia list workspace connections to get the destination and source ids already configured using the airbyte UI. I am trying to generate the same connection (with same source and destination IDs) with different source and destination names…
Vaidehi Khalpada
  • 1
  • 1
0
votes
0 answers

ASW DataSync - file added during the DataSync task execution scenario

Let's assume we have some EC2 machine with the /my_files directory. In the folder, we have 100 files: 1.pdf, 2.pdf ... 100.pdf. During the DataSync task, someone added the file 101.pdf to /my_folder directory. What's the expected behavior? Is the…
barambuk
  • 113
  • 2
  • 9
0
votes
1 answer

Airbyte Missing Columns in tables

I am considering Airbyte as a replacement for singer-io because of its user-friendly UI and code reduction features. However, facing an issue where Airbyte is unable to detect certain columns in our source (Atlas hosted on AWS), which is a…
Manjul Tripathi
  • 1
0
votes
0 answers

Any way to exclude source sudirectories with AWS DataSync?

I have an S3 bucket with two directories in it: /data /app I want to sync the contents under both directories into the same destination directory on an EFS. I've setup an AWS Datasync task to do this, however when I run the task, it creates /data…
Nstevens
  • 267
  • 2
  • 8
0
votes
1 answer

Efficient way to Copy/Replicate S3 Objects?

I need to replicate Millions (one time) of S3 Objs by modifying the metadata (within the same bucket, and obj path) To perform this, we've various options mentioned below, we need to choose cost-effective method: AWS COPY requests AWS Batch…
DilLip_Chowdary
  • 841
  • 4
  • 18
0
votes
1 answer

How to set schedule of DataSync Task to "Not Scheduled" from AWS Lambda?

The boto3 documentation seems to provide information on how to set the scheduling on a DataSync Task. From the boto3 doc (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/datasync.html#DataSync.Client.update_task), the…
Ankan Das
  • 268
  • 1
  • 2
  • 11
0
votes
0 answers

AWS CloudFormation - DataSync Problem with autogenerated Tags

i try to deploy resources for the Amazon DataSync Service with CloudFormation. The Creation of the resources worked. But i ran into an Error when starting the DataSync Task, created with CloudFormation. CloudFormation automatically tags my Task with…
BieLac
  • 1
  • 1
0
votes
0 answers

How do I transfer files between FSx for Lustre using AWS DataSync in two different VPC's?

I have tried asking this on reddit and also on another AWS forum and have had no luck getting an answer. I'm hoping I can find an answer here. I have an AWS account that was using the default VPC. We are moving away from it and have a new one…
emccl78
  • 1
0
votes
0 answers

AWS DataSync agent ran into an error connecting to AWS

I'm trying to sync local S3 object storage to AWS S3 using AWS DataSync. I configured agent and it shows as online in AWS DataSync dashboard under Agents tab. This is screen of local datasync agent: Agent screen Also it passes all of the network…
Alexandre Begijanovi
  • 1
  • 1
0
votes
1 answer

Migrating a petabyte of data from an SMB drive to s3 while maintaining file creation data metadata?

I am currently trying to migrate about a petabyte of data that currently sits on a local SMB share to AWS S3. The problem I am having is the loss of the original file creation metadata when using Datasync or copying the file to the S3 bucket. I need…
matthew altenburg
  • 1
  • 2
1
2
3