Stack Overflow
Stack Overflow

Questions tagged [amazon-s3-select]

Amazon S3 Select enables applications to retrieve only a subset of data from an Amazon S3 object by using simple SQL expressions.

See: SQL Functions - Amazon Simple Storage Service

91 questions
4
votes
1 answer

Getting OverMaxRecordSize while fetching single record from AWS S3 Select?

I am getting an error OverMaxRecordSize: The character number in one record is more than our max threshold, maxCharsPerRecord: 1,048,576 when I'm trying to fetch any particular record from AWS S3 using selectObjectContent function. If I'm fetching…
Nishant Khandelwal
  • 199
  • 1
  • 5
  • 13
3
votes
0 answers

Parse S3Select output AWS SDK C++

I have the following code snippet. #include #include #include #include #include #include…
dkoutsou
  • 87
  • 5
3
votes
0 answers

Perl PAWS SelectObjectContent - "Precondition Failed"

I receive a "Precondition Failed" error when I try to run: my $obj = $s3->SelectObjectContent( Bucket => 'MyBucket', Expression => 'SELECT * FROM s3object s', ExpressionType => 'SQL', InputSerialization =>…
abe-aa
  • 31
  • 2
3
votes
2 answers

escape single quote in s3 select query

Sample data, stored in a file in S3. As you can see the format of my data is one json per line {"first": "John", "last": "Smith"} {"first": "Mary", "last": "O'Hara"} {"first": "Mary", "last": "Oats"} My ultimate objective is to query by the last…
flybywire
  • 261,858
  • 191
  • 397
  • 503
3
votes
1 answer

Can't make a query on a JSON by using AWS S3 Select

I am trying to query a JSON object by using aws s3-select. My JSON array structure is like this: [ { "title": "s3", "url": "https://...", "time": "2019-07-02", "summary": "by using s3 select..." }, { …
Huseyin Sahin
  • 211
  • 4
  • 16
3
votes
1 answer

How can I create a mock for the EventStream which is used but S3 Select API for getting contents based on S3 Select query?

I am creating a unit test for a function that reads objects from S3 buckets using the boto3's s3 client function 'select_object_content'. The response i am looking to mock is { 'Payload': EventStream({ 'Records': { …
qazplm
  • 33
  • 7
3
votes
2 answers

AWS S3 deletion of files that haven't been accessed

I'm writing a service that takes screenshots of a lot of URLs and saves them in a public S3 bucket. Due to storage costs, I'd like to periodically purge the aforementioned bucket and delete every screenshot that hasn't been accessed in the last X…
KidCrippler
  • 1,633
  • 2
  • 19
  • 34
3
votes
1 answer

python boto3 How to configure AWS s3select on parquet?

I am trying to use AWS s3select feature to query a parquet file. According to the documentation it's supported but I have tried various configurations and can't get it to work. In each of the InputSerialization attempts I've shown commented-out,…
mojoken
  • 1,256
  • 1
  • 15
  • 27
2
votes
0 answers

Querying Parquet from S3 using Bloom filter

I have some data in an s3 bucket in Parquet format. The data consists of various datasets containing a UUID key followed by values. I need to query individual UUIDs. My question is whether it is possible to use the metadata provided by each Parquet…
sancholp
  • 67
  • 7
2
votes
1 answer

Amazon S3 SelectObjectContent Support Missing for .NET v 5.0?

I found what appears to be a breaking change in the AWS SDK between .NET v4.8 and .NET v5.0 / .NETCore v3.1. Following the tutorial in an official blog, I found I could not compile/run the examples in .NET v5.0 or .NETCore v3.1 using the synchronous…
jam
  • 33
  • 6
2
votes
0 answers

What is the general speed of Amazon S3 Select on a JSON file?

I am looking to consider S3 as a backup storage to a primary Redis DB. I would like to be able to archive data out from Redis and into S3 which is rarely used. This however brings up the question of how quick is an S3 select? Is it quick enough for…
SomeAssembelyRequired
  • 61
  • 3
2
votes
1 answer

Amazon S3 Select Issue : not supporting line break occurring inside fields

I am trying to use Amazon S3 Select to read records from a CSV file and if the field contains a line break(\n), then the record is not being parsed as a single record. Also, the line break inside the field has been properly escaped by double quotes…
Swathy S
  • 21
  • 2
2
votes
1 answer

Aws S3 Select ScanRange is not working as expected and providing entire file result

I am trying to scan and get a chunk of data from my object in S3 using S3 select using Java SDK. I am using scanRange class to provide the start and end bytes range of the S3 object. The s3 object is a 9MB parquet file having an uncompressed size of…
CodeTalker
  • 1,683
  • 2
  • 21
  • 31
2
votes
1 answer

AWS S3 Select: WHERE clause with json array

I have the following json structure: { "guid": "60f1a5d3-f082-4432-a023-eec2e211f586", "notebook": "esse", "title": "Tempor cillum consequat sunt aliquip laborum deserunt.", "tags": [ "magna", …
Nico Knabe
  • 53
  • 4
2
votes
1 answer

Missing Headers / Invalid Column Index errors with AWS S3 Select

I am trying to make a select request from data stored in CSV format with AWS S3 SelectObjectContent and an SQL expression. I am working with Laravel 6. When I request all the data from the object ('Expression' => 'SELECT * from S3Object'),…
jamawe
  • 229
  • 3
  • 12
1
2
3 4 5 6 7