how to search for file contents in amazon S3 bucket without downloading the file

Question

i have n number of files uploaded to amazon S3 i need*search* those files based on occurrence of an string in its contents , i tried one method of downloading the files from S3 bucket converting input stream to string and then search for the word in content , but if their are more than five to six files it takes lot of time to do the above process,

is their any other way to do this , please help thanks in advance.

If you want to search the content, I don't think there is any alternate to downloading the file — Arun P Johny, Feb 28 '13 at 03:54
Going forward, you can stream the files to CloudWatch Logs instead of or in addition to archiving on S3. Then you can easily search their contents. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html — Jeff, Mar 21 '22 at 14:48

score 3 · Answer 1 · answered Mar 06 '18 at 11:37

3

If your files contain CSV, TSV, JSON, Parquet or ORC, you can take a look at AWS's Athena: https://aws.amazon.com/athena/

From their intro:

Amazon Athena is a fast, cost-effective, interactive query service that makes it easy to analyze petabytes of data in S3 with no data warehouses or clusters to manage.

Unlikely to help you though as it sounds like you have plain text to search through.

Thought I'd mention it as it might help others looking to solve a similar problem.

answered Mar 06 '18 at 11:37

Adrian Lynch

8,237
2
32
40

Please don't. Athena can query *structured* files (CSV, JSON etc) to generate reports. For full-text search use AWS Elastic or AWS CloudSearch https://stackoverflow.com/questions/28557719/how-do-you-full-text-search-an-amazon-s3-bucket – Alex from Jitbit Feb 13 '21 at 16:30

score 2 · Answer 2 · answered Feb 28 '13 at 03:54

2

Nope!

If you can't infer where the matches are from object metadata (like, the file name), then you're stuck with downloading & searching manually. If you have spare bandwidth, I suggest downloading a few files at a time to speed things up.

answered Feb 28 '13 at 03:54

phs

10,687
4
58
84

score 2 · Answer 3 · answered Feb 28 '13 at 03:56

2

In single word NO!!

I think you can do to imprrve the performance will be to cache the files locally so that you don't have to download the file again and again

Probably you can use Last-Modified header to check whether the local file is dirty, then download it again

answered Feb 28 '13 at 03:56

Arun P Johny

384,651
66
527
531

i am not permitted to store to the file locally because everyone cannot download the file they can only view it.if i store it locally it will be accessible to the user right hence i cannot store it locally . – Ekata Feb 28 '13 at 03:59
are you doing the search from server side or client? – Arun P Johny Feb 28 '13 at 03:59
client. its like a get the files based on content screen , the user will enter a string and i have to get all the files in the bucket and search each and every file content for that string – Ekata Feb 28 '13 at 04:07
1

@Ekata: *"cannot download the file they can only view it"*: I hope that is not a security requirement, because let me tell you that if a client can view the file then generally they can also store it to their hard drive, even if it takes modifying the client software to do so... – thkala Feb 28 '13 at 04:09
but if i just get the input stream and if i dont use write function on the file i guess it will not store the locally right !! – Ekata Feb 28 '13 at 04:16

score 1 · Answer 4 · answered Feb 28 '13 at 12:14

My suggestion, since you seem to own the files, is to index them manually, based on content. If there is a lot of "keywords", or metadata associated with each file, you can help yourself by using a lightweight database, where you will perform your queries and get the exact file(s) users are looking for. This will preserve bandwidth and also be much faster, at the cost of maintaining kind of an "indexing" system.

Another option (if each file does not contain much metadata) would be to reorganize the files in your buckets, adding prefixes which would "auto-index" them, like follows:

/foo/bar/randomFileContainingFooBar.dat /foo/zar/anotherRandomFileContainingFooZar.dat.

This way you might end up scanning the whole bucket in order to find the set of files you need (this is why I suggested this option only if you have little metadata), but you will only download the matching ones, which is still much better than your original approach.

score 1 · Answer 5 · answered Jun 08 '22 at 06:07

Yes, now it is possible with AWS S3 Select. If your objects stored in CSV, JSON, or Apache Parquet format.

AWS details: https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-javascript/

Aws S3 Select getting started examples: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-select.html

Just if anyone will be looking the same.

i.e With SDK:

if you have a csv like this:

user_name,age
jsrocks,13
node4life,22
esfuture,29
...

And for example we would like to retrieve something like:

SELECT user_name FROM S3Object WHERE cast(age as int) > 20

Then in AWS SDK on JavaScript we do the following:

const S3 = require('aws-sdk/clients/s3');
const client = new S3({
    region: 'us-west-2'
});

const params = {
    Bucket: 'my-bucket,
    Key: 'target-file.csv',
    ExpressionType: 'SQL,
    Expression: 'SELECT user_name FROM S3Object WHERE cast(age as int) > 20',
    InputSerialization: {
        CSV: {
            FileHeaderInfo: 'USE',
            RecordDelimiter: '\n',
            FieldDelimiter: ','
        }
    },
    OutputSerialization: {
        CSV: {}
    }
};

thkala · Accepted Answer · 2013-02-28T04:18:51.247

I am not familiar with Amazon S3, but the general way to deal with searching remote files is to use indexing, with the index itself being stored on the remote server. That way each search will use the index to deduce a relatively small number of potential matching files and only those will be scanned directly to verify if they are indeed a match or not. Depending on your search terms and the complexity of the pattern, it might even be possible to avoid the direct file scan altogether.

That said, I do not know whether Amazon S3 has an indexing engine that you can use or whether there are supplemental libraries that do that for you, but the concept is simple enough that you should be able to get something working by yourself without too much work.

EDIT:

Generally the tokens that exist in each file are what is indexed. For example if you want to search for "foo bar" the index will tell you which files contain "foo" and which contain "bar". The cross-section of these results will be the files that contain both "foo" and "bar". You will have to scan those files directly to select those (if any) where "foo" and "bar" are right next to each other in the right order.

In any case, the amount of data that is downloaded to the client would be far less than downloading and scanning everything, although that would also depend on how your files are structured and what your search patterns look like.

but i guess using indexing we can search only the metadata right?? and not the complete content — Ekata, Feb 28 '13 at 04:11

how to search for file contents in amazon S3 bucket without downloading the file

6 Answers6

Linked