4

I've got a RDS database with a table containing a ton of data in several columns (some with geo spatial data) I want to search across. SQL queries and good covering indexes on this data is still far too slow to use for something like an AJAX type ahead suggestion field.

As such, I'm investigating options for search and came across Amazon CloudSearch (now powered by Apache Solr) and it seems to fit my needs. The problem is, I can't seem to find a way via the AWS console to import or provide data from RDS. Am I missing something? Other solutions like ElasticSearch have plugins like river to connect an transform MySQL data.

I know there are command line tools for uploading CSV and XML data into CloudSearch. So far the easiest thing I can find is to mysqldump table into CSV or XML format and manually load it with the CLI tools. Is this with some re-occuring cron job the best way to do get data?

Ray
  • 40,256
  • 21
  • 101
  • 138

3 Answers3

3

As of 2014-06-17 this feature is not available on Amazon Cloudsearch.

Ray
  • 40,256
  • 21
  • 101
  • 138
  • So which approach did you ultimately end up taking? I have the same problem now and though I haven't tried it yet, I assume it's possible to upload a document to CloudSearch via a background job or message queue while I'm saving the record to RDS? – Zero Jul 01 '15 at 21:08
0

I think AWS Data Pipeline can help. It works like a cron and you can program reoccuring jobs easily using this.

fesnault
  • 1
  • 2
0

Ran into the same thing, it is only possible to pull directly from RDS if you are using noSQL and AWS's dynamoDB.

Looking into Elasticsearch after finding this out.

ecoding5
  • 404
  • 6
  • 19
  • So which approach did you ultimately end up taking? I have the same problem now and though I haven't tried it yet, I assume it's possible to upload a document to CloudSearch via a background job or message queue while I'm saving the record to RDS? – Zero Jul 01 '15 at 21:05
  • I ended up using Heroku with Elasticsearch because I already was familiar with Heroku. – ecoding5 Jul 01 '15 at 21:15
  • Oh. I assume you used an Elasticsearch Heroku add-on. In that case, that only adds to my options. I'll attempt the background job or SQS route, then Logstash, then database triggers. If all fail, I'll consider using such an add-on on Heroku. – Zero Jul 01 '15 at 21:42
  • I need to upload data to cloudsearch domain, all my data are in rds. I see two of you had same type of question...What did you end up doing? – Nirmal Mar 20 '17 at 21:45
  • I ended up using the Elasticsearch Heroku add on, I have since moved away from that project so I don't know about that issue anymore. The AWS support is pretty good in my experience, I would contact them if I were you to see what they can offer. – ecoding5 Mar 22 '17 at 03:47