0

I'm attempting to write a simple Ruby/Nokogiri scraper to get event information from multiple pages and then output it to a CSV that is attached to an email sent out weekly.

I have completed the scraping components and the CSV component and it's working perfectly. However, I now realize that I need to know when new events are added, which means I need some sort of database. Ideally I would just store this locally.

I've dabbled a bit with using the ruby gem 'sequel', but the data does not seem to persist beyond the running of the program. Do I need to download some database software to work with 'sequel'? Also I'm not using the Rails framework, just Ruby.

Any and all guidance is deeply appreciated!

Nicholas S
  • 25
  • 6
  • 1
    Do you have an RDBMS installed on your resident machine to allow you to persist the data? – MageeWorld Jun 30 '16 at 17:59
  • Before abandoning Sequel, you need to do more than "dabble". If the data isn't persisting, it's because you're using an in-memory SQLite database. While they're very useful for some tasks, a persisting database isn't usually one of them. Instead, look at using SQLite on disk, or PostgreSQL, MySQL or whatever else you have [that Sequel supports](http://sequel.jeremyevans.net/rdoc-adapters/index.html). It's extremely capable. http://sequel.jeremyevans.net/rdoc/files/README_rdoc.html#label-Connecting+to+a+database describes how to use an on-disk SQLite DB. – the Tin Man Jun 30 '16 at 18:07
  • You were absolutely right @the-Tin-Man. And that link is very helpful! Thanks so much! – Nicholas S Jun 30 '16 at 18:44

1 Answers1

3

I'm guessing you did Sequel.sqlite, as in the first example in the Sequel README, which creates an in-memory SQLite database. To create a database in your filesystem instead of memory, just pass it a path, e.g.:

Sequel.sqlite("./my-database.db")

This is, of course, assuming that you have the sqlite3 gem installed. If the given file doesn't exist, it will be created.

This is covered in the Sequel docs.

Jordan Running
  • 102,619
  • 17
  • 182
  • 182
  • Ah of course!! I was following the example from the Sequel README and didn't realize that when it said in-memory it would be destroyed once the program concluded running. Thanks very much @Jordan. I appreciate the help! – Nicholas S Jun 30 '16 at 18:42