0

I have a script in Ruby which uses scraperwiki gem. In the directory of this script there's file titled scraperwiki.sqlite. 

items.each do |x|
   if ScraperWiki.select("* from data where .... { x['key123']}'").empty? 
     ScraperWiki.save_sqlite(['key123'], x)
   else
    puts "Skipping already saved record " + record['key123']
   end
end

But nonetheless when I run it I get an error:

/Users/alex/.rvm/gems/ruby-2.1.2/gems/sqlite_magic-0.0.3/lib/sqlite_magic.rb:49:in `rescue in execute': no such table: data (SqliteMagic::NoSuchTable)
  from /Users/alex/.rvm/gems/ruby-2.1.2/gems/sqlite_magic-0.0.3/lib/sqlite_magic.rb:42:in `execute'
  from /Users/alex/.rvm/gems/ruby-2.1.2/gems/scraperwiki-3.0.2/lib/scraperwiki.rb:186:in `select'
Incerteza
  • 32,326
  • 47
  • 154
  • 261

1 Answers1

1

Two things:

  1. The ScraperWiki gem does not create the data (actually swdata) table until you've saved some data
  2. Once you have saved some data, the table appears to be called swdata, not data

Note: swdata is the default but you can change the table_name to data using arguments/configuration variables.

Also note: this create-on-save (not-on-query) is not necessarily part of ScraperWiki's gem and is instead the way it's dependency of https://github.com/openc/sqlite_magic works. To see for yourself, look at the code at https://github.com/openc/sqlite_magic/blob/master/lib/sqlite_magic.rb and see how #create_table is only called from #insert_or_update and #save_data

Mark Silverberg
  • 1,249
  • 2
  • 8
  • 21
  • even if I rename data to swdata, the error persists. – Incerteza Sep 10 '14 at 06:07
  • `Note: swdata is the default but you can change the table_name to data using arguments/configuration variables.` - where? It's a simple one-file script. – Incerteza Sep 10 '14 at 06:08
  • @AlexanderSupertramp have you saved data like I mentioned in #1? https://github.com/scraperwiki/scraperwiki-ruby mentions optional `table_name` parameter – Mark Silverberg Sep 10 '14 at 12:00
  • You see my code - `ScraperWiki.save_sqlite(['key123'], x)`. But the exception is thrown at `ScraperWiki.select("* from data where .... { x['key123']}'").empty? ` As I said, when I rename `"* from data where` to `"* from swdata where` the error remains. – Incerteza Sep 10 '14 at 14:47
  • @AlexanderSupertramp yes, but since it errors before it ever gets to `save_sqlite`, it never saves any data and therefore never creates the table. Do a dummy save before your `if` and `each` statement and this should all be fixed. – Mark Silverberg Sep 10 '14 at 15:04
  • That works. But when I run the 2nd time, how do I check if a dummy record already exists? If I check it and the table doesn't exists yet then I'll run into the same issue as here `ScraperWiki.select("* from data where .... { x['key123']}'").empty?` In other words, I have to check ``ScraperWiki.select(...` first and that can give me an exception. – Incerteza Sep 10 '14 at 18:12
  • I wonder, is checking if a record exists necessary? Even if I save a record with the same unique key 2 or more time, nothing will happen, there'll be only 1 record and it'll be updated, won't it? – Incerteza Sep 10 '14 at 19:51
  • @AlexanderSupertramp if you set up the keys correctly, correct. – Mark Silverberg Sep 10 '14 at 22:06