1

I have a instance of Crate 1.0.2 and I dropped a table from it. Then re-created table with same name and slightly modified schema. Then I imported data using copy from command. File argument to copy from command consists of 10,000 records and copy from command runs ok. When I check table tab in crate web console, it shows many partitions added and each partition having few records. If I add number of records column on this tab, it comes close to 10k but when I fire a command "select count(*) from mytable", it returns around 8000 records only. On further investigation found that there are certain partitions on which data cannot be queried at all. Has any one seen this problem? Does it have anything to do with table drop and creation with same name ? I also observed that when a table is dropped, not all files related to that table are deleted from path.data. Are these directories a reason for those partitions become non-query able? While importing, I saw "Document already exists" exception. I know my data does not have any duplicate value for primary column.

Cœur
  • 37,241
  • 25
  • 195
  • 267
vinayakshukre
  • 185
  • 1
  • 10

1 Answers1

1

Some questions to clarify the issue:

  • Have you run refresh table mytable after your copy command has finished?
  • Are you sure that with the new schema of the table, there are no duplicate records?
  • Since 1.x versions are not supported anymore, could you try with CrateDB 2.1.6 which is the current stable version to see if the problem persists?
Marios Trivyzas
  • 21
  • 2
  • 1) Default refresh time is few seconds only. I saw this behavior even after more than 1 hour. 2) I am sure I do not have duplicates. 3) Upgrade to 2.1.6 failed with error "[ERROR][o.e.g.GatewayMetaState ] [Cima del Desenigo] failed to read local state, exiting... java.nio.file.FileSystemException: /opt/crate/data/cratec012/nodes/0/indices/gra..partitioned.mytable.04732d9g68r38e9g60o30c1g -> /opt/crate/data/cratec012/nodes/0/indices/zFk9JkUDTIG5ktVuQjGVPQ: Directory not empty my-pc crate: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) – vinayakshukre Sep 15 '17 at 09:51
  • I also see below line crate: [INFO ][o.e.c.u.IndexFolderUpgrader] [gra..partitione d.mytable.04732d9g64p3cdhm60o30c1g/EIYpqrubRjiquuGx2zIc_A] upgrading [/opt/crate/data/crate012/nodes/0/ind ices/gra..partitioned.mytable.04732d9g64p3cdhm60o30c1g] to new naming convention What is this new naming convention during upgrade from 1.0.2. to 2.1.6 ? – vinayakshukre Sep 15 '17 at 10:44
  • You need to go through 1.1.x before upgrading to >= 2. Have you read the notes here: https://crate.io/docs/crate/reference/release_notes/2.1.6.html#version-2-1-6? You should d also be aware of the Breaking changes of 2.0: https://crate.io/docs/crate/reference/release_notes/2.0.0.html#breaking-changes and the upgrade notes of https://crate.io/docs/crate/reference/release_notes/2.0.0.html#upgrade-notes – Marios Trivyzas Sep 15 '17 at 12:56
  • Yep. Realized that later. Now upgrade is successful. – vinayakshukre Sep 18 '17 at 04:56
  • Do you still have the issue with 2.1? – Marios Trivyzas Sep 18 '17 at 09:32
  • I did not see that yet. But need to try adding, deleting and adding more data. – vinayakshukre Sep 18 '17 at 12:36