I'm running cassandra 3.11.3 in three nodes with linux el7. The repair process always succeeded before untill it got problem recently and never recovered. I finally find it is because there is a failure in one of the 3 nodes during cassandra repair, in that server log, it tell
org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException: /var/db/data/data/wiotp/device_mementos-53b0fe
10e2bf11e98af1457e3e7c4184/snapshots/b0b3e1e0-6a78-11ea-9eca-33d64338b8d7/mc-186-big-Statistics.db -> /var/db/data/data
/wiotp/device_mementos-53b0fe10e2bf11e98af1457e3e7c4184/mc-186-big-Statistics.db: Input/output error
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:103) ~[apache-cassandra-3.11.3.jar:3.11
.3]
investigation done so far:
I checked the access permission, no issue, all folders and sub-folder looks like this
-rw-r--r--. 1 cassandra
I verified the destination file, it existed
$ ls /var/db/data/data/wiotp/device_mementos-53b0fe10e2bf11e98af1457e3e7c4184/mc-186-big-Statistics.db -l
-rw-r--r--. 1 cassandra 2033 4780 Feb 15 01:31 /var/db/data/data/wiotp/device_mementos-53b0fe10e2bf11e98af1457e3e7c4184/mc-186-big-Statistics.db
I manually created the hard link file, it can not be created successfully.
but a question here is only if I firstly created the sub-folder b0b3e1e0-6a78-11ea-9eca-33d64338b8d7 under the folder snapshot then this hard link file can be created, or it will failed as No such file or directory.
another thing is, under the folder snapshot, other sub-folder's name is a numeric string, like 1581814812868 except the above one which seems very different.
I check the related code, and find it should run into Files.createLink(to.toPath(), from.toPath()); as this code clip show
enter image description here
but still not sure what the error Input/output error really complain about.
I searched similar case here, seems this cassandra snapshot : Unable to create hard link from errno 1 is closest to my error, but unfortunately the solution there doesn't work for me.
Dose anyone has any idea? thank you!
