I am running leofs 1.2.22 on a single node. Everything was working ok but when I started leofs today I am unable to list the contents of any buckets. I am getting errors stating the node is down.
A leofs-adm status shows
$ leofs-adm status
[System Confiuration]
-----------------------------------+----------
Item | Value
-----------------------------------+----------
Basic/Consistency level
-----------------------------------+----------
system version | 1.2.22
cluster Id | leofs_1
DC Id | dc_1
Total replicas | 1
number of successes of R | 1
number of successes of W | 1
number of successes of D | 1
number of rack-awareness replicas | 0
ring size | 2^128
-----------------------------------+----------
Multi DC replication settings
-----------------------------------+----------
max number of joinable DCs | 2
number of replicas a DC | 1
-----------------------------------+----------
Manager RING hash
-----------------------------------+----------
current ring-hash | 433fe365
previous ring-hash | 433fe365
-----------------------------------+----------
[State of Node(s)]
-------+--------------------------+--------------+----------------+----------- -----+----------------------------
type | node | state | current ring | prev ring | updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
S | storage_0@127.0.0.1 | running | 433fe365 | 433fe365 | 2017-06-27 01:00:50 -0400
G | gateway_0@127.0.0.1 | running | 433fe365 | 433fe365 | 2017-10-09 06:49:48 -0400
-------+--------------------------+--------------+----------------+----------------+----------------------------
This shows the storage node is running. However if checking the details of the storage node it returns
$ leofs-adm du storage_0@127.0.0.1
[ERROR] nodedown
I tried to recover the node but this also fails
$ leofs-adm recover-node storage_0@127.0.0.1
[ERROR] Could not connect
I am able to list the buckets
$ leofs-adm get-buckets
cluster id | bucket | owner | permissions | created at
-------------+-------------------------+--------+----------------------------------------+---------------------------
leofs_1 | workflow | simon | Me(full_control) | 2017-06-28 20:47:08 -0400
leofs_1 | weather | simon | Me(full_control) | 2017-06-26 08:27:26 -0400
leofs_1 | workers | simon | Me(full_control), Everyone(read,write) | 2017-06-26 08:30:30 -0400
But I am unable to list the contents of any bucket
$ s3cmd ls s3://weather/
WARNING: Retrying failed request: /?delimiter=/
WARNING: 500 (InternalError): We encountered an internal error. Please try again.
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?delimiter=/
WARNING: 500 (InternalError): We encountered an internal error. Please try again.
I am not sure how to recover the node and have not found anything online to help. Updating the version of leofs is not an option as I was unable to get Python boto2 to communicate with a later version of LeoFS.
Regards
Simon
