2

I am using Ceph, uploading many files through radosgw. After, I want to delete the files. I am trying to do that in Python, like this:

bucket = conn.get_bucket(BUCKET)
for key in bucket.list():
    bucket.delete_key(key)

Afterwards, I use bucket.list() to list files in the bucket, and this says that the bucket is now empty, as I intended.

However, when I run ceph df on the mon, it shows that the OSDs still have high utilization (e.g. %RAW USED 90.91). If I continue writing (thinking that the status data just hasn't caught up with the state yet), Ceph essentially locks up (100% utilization).

What's going on?

Note: I do have these standing out in ceph status:

   health HEALTH_WARN
            3 near full osd(s)
            too many PGs per OSD (2168 > max 300)
            pool default.rgw.buckets.data has many more objects per pg than average (too few pgs?)

From what I gather online, this wouldn't cause my particular issue. But I'm new to Ceph and could be wrong.

I have one mon and 3 OSDs. This is just for testing.

Turtle V. Rabbit
  • 313
  • 1
  • 3
  • 8
  • Is the file still opened by a process? – hek2mgl Oct 15 '16 at 22:18
  • @hek2mgl - The process uploads 1000 copies of a file to Ceph, then deletes them as above. Then the process exists. Is the file you're asking about the file that I'm uploading to Ceph? It's not in use. – Turtle V. Rabbit Oct 15 '16 at 22:42
  • @hek2mgl - I've added some more possibly relevant information – Turtle V. Rabbit Oct 15 '16 at 22:48
  • Just found the same problem with CephFS too. Deleting files in CephFS doesn't free up the spaces either. Am trying to figure out the cause, and whether remounting all CephFS partitions might help. – wangguoqin1001 Jan 01 '17 at 21:11
  • Oops figured out that I've got snapshot enabled. Disabling snapshot frees up the extra space immediately. – wangguoqin1001 Jan 08 '17 at 23:35
  • @wangguoqin1001 how did you do that? – SHM Jul 24 '18 at 04:56
  • @SHM Snapshot isn't stable yet for CephFS, so you have to explicitly enable it to get it work. Didn't remember the situation in detail, but there had been more problems before. – wangguoqin1001 Jul 24 '18 at 06:01

1 Answers1

1

You can check if the object is really deleted by rados -p $pool list, I knew for cephfs, when you delete a file, it will return ok when mds mark it as deleted in local memory and then do real delete by sending delete messages to related osd. Maybe radosgw use the same design to speed up delete

hjwsm1989
  • 83
  • 1
  • 9