0

I am setting the OPENSTACK yoga version. After building all modules, I encountered a problem.In the process of creating a volume, if I do not select the volume source, I can create a blank volume normally. But if I choose a volume source (such as image), an error will occur, and the volume cannot be created. The error content is: 'ERROR oslo_messaging.rpc.server Stderr: 'qemu: qemu_thread_create: Resource temporarily unavailable\n''. But I checked my system resources, it is completely enough.I guess there should be a problem in the process of mirroring into the volume.How should I solve this situation?

The following error is reported in /var/log/cinder/volume.log of my compute node:

2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-751>", line 2, in create_volume
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/objects/cleanable.py", line 208, in wrapper
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 815, in create_volume
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     _run_flow()
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 807, in _run_flow
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     flow_engine.run()
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     failures[0].reraise()
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/types/failure.py", line 346, in reraise
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     six.reraise(*self._exc_info)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     raise value
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1214, in execute
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     **volume_spec)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 664, in _wrapper
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     return r(f, *args, **kwargs)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 423, in __call__
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 360, in iter
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     return fut.result()
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     return self.__get_result()
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     raise self._exception
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 426, in __call__
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1108, in _create_from_image
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     image_service)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 980, in _create_from_image_cache_or_download
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     data = image_utils.qemu_img_info(tmp_image)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 173, in qemu_img_info
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     prlimit=QEMU_IMG_LIMITS)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 172, in execute
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server Command: /usr/bin/python3 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /var/lib/cinder/conversion/image_fetch_eb32b72c-4cae-4e74-af71-f509593b4b92_6doh3atxcompute01@lvm
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server Exit code: 134
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server Stdout: ''
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server Stderr: 'qemu: qemu_thread_create: Resource temporarily unavailable\n'
2023-05-24 11:46:15.215 2951 ERROR oslo_messaging.rpc.server

============================================================================

The following is the configuration of controller node/etc/cinder/cinder.conf

[database]
connection = mysql+pymysql://cinder:CINDER_DBPASS@controller/cinder
[DEFAULT]
transport_url = rabbit://openstack:RABBIT_PASS@controller
auth_strategy = keystone
[keystone_authtoken]
www_authenticate_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = CINDER_PASS
my_ip = 192.168.1.10
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

============================================================================ The following is the configuration of storage node/etc/cinder/cinder.conf

[DEFAULT]
transport_url = rabbit://openstack:RABBIT_PASS@controller
auth_strategy = keystone
my_ip = 192.168.1.11
enabled_backends = lvm
glance_api_servers = http://controller:9292
[database]
connection = mysql+pymysql://cinder:CINDER_DBPASS@controller/cinder
[keystone_authtoken]
www_authenticate_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = CINDER_PASS
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
target_protocol = iscsi
target_helper = lioadm
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

The following error is reported in /var/log/cinder/scheduler.log of my control node

2023-05-24 11:46:14.946 51880 ERROR cinder.scheduler.filter_scheduler [req-5d9429f6-7055-4b32-9dec-5f8b7a3ba453 a1bb5408b69c405a9c75d25b17db2fd7 c8c4f6b528ce4b0bb1b02e60a8e7681a - - -] Error scheduling 9a3d631f-afbf-48a2-9cd2-f1c1e4c45ef0 from last vol-service: compute01@lvm#lvm : ['Traceback (most recent call last):\n', '  File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task\n    result = task.execute(**arguments)\n', '  File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1214, in execute\n    **volume_spec)\n', '  File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 664, in _wrapper\n    return r(f, *args, **kwargs)\n', '  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 423, in __call__\n    do = self.iter(retry_state=retry_state)\n', '  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 360, in iter\n    return fut.result()\n', '  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result\n    return self.__get_result()\n', '  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result\n    raise self._exception\n', '  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 426, in __call__\n    result = fn(*args, **kwargs)\n', '  File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1108, in _create_from_image\n    image_service)\n', '  File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 980, in _create_from_image_cache_or_download\n    data = image_utils.qemu_img_info(tmp_image)\n', '  File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 173, in qemu_img_info\n    prlimit=QEMU_IMG_LIMITS)\n', '  File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 172, in execute\n    return processutils.execute(*cmd, **kwargs)\n', '  File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute\n    cmd=sanitized_cmd)\n', "oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.\nCommand: /usr/bin/python3 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info --output=json /var/lib/cinder/conversion/image_fetch_eb32b72c-4cae-4e74-af71-f509593b4b92_6doh3atxcompute01@lvm\nExit code: 134\nStdout: ''\nStderr: 'qemu: qemu_thread_create: Resource temporarily unavailable\\n'\n"]
2023-05-24 11:46:14.947 51880 INFO cinder.message.api [req-5d9429f6-7055-4b32-9dec-5f8b7a3ba453 a1bb5408b69c405a9c75d25b17db2fd7 c8c4f6b528ce4b0bb1b02e60a8e7681a - - -] Creating message record for request_id = req-5d9429f6-7055-4b32-9dec-5f8b7a3ba453
2023-05-24 11:46:14.957 51880 ERROR cinder.scheduler.flows.create_volume [req-5d9429f6-7055-4b32-9dec-5f8b7a3ba453 a1bb5408b69c405a9c75d25b17db2fd7 c8c4f6b528ce4b0bb1b02e60a8e7681a - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource 9a3d631f-afbf-48a2-9cd2-f1c1e4c45ef0: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource 9a3d631f-afbf-48a2-9cd2-f1c1e4c45ef
lhclhc1231
  • 11
  • 2
  • Can you explain a bit more what you mean by volume source "mirror"? Do you mean an image (from glance store)? From the error message it seems like cinder can't fetch the requested image to convert. Usually, if you create a volume from image cinder tries to download it to the local filesystem `/var/lib/cinder/conversion/` and the converts it with the qemu-img tool. Check the cinder logs as well, maybe they show more information, also try debug logs to see more details. – eblock May 22 '23 at 07:48
  • Your understanding is correct. When I create a volume with image, the above error will appear.I checked /var/log/cinder/scheduler.log, and the error reported in this log is as follows, which points to the same problem as the error reported in /var/log/cinder/volume.log – lhclhc1231 May 23 '23 at 07:07
  • 2023-05-23 15:06:23.934 1832 ERROR cinder.scheduler.flows.create_volume [req-6d3bfc03-f9ff-4515-aa88-3f66e08b8c6f a1bb5408b69c405a9c75d25b17db2fd7 c8c4f6b528ce4b0bb1b02e60a8e7681a - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource 778e2be6-8349-4f1f-b735-3ff8a2cba4ed: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource 778e2be6-8349-4f1f-b735-3ff8a2cba4ed – lhclhc1231 May 23 '23 at 07:08
  • Do not comment but add the log output to your question and use appropriate code tags. Please also share your cinder.conf, there must be a mistake. What are your storage backends for glance, cinder and nova? Can you launch VMs? – eblock May 23 '23 at 07:45
  • I updated my question. It contains my storage backends--lvm, and I also use appropriate code tags to show the errors I encountered. I can launch VMs but only can be started by image and select No on Create New Volume option. If i select YES on Create New Volume option, i will encounter this error. Then I tried to create an empty volume and it was successful. But if I selected the image as the Volume Source when creating the volume, the same error would occur. So I think the problem may lie in the interaction between image and cinder. – lhclhc1231 May 24 '23 at 04:17
  • I would use the same cinder config for both controller and storage node. It seems like the controller is missing the glance_api_servers config, maybe that would already help. – eblock May 24 '23 at 09:09
  • I tried to add glance_api_servers but still the same error – lhclhc1231 May 24 '23 at 11:52
  • Before trying each config option individually I would just use the whole cinder.conf from the storage node on the control node (make required changes, of course, like my_ip etc.). For example, "enabled_backends" is also missing on the controller. – eblock May 24 '23 at 13:27

0 Answers0