0

I try to deploy memsql 5.5 offline in LXC container environment. Install of memsql-ops succeeded. When I ran following command to add memsql offline media "memsqlbin_amd64.tar.gz", the command failed with error message about "out of space".

Here is the full error message:

# memsql-ops file-add -t memsql memsqlbin_amd64.tar.gz            
Traceback (most recent call last):
  File "/usr/local/updated-openssl/lib/python3.4/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
  File "/memsql_platform/bin/memsql-ops", line 18, in <module>
  File "/memsql_platform/memsql_platform/agent/cli/client.py", line 210, in parse
  File "/memsql_platform/memsql_platform/agent/cli/command_file_add.py", line 27, in run
  File "/memsql_platform/memsql_platform/agent/cli/file_add_mixin.py", line 61, in add_file
  File "/memsql_platform/memsql_platform/network/api_client.py", line 126, in call
  File "/memsql_platform/memsql_platform/network/api_client.py", line 92, in _call
memsql_platform.api.exceptions.DeviceOutOfDisk: The API is currently disabled because the disk that MemSQL Ops is installed on is out of space.

Running df -h can see disk space is not a problem from both host side or inside container.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        559G  3.2G  554G   1% /
none            492K     0  492K   0% /dev
/dev/sdc        559G  3.2G  554G   1% /software
cgroup_root      10M     0   10M   0% /sys/fs/cgroup
tmpfs            63G     0   63G   0% /dev/shm
tmpfs            63G  8.1M   63G   1% /run
tmpfs            13G     0   13G   0% /run/user/0

I suspected that LXC container may inherit some system directories from host side and prevent application writes into those directories. But search around Internet let me believe there is no such limitation from LXC containers.

Search the same error messages on Google gives me nothing. I hope this issue can be worked around, but I also want to know why this can happen in container. As far as I know, memsql is successfully deployed into docker officially. I am eager to know how this can be achieved.

ymonad
  • 11,710
  • 1
  • 38
  • 49
robert
  • 397
  • 1
  • 3
  • 14

1 Answers1

0

i'm assuming cgroup_root is the mount for your LXC container - it appears you only have 10M available for that container.

MemSQL Ops requires a minimum of 50M of free space on the host system and if it does not have that, it will go into a hibernation state and avoid doing anything to the system. this is obviously not what you want here, but it is a safeguard for systems with production workloads nearing disk capacity.

you'll need to allocate more space to your LXC container. keep in mind that MemSQL Ops will install MemSQL onto this filesystem as well so you'll need to allocate enough space for the binary and shared libraries as well as the data in MemSQL.

memsqlite
  • 61
  • 3
  • I don't quite know the entry of "cgroup_root" here. lxc man page does not tell that. But "df -h /" returns "/dev/sdc 559G 3.2G 554G 1% /". This makes me assume the container root is located in /dev/sdc, that is the actual location of the container. – robert Nov 22 '16 at 18:22
  • MemSQL Ops checks the disk that its data directory is on (by default this is /var/lib/memsql-ops) as well as your tmpdir It checks the disk usage using `psutil.disk_usage([OPS_DATA_DIR, tmpdir])` which eventually calls out to this: https://github.com/giampaolo/psutil/blob/master/psutil/_psposix.py#L123 – memsqlite Nov 22 '16 at 23:17
  • While this answer does not directly fix the issue. But it inspired me to try different backing store (storage for LXC container), like **ZFS** and **directory** which can successfully deploy MemSQL within container. So I like to treat this is the solution for the question. But to the root cause of the issue, I think it'd better to let MemSQL checks the free space with something like `df /var`. – robert Jan 26 '17 at 19:43