0

I can no longer generate new columns in my database. My stack is below. I am new to managing my own server and am unfamiliar with Apache, Ubuntu and AWS. Currently I am receiving an error saying that I have no space left.

It looks like my folder /dev/xvda1 is at 100% capacity. How can I identify how space is being allocated and how can I open up more space?

I will eagerly share more information if you can help me with what to share.

My Error

irb(main):107:0> Survey.generate_for_appointment(a, b)
PG::DiskFull: ERROR:  could not extend file "base/16402/25415": No space left on device
HINT:  Check free disk space.
: INSERT INTO "questions" ("choices", "content", "created_at", "position", "question_type", "survey_group", "survey_group_question_id", "survey_id", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING "id"
log writing failed. No space left on device @ io_write - /etc/projects/myapp/log/production.log
ActiveRecord::StatementInvalid: PG::DiskFull: ERROR:  could not extend file "base/16402/25415": No space left on device
HINT:  Check free disk space.

Storage

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ sudo du -h --max-depth=1 /
4.0K    /media
752K    /tmp
0   /sys
9.6M    /bin
1.3G    /home
908M    /usr
4.0K    /mnt
20M /boot
24K /root
4.0K    /opt
3.7G    /etc
du: cannot access ‘/proc/7489/task/7489/fd/4’: No such file or directory
du: cannot access ‘/proc/7489/task/7489/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/7489/fd/4’: No such file or directory
du: cannot access ‘/proc/7489/fdinfo/4’: No such file or directory
0   /proc
9.4M    /sbin
4.0K    /srv
16K /lost+found
60M /lib
12K /dev
649M    /var
4.0K    /lib64
212K    /run
7.5G    /

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  7.5G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            284M   12K  284M   1% /dev
tmpfs            59M  212K   59M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            295M     0  295M   0% /run/shm
none            100M     0  100M   0% /run/user

Detailed Storage

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ sudo du -x / | sort -n | tail -40
98880   /home/ubuntu/.rvm/rubies/ruby-2.1.2
98908   /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.4.1
98908   /home/ubuntu/.rvm/rubies/ruby-2.1.5/lib
99436   /home/ubuntu/.rvm/rubies/ruby-2.1.5
107920  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp/x86_64-linux-gnu/ports
107924  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp/x86_64-linux-gnu
107928  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp
108520  /var/lib/apt/lists
108572  /var/lib/apt
111044  /usr/src
119420  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri
119424  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext
145780  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1
146972  /var/cache
155528  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/passenger-4.0.45/buildout
155916  /usr/lib/x86_64-linux-gnu
158396  /etc/projects/myapp/public/assets
163068  /etc/projects/myapp/public
198320  /home/ubuntu/.rvm/rubies
200824  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/passenger-4.0.45
207540  /home/ubuntu/.rvm/gems/ruby-2.1.5/gems
231732  /var/lib
264856  /home/ubuntu/.rvm/gems/ruby-2.1.5
271292  /var/log/apache2
284432  /var/log
292684  /usr/share
403300  /usr/lib
583604  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems
650888  /home/ubuntu/.rvm/gems/ruby-2.1.2
664560  /var
915752  /home/ubuntu/.rvm/gems
929616  /usr
1174228 /home/ubuntu/.rvm
1262052 /home/ubuntu
1262056 /home
3470180 /etc/projects/myapp/log
3782520 /etc/projects/myapp
3782524 /etc/projects
3789104 /etc
7795236 /

Nodes in Use

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 153278 371010   30% /
none            75346      2  75344    1% /sys/fs/cgroup
udev            72620    381  72239    1% /dev
tmpfs           75346    266  75080    1% /run
none            75346      3  75343    1% /run/lock
none            75346      1  75345    1% /run/shm
none            75346      2  75344    1% /run/user

Passanger Version

Phusion Passenger version 4.0.55

Apache Version

Server version: Apache/2.4.7 (Ubuntu)

RVM Version

rvm 1.26.4 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]

Ruby Versions

rvm rubies

   ruby-2.1.2 [ x86_64 ]
=* ruby-2.1.5 [ x86_64 ]

# => - current
# =* - current && default
#  * - default
MicFin
  • 109
  • 1
  • 6
  • Why is `/etc` 3.7GB? That is an unusually large size for `/etc`. – Zoredache Jun 08 '15 at 17:10
  • @zoredache I believe I created a model that generated far too many rows in Postgres. Do you know how I could check that? – MicFin Jun 08 '15 at 17:14
  • I don't know postgres, but if you are using standard packages, I highly doubt it is storing data under `/etc`. – Zoredache Jun 08 '15 at 17:15
  • @zoredache thanks for the advice. I deleted 3470180 from /etc/projects/myapp/log by using the rake clear command. It looks like my logs were taking up half my storage :/ – MicFin Jun 08 '15 at 17:20

1 Answers1

1

You have to explore your HDD and clean up outdated and/or unused files.

The easiest way is to stop all the services like postgres/apache/postfix etc (to prevent data loss) and dig into directory sizes. The command 

du -d 1 | sort -nr

list you all the nested directories from the current location and their sizes. After certain iterations you'll find the space-mongers and then you'll need an anothed advice - how to deal with them.

Kondybas
  • 6,964
  • 2
  • 20
  • 24
  • thanks! I used that list to remove 3470180 from /etc/projects/myapp/log by using the rake clear command. It looks like my logs were taking up half my storage – MicFin Jun 08 '15 at 17:21
  • You have to setup periodic log rotating. All popular services already have mature tradition of log rotation, as well as well known traps novice can step in. Googling `%SERVICENAME% log rotate` would be very helpful to you. – Kondybas Jun 08 '15 at 21:29