2

We're trying to build our own Vespa cluster starting with a Kubernetes quick start, and indexing a document returns a "no space" error.

The problem is that we can't really understand where do we add that space. Any pointers on what to read/change?

The error we're getting:

{"pathId":"/document/v1/product/product/docid/1739707:651",
"id":"id:product:product::1739707:651",
"message":"[UNKNOWN(251009) @ tcp/vespa-0.vespa.staging.svc.cluster.local:19112/default]: ReturnCode(NO_SPACE, External feed is blocked due to resource exhaustion: disk on node 0 [vespa-0.vespa.staging.svc.cluster.local] (0.838 > 0.800)) "}

(are those numbers - 838 and 800 - indicating space necessary?)

Our application configuration looks like this:

> cat hosts.xml
<?xml version="1.0" encoding="utf-8" ?>
<!-- https://docs.vespa.ai/en/reference/hosts.html -->
<hosts>
  <host name="localhost">
    <alias>node1</alias>
  </host>
</hosts>

> cat services.xml
<?xml version="1.0" encoding="utf-8" ?>
<services version="1.0">
  <container version="1.0" id="default">
    <document-api/>
    <search/>
    <nodes>
      <node hostalias="node1" />
    </nodes>
  </container>

  <content id="product" version="1.0">
    <redundancy>1</redundancy>
    <documents>
      <document type="product" mode="index" />
    </documents>
    <nodes>
      <node hostalias="node1" distribution-key="0" />
    </nodes>
  </content>
</services>
Alexander Solovyov
  • 1,526
  • 1
  • 13
  • 21

1 Answers1

4

Okay, properly asking question is a half way to an answer. Our disk usage looked like this:

overlay               98G   77G   17G  83% /

So Vespa wanted 80% of used space max and we provided storage with 83% used disk space. The solution is to add section like this side content tag:

    <tuning>
      <resource-limits>
        <disk>0.9</disk>
      </resource-limits>
    </tuning>
Alexander Solovyov
  • 1,526
  • 1
  • 13
  • 21
  • 1
    This will work, but we usually only recommend this as a temporary workaround if your cluster is "stuck" in high resource utilization, to be able to return to a lower resource utilization. You should have 20% free disk space as the Vespa storage nodes need some overhead to do maintenance operations, such as index merging. Going 100% out of disk space is bad and may in the worst case lead to a corrupted index, which will need to be wiped and reindexed from scratch. A good "target" utilization is 70%. – andreer Jan 11 '22 at 09:44
  • 1
    Also described here https://docs.vespa.ai/en/operations/feed-block.html – Jo Kristian Bergum Jan 11 '22 at 09:51
  • Yeah, it's more of a "let's test this" scenario than real production-ready deployment. – Alexander Solovyov Jan 11 '22 at 14:29
  • When you're just getting started with development using the docker container, this is really frustrating friction to spend a bunch of time running this down. The top of the feed-block page says "Fix this by increasing allocated storage for the Docker daemon, clean up unused volumes or remove unused Docker images.", which is very misleading. When you 30GB free and can't index a single document it's perplexing. Glad I found this StackOverflow. – john May 08 '23 at 19:40