22

I am not understanding when to use omitNorms="true". I read 2-3 links but still I am not clear with its meaning. what does it mean "Set to true to omit the norms associated with this field (this disables length normalization and index-time boosting for the field, and saves some memory). Only full-text fields or fields that need an index-time boost need norms." at http://wiki.apache.org/solr/SchemaXml page

Kamal Kishore
  • 325
  • 2
  • 4
  • 15

1 Answers1

31

Norms are stored as a Single byte information in the index per document per field. This will hold information for the index time boost applied to the field or Length information.
Length information would allow you to boost shorter fields more that longer fields.
Also, Index time boost will allow one field to be boosted higher then other.

As it takes up space, it should be turned off if not needed.
If no index time boost is used OR if the fields are short text fields or non-text fields which do not need any length normalization.

You can find a little detailed explanation here.

When norms are loaded up into an IndexReader, they are loaded into a byte[maxdoc] array for each field – so even if one document out of 400 million has a field, it is still going to load byte[maxdoc] for that field, potentially using a lot of RAM.

As an example of how much RAM we are talking about, one field in a 10 million doc index will take up just under 10 MB of RAM. One hundred such fields will take nearly a gigabyte of RAM.

notapatch
  • 6,569
  • 6
  • 41
  • 45
Jayendra
  • 52,349
  • 4
  • 80
  • 90
  • 3
    Since the space taken up by norms is pretty trivial in most cases, I don't agree that it should be turned off if not needed. I fall more on the side of: it should be left on unless it presents a problem. If you are forced to optimize out functionality in favor of performance, that's fine, but otherwise, even if you aren't using index-time boosts, length norms should be used, by default, for the best scoring of full-text searches. – femtoRgon Sep 09 '13 at 15:43
  • Pls read the article in the answer, also added the relevant section in the answer which presents stats for the memory usage. Also, it is advisable to have it only for fields which would be used for full text searches. – Jayendra Sep 10 '13 at 03:49
  • The cited RAM usage may have been true at the time of writing, however, optimizations to that have been made, if I understand, in [2.9](https://issues.apache.org/jira/browse/LUCENE-505) and more in [4.0](https://issues.apache.org/jira/browse/LUCENE-830). Also, I don't argue that it can be used to optimize, I just disagree with the suggestion that turning it off should be your go-to default. For the most part, using correct field types should handle this, since `omitNorms` defaults to true for primitive types. – femtoRgon Sep 10 '13 at 04:34
  • I never mentioned it should be turned off by default. The answers mentions the case and should be turned off if not needed. Most of the fields definition already has it in Solr so usually it is not needed to be done. – Jayendra Sep 10 '13 at 04:39
  • It looks like the default for text types is false then? – Beth Lang Feb 27 '14 at 20:57
  • Yes, the default is omitNorms=false (i.e. norms are stored and loaded) for text type. See [this forum thread](http://www.manning-sandbox.com/thread.jspa?messageID=137631). Also, while I am not sure about the exact RAM usage, it it still the case in at least 4.5.0 that norms are loaded into memory, the memory usage is proportional to the number of fields and documents, and the memory taken is not freed until the associated index readers are closed. That is something to be considered if your index's document count is very large. – Kai Chan Feb 27 '14 at 22:07
  • Link to the reference article is broken - Read the [article](https://lucidworks.com/2009/09/02/scaling-lucene-and-solr/) here. – Kishore Kirdat Feb 03 '17 at 15:38
  • I was facing an issue of large index size when I migrated from solr 5.0 to 6.4.1. Explicitly setting omitNorms=true for one filed in schema, solved my problem. Solr documentation says that the default is true for primitive types, my field was text_general. Was the default for text_general changed in solr 6? since I had no such issue in solr 5. – Pratik Patel Feb 21 '17 at 20:32
  • Reading this [forum thread](https://lucene.472066.n3.nabble.com/What-is-omitNorms-td2987547.html) is good for better understanding – saba safavi Apr 25 '21 at 08:04