Best way to approximate "Includes" in Sphinx?

Question

I have a very long and complex sphinx config file with multiple indexes. The main one (idx_Main) is not only huge it indexes a 2 million+ record table. Sometimes I encounter unexpected search (SphinxQL) results and testing becomes a challenge due to the hour long rotate. Ideally I could run this exact same index on a test table with a few sample records.

However I haven't found a good way to do that;

If I copy and paste using another index (e.g. idx_test) I quickly get out of 'synch (changes to the idx_test not carrying forward to changes in idx_Main index and vice-versa)

if I just change the Select I overwrite the original index

if I change the Select and the index name and the src_index and rotate I seem to mess up Sphinx since I now have a stored index (idx_Main) that is no longer in the configuration file.

In an ideal world I could have an Include in the config so I could store e.g.

{ idx_Main Various Settings }

{ idx_Test Include idx_Main Settings }

But I see no way to do that. What would the most efficient way be then to test changes of a large production index on a smaller table using the same settings w/o corrupting/overwriting the production/main index?

barryhunter · Accepted Answer · 2017-03-04T22:12:01.243

Its not well publicised, but the sphinx config file can itself be a script. (eg PHP, Perl, even SH etc) PHP in particular is great for implementing includes.

sphinx.conf:

#!/usr/bin/php
source src {

...
index inx1 {
  <?php include "main_settings.conf"; ?>
  ...

http://sphinxsearch.com/blog/2013/11/05/sphinx-configuration-features-and-tricks/

But also can just use inheritance in most cases - define an bulk of the settings in a 'template' index - then the final index just needs settings it wants to override (like path!) Its demonstrated in the section on delta http://sphinxsearch.com/docs/current.html#delta-updates

Edited to add, on the point of having a test index, inheritance, along with sql_query_range is rather cool.

Can setup the main index exactly as want (but using sql_query_range), then create a second test index, that ONLY overrides sql_query_range. No need to redeclare sql_query, or any of the sql_attr_* field etc. So change the main index, the test index updates too.

source main {
....
    sql_query_range = SELECT MIN(id),MAX(id) FROM documents
    sql_query = SELECT * FROM documents WHERE id>=$start AND id<=$end
    sql_attr_uint = category_id
...
}

...then create test indexes, that ONLY redeclare sql_query_range

source main_test : main {
    sql_query_range = SELECT 1,1000
}

index main_test : main {
    path = /var/sphinx/main_test
}

That is literally all that is needed to create a second 'sample' index, that reused the exact same settings (other than number of documents!)

(ok, maybe not, if use sql_joined_field and/or sql_attr_multi its more complicated http://sphinxsearch.com/bugs/view.php?id=1591 )

Best way to approximate "Includes" in Sphinx?

1 Answers1