How can I implement solr redundancy on the haystack side?

Question

I've configured solr+zookeeper and now I want to set up redundancy in haystack. I mean that if haystack cannot connect to master server he should choose slave ...

settings.py:

HAYSTACK_CONNECTIONS = {
    'master': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr',
        'TIMEOUT': 10,
        'SILENTLY_FAIL': True,
        'EXCLUDED_INDEXES': [],
        'INCLUDE_SPELLING': True,
    },
    'slave': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:7574/solr',
        'TIMEOUT': 10,
        'SILENTLY_FAIL': True,
        'EXCLUDED_INDEXES': [],
        'INCLUDE_SPELLING': True
    },
}
HAYSTACK_CONNECTIONS['default'] = HAYSTACK_CONNECTIONS['master']
HAYSTACK_ROUTERS = [
    'apps.search.routers.MasterRouter',
    'apps.search.routers.SlaveRouter',
    'haystack.routers.DefaultRouter',
]

I was thinking that with configuration above the system will be able to choose the correct server. I turned off "master" server and I see that haystack still try to connect to the "master". Why? Should I set something else?

UPDATE

So, after I was thinking a lot of this issue I updated my router.py file:

class MasterRouter(routers.BaseRouter):
    def for_write(self, **hints):
        return 'master'

    def for_read(self, **hints):
        if check_connection('master'):
          return 'master'
        elif check_connection('slave'):
          return 'slave'

        return None


class SlaveRouter(routers.BaseRouter):
    def for_write(self, **hints):
        return None

    def for_read(self, **hints):
        return 'slave'


def check_connection(type):
  url = settings.HAYSTACK_CONNECTIONS[type]['URL']
  try:
    urllib2.urlopen(url,timeout=1)
    return True
  except urllib2.URLError as err: pass
  return False

Is this a good approach ?

What is in your `apps.search.routers.MasterRouter` and `apps.search.routers.SlaveRouter`? — twil, Aug 04 '13 at 16:30
Now I added there a special code for checking connection availability. Is it good approach? — user1119698, Aug 05 '13 at 07:26

score 0 · Answer 1 · answered Aug 05 '13 at 22:34

Well it should work. But I don't see need for redundant haystack.routers.DefaultRouter and apps.search.routers.SlaveRouter (all your logic in MasterRouter).

As for check_connection() then more straightforward version is

def check_connection(type):
  url = settings.HAYSTACK_CONNECTIONS[type]['URL']
  try:
    urllib2.urlopen(url,timeout=1)
    return True
  except urllib2.URLError as err:
    return False

If you want further to improve it you can send some real query and check for results, eg.

def check_connection(type):
  url = "{}?{}".format(settings.HAYSTACK_CONNECTIONS[type]['URL'], 'q=test')
  try:
    res = urllib2.urlopen(url, timeout=1)
    # TODO: parse Solr output and decide what to return 
    return True
  except urllib2.URLError as err:
    return False

Refere to Solr docs for help on queries and output.

How can I implement solr redundancy on the haystack side?

1 Answers1