8

So, back again

I have a JHipster generated project which uses an elasticsearch java client embedded in spring boot.

I have recently done some major changes to the datasets since we've been migrating a whole new bunch of data from different repositories

When deploying the application it all works fine, all SearchRepositories are loaded with no problem and all search capabilities roll smooth

The issues come when running from the test environment. There have been no changes what so ever to the application-test.yml file nor to the elasticsearch java config file.

We have some code which updates the indices and I've run it several times, it seems to update the clusters indices just fine, but where I'm suffering is in the target folder, it just won't create the new indices

There are 12 indices that I cannot get in to the target folder when running in test mode, however, only 5 of them fail in their ResourceIntTest because of the error mentioned in the title

I don't want to fill this post with hundreds of irrelevant lines of code, so suffice for now to include the workaround that helps test not to fail:

When in the initTest of the 5 failing test cases, if I write the following line (obviously changing the class name in each different case):

surveyDataQualitySearchRepository.save(surveyDataQualityRepository.findAll());

Then the index will create itself and the testcase will not fail, however this shouldn't be necessary to do manually, it should be created when the resetIndex method in the IndexReinitializer class is called upon deployment

resetIndex:

@PostConstruct
public void resetIndex() {
        long t = currentTimeMillis();
        elasticsearchTemplate.deleteIndex("_all");
        t = currentTimeMillis() - t;
        logger.debug("ElasticSearch indexes reset in {} ms", t);
    }

Commenting this piece of code also allows all indices to be loaded, but it should not be commented as this serves as an updater for the indices, plus it works fine in an old version of the application which is still pointing to the old dataset

All help will be very welcome, I've been on this almost a full day now trying to understand where the error is coming from, I'm also more than happy to upload any pieces of code that may be relevant to anyone willing to help here.

EDIT To add code for the indices rebuild as requested via comments

@Test
    public void synchronizeData() throws Exception{
        resetIndex();
        activePharmaIngredientSearchRepository.save(activePharmaIngredientRepository.findAll());
        countrySearchRepository.save(countryRepository.findAll());
        dosageUnitSearchRepository.save(dosageUnitRepository.findAll());
        drugCategorySearchRepository.save(drugCategoryRepository.findAll());
        drugQualityCategorySearchRepository.save(drugQualityCategoryRepository.findAll());

        formulationSearchRepository.save(formulationRepository.findAll());
        innDrugSearchRepository.save(innDrugRepository.findAll());
        locationSearchRepository.save(locationRepository.findAll());
        manufacturerSearchRepository.save(manufacturerRepository.findAll());
        outletTypeSearchRepository.save(outletTypeRepository.findAll());
        publicationSearchRepository.save(publicationRepository.findAll());
        publicationTypeSearchRepository.save(publicationTypeRepository.findAll());
        qualityReferenceSearchRepository.save(qualityReferenceRepository.findAll());
        reportQualityAssessmentAssaySearchRepository.save(reportQualityAssessmentAssayRepository.findAll());
        //rqaaQualitySearchRepository.save(rqaaQualityRepository.findAll());
        rqaaTechniqueSearchRepository.save(rqaaTechniqueRepository.findAll());
        samplingTypeSearchRepository.save(samplingTypeRepository.findAll());
        //surveyDataQualitySearchRepository.save(surveyDataQualityRepository.findAll());
        surveyDataSearchRepository.save(surveyDataRepository.findAll());
        techniqueSearchRepository.save(techniqueRepository.findAll());
        tradeDrugApiSearchRepository.save(tradeDrugApiRepository.findAll());
        tradeDrugSearchRepository.save(tradeDrugRepository.findAll());
        publicationDrugTypesSearchRepository.save(publicationDrugTypesRepository.findAll());
        wrongApiSearchRepository.save(wrongApiRepository.findAll());
    }

    private void resetIndex() {
        long t = currentTimeMillis();
        elasticsearchTemplate.deleteIndex("_all");
        t = currentTimeMillis() - t;
        logger.debug("ElasticSearch indexes reset in {} ms", t);
    }
Steven
  • 1,236
  • 1
  • 13
  • 37
  • 1
    Are the ES indexes related to an entity, or are they created separately? That `resetIndex` method deletes all indexes from the embedded elasticsearch (which stores data in target) – Jon Ruddell May 17 '17 at 18:15
  • The indexes are related to all entities in the domain. I don't fully understand everything that goes on in the embedded elasticsearch so I can't fully explain how the resetIndex works, but what I know for sure is that it's not the culprit of the indexes not creating themselves, as the code deletes all indices, so if this was the culprit I wouldn't be facing only having some indices – Steven May 17 '17 at 18:28
  • 2
    I has a similar problem. Running spring-data-es with updates worked fine in normal run configuration but would not create an index when running tests. Debugging the es client I found two mappings for an index that were off. One mapping was mine, the other I have no idea where it came from. My advice: if you have a workaround just go with it, make the tests work and move on, it isn't worth the time to figure out what is going on. I lost like 2 days :( I think the only other option would be to reproduce the error in a simple project and report to spring-data-es github and link your test repo. – Tim Schimandle May 18 '17 at 15:19
  • Yeah, I have a tendency to do that too, there are several open questions of mine on SO that I never found a real solution to, but on this particular one, I'm really willing to understand what is going on, as data is an essential component of our applications, we are basically data driven organization if that actually means anything. Thank you for your comment! – Steven May 18 '17 at 15:26
  • Given that the issue comes from the creation of indices in your embedded instance, perhaps sharing the code that creates the indices that "are not found in the target folder" could help understand your issue. – Adonis May 24 '17 at 09:11
  • @asettouf done! – Steven May 24 '17 at 10:06
  • BTW, after going over your question again, I'm not quite sure what code you needed, the code I've uploaded is the one we activate manually in order to synchronize the ES clusters on the servers. What should take care of creating the indices upon deployment is an annotation on the domain class ie:`@Document(indexName = "country")` – Steven May 24 '17 at 10:54

1 Answers1

4

Please try to update to the latest version of spring-data-elasticsearch

ignacio.suay
  • 737
  • 6
  • 7