Spring Mongo Populator one by one

Question

I'm using MongoDB and Spring over Kotlin and i want my application to populate a MongoDB collection upon startup. (and clean it every time it starts)

My question is, how can i populate the data one by one in order to be fault tolerant in case some of the data I'm populating with is problematic?

my code:

@Configuration
class IndicatorPopulator {
    @Value("classpath:indicatorData.json")
    private lateinit var data: Resource

    @Autowired
    private lateinit var indicatorRepository: IndicatorRepository

    @Bean
    @Autowired
    fun repositoryPopulator(objectMapper: ObjectMapper): Jackson2RepositoryPopulatorFactoryBean {
        val factory = Jackson2RepositoryPopulatorFactoryBean()
        indicatorRepository.deleteAll()
        factory.setMapper(objectMapper)
        factory.setResources(arrayOf(data))
        return factory
    }

What I am looking for is something like:

@Bean
@Autowired
fun repositoryPopulator(objectMapper: ObjectMapper): Jackson2RepositoryPopulatorFactoryBean {
    val factory = Jackson2RepositoryPopulatorFactoryBean()
    indicatorRepository.deleteAll()
    factory.setMapper(objectMapper)
    val arrayOfResources: Array<Resource> = arrayOf(data)
    for (resource in arrayOfResources){
            try{
             factory.setResources(resource)
            } catch(e: Exception){
                 logger.log(e.message)
            }

    }
    return factory
}

Any idea on how to do something like that would be helpful... Thanks in advance.

so if some resources of many contain malformed JSON data, you want to log errors and skip "bad" resources? and you want to do it using `Jackson2RepositoryPopulatorFactoryBean` or anything similar? — naXa stands with Ukraine, Apr 06 '20 at 08:53
Pretty much... not only malformed JSON data also if they are a valid JSON but just not fit for the entity _class. — Shahar Wider, Apr 06 '20 at 12:01

s7vr · Accepted Answer · 2020-04-10T15:42:52.050

There is no built in support for your ask but you can easily provide by tweaking few classes.

Add Custom Jackson 2 Reader

public class CustomJackson2ResourceReader implements ResourceReader {

    private static final Logger logger = LoggerFactory.getLogger(CustomJackson2ResourceReader.class);

    private final Jackson2ResourceReader resourceReader = new Jackson2ResourceReader();

    @Override
    public Object readFrom(Resource resource, ClassLoader classLoader) throws Exception {
        Object result;
        try {
            result = resourceReader.readFrom(resource, classLoader);
        } catch(Exception e) {
            logger.warn("Can't read from resource", e);
            return Collections.EMPTY_LIST;
        }
        return result;
    }
}

Add Custom Jackson 2 Populator

public class CustomJackson2RepositoryPopulatorFactoryBean extends Jackson2RepositoryPopulatorFactoryBean {
    @Override
    protected ResourceReader getResourceReader() {
        return new CustomJackson2ResourceReader();
    }
}

Configuration

@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @Bean
    public AbstractRepositoryPopulatorFactoryBean repositoryPopulator(ObjectMapper objectMapper, KeyValueRepository keyValueRepository) {
        Jackson2RepositoryPopulatorFactoryBean factory = new CustomJackson2RepositoryPopulatorFactoryBean();
        keyValueRepository.deleteAll();
        factory.setMapper(objectMapper);
        factory.setResources(new Resource[]{new ClassPathResource("badclassname.json"), new ClassPathResource("good.json"), new ClassPathResource("malformatted.json")});
        return factory;
    }

}

I've uploading a working example here

score 1 · Answer 2 · answered Apr 19 '20 at 10:41

Using Sagar's Reader & Factory I just adjusted it to fit my needs (Kotlin, and reading resources all from the same JSON file) got me this answer:

@Configuration
class IndicatorPopulator {

@Value("classpath:indicatorData.json")
private lateinit var data: Resource

@Autowired
private lateinit var indicatorRepository: IndicatorRepository

@Autowired
@Bean
fun repositoryPopulator(objectMapper: ObjectMapper): Jackson2RepositoryPopulatorFactoryBean {
    val factory: Jackson2RepositoryPopulatorFactoryBean = CustomJackson2RepositoryPopulatorFactoryBean()
    factory.setMapper(objectMapper)
    // inject your Jackson Object Mapper if you need to customize it:
    indicatorRepository.deleteAll()


    val resources = mutableListOf<Resource>()
    val readTree: ArrayNode = objectMapper.readTree(data.inputStream) as ArrayNode
    for (node in readTree){
        resources.add( InputStreamResource(node.toString().byteInputStream()))
    }
    factory.setResources(resources.toTypedArray())
    return factory
}

}

Spring Mongo Populator one by one

2 Answers2

Linked