Introduction
I am currently working on a project to achieve a bulk insertion in a Couchbase database. For this, I chose Quarkus Reactive to get data from the client. And I used a Flux to insert all the data to database. For each data inserted, I send a response, example of a response:
[
{
"data": {
"id": "1"
"created": "2022-02-22T00:28:34Z",
"lastName": "DUPOND",
"firstName": "Marianne",
"gender": "FEMALE",
"title": "MRS",
"addresses": [
{
"street1": "rue du Testeur",
"zipCode": "34000",
"city": "MONTPELLIER",
"country": "France",
"typeKey": "PERSONAL"
}
],
"phones": [
{
"number": "0610529856",
"typeKey": "PERSONAL"
}
],
"emails": [
{
"email": "marianne.dupond@test.com",
"typeKey": "PERSONAL"
}
],
"birthday": "1987-11-13",
"languages": [
"FR",
"EN"
],
"socialNetworks": [
{
"socialNetworkId": "marianne.dupond@twitter",
"typeKey": "PERSONAL"
}
],
"customFields": [
{
"customFieldKey": "MAIDEN_NAME",
"value": "DURAND"
}
]
},
"result": "201",
"id":"1"
},
{
"data": {
"id" : "2"
"created": "2022-02-22T00:28:34Z",
"lastName": "BOND",
"firstName": "James",
"gender": "MALE",
"title": "MR",
"addresses": [
{
"street1": "boulevard des services secrets",
"zipCode": "10000",
"city": "LONDRES",
"country": "Royaume-Uni",
"typeKey": "PERSONAL"
}
],
"phones": [
{
"number": "0600700700",
"typeKey": "PERSONAL"
}
],
"emails": [
{
"email": "james.bond@test.com",
"typeKey": "PERSONAL"
}
],
"birthday": "1950-07-07",
"languages": [
"EN"
],
"socialNetworks": [
{
"socialNetworkId": "james.bond@facebook",
"typeKey": "PERSONAL"
}
],
"customFields": [
{
"customFieldKey": "VIP",
"value": "1"
}
]
},
"result": "201",
"id": "2"
}
]
Problematic
But currently the returned response is:
[
{
"data": {
"id": "1"
"created": "2022-02-22T00:37:39Z",
"lastName": "DUPOND",
"firstName": "Marianne",
"gender": "FEMALE",
"title": "MRS",
"addresses": [
{
"street1": "rue du Testeur",
"zipCode": "34000",
"city": "MONTPELLIER",
"country": "France",
"typeKey": "PERSONAL"
}
],
"phones": [
{
"number": "0610529856",
"typeKey": "PERSONAL"
}
],
"emails": [
{
"email": "marianne.dupond@test.com",
"typeKey": "PERSONAL"
}
],
"birthday": "1987-11-13",
"languages": [
"FR",
"EN"
],
"socialNetworks": [
{
"socialNetworkId": "marianne.dupond@twitter",
"typeKey": "PERSONAL"
}
],
"customFields": [
{
"customFieldKey": "MAIDEN_NAME",
"value": "DURAND"
}
]
},
"result": "201"
},
{
"data": {
"created": "2022-02-22T00:37:39Z",
"lastName": "BOND",
"firstName": "James",
"gender": "MALE",
"title": "MR",
"addresses": [
{
"street1": "boulevard des services secrets",
"zipCode": "10000",
"city": "LONDRES",
"country": "Royaume-Uni",
"typeKey": "PERSONAL"
}
],
"phones": [
{
"number": "0600700700",
"typeKey": "PERSONAL"
}
],
"emails": [
{
"email": "james.bond@test.com",
"typeKey": "PERSONAL"
}
],
"birthday": "1950-07-07",
"languages": [
"EN"
],
"socialNetworks": [
{
"socialNetworkId": "james.bond@facebook",
"typeKey": "PERSONAL"
}
],
"customFields": [
{
"customFieldKey": "VIP",
"value": "1"
}
]
},
"result": "201"
}
]
The id field inside the data and the response field do not appear, and sometimes the id field inside the data of the first element appears, sometimes not.
Affected code
Bulk insertion(flux of single insertion) code:
public Flux<SingleResponseBulk> bulkInsertCustomerProfile(Multi<SingleResponseBulk> singleResponses) {
return Flux.from(singleResponses)
.doOnEach(signal -> {
if (signal.hasValue()) {
SingleResponseBulk singleResponseBulk = Objects.requireNonNull(signal.get());
insertCustomerProfileWithSingleResponseBulk(singleResponseBulk);
}
});
}
private void insertCustomerProfileWithSingleResponseBulk(SingleResponseBulk response) {
try {
insertCustomerProfile(response.getData()).subscribe();
response.setId(response.getData().getId());
response.setResult("201");
} catch (CouchbaseException e) {
response.setResult("400");
response.setError(e.getMessage());
}
}
Single insertion code:
public Mono<CustomerProfile> insertCustomerProfile(CustomerProfile customerProfile) {
customerProfile.setCreated(LocalDateTime.now());
return getNextId().flatMap(counterResult -> {
String id = String.valueOf(counterResult.content());
customerProfile.setId(id);
String key = getDocumentKey(id);
return collection.insert(key, customerProfile)
.flatMap(result -> {
customerProfile.setCas(result.cas());
return Mono.just(customerProfile);
})
.doOnError(CouchbaseException.class, mapCouchbaseExceptionConsumer(id));
});
}
Id generation code:
private Mono<CounterResult> getNextId() {
return collection.binary()
.increment("counter" + COLLECTION_DELIMITER + CUSTOMER_PROFILE_COLLECTION)
.doOnError(CouchbaseException.class, error -> {
String errorMessage = "An exception occurred during id generation.";
throw new RepositoryException(errorMessage, error);
});
}
Question
So I was wondering if it would be possible to solve the problem. If so, how can we do it?
