I'm trying to save a large List ( 100K items) created from an XLS file with the CrudRepository.saveAll() method. Because of data size, the saveAll method takes a long time to run, so I'm trying to partition the list using Apache Commons Collections ListUtils.partition and calling the saveAll method in a loop, to improve performance:
public interface ClaimsSegmentRepository extends CrudRepository<ClaimsSegment, String> {}
public class FileLoaderService {
@Async("asyncFileLoadExecutor")
public CompletableFuture<List<String>> loadFile(String fileName) throws Exception {
for(List<ClaimsSegment> claimsSegmentGrp : ListUtils.partition(claimsSegmentList, MAX_PARTITION_SIZE)) {
CompletableFuture.supplyAsync(() -> claimSegmentRepo.saveAll(claimsSegmentGrp))
.thenAccept(claimSegmentList -> {
//Processing child objects for Claim Segment List
});
}
}
This is throwing the error HHH000010: On release of batch it still contained JDBC statements, so I read some more online and saw this recommendation for my repo class:
public interface ClaimsSegmentRepository extends CrudRepository<ClaimsSegment, String> {
@Async
<S extends ClaimsSegment> CompletableFuture<S> saveAll()
}
However, that is resulting in a compilation error complaining about return type mismatch:
org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name 'loadController':
Unsatisfied dependency expressed through field 'loaderSvc'; nested exception is org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'fileLoaderService' defined in file [Path\to\file\FileLoaderService.class]:
Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException:
Failed to instantiate [nom.side.xls.service.FileLoaderService]:
Constructor threw exception; nested exception is java.lang.Error: Unresolved compilation problem:
The method saveAll(Iterable<ClaimsSegment>) is ambiguous for the type ClaimsSegmentRepository
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) ~[spring-beans-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-2.7.4.jar:2.7.4]
at nom.side.xls.ExcelFileLoaderApplication.main(ExcelFileLoaderApplication.java:10) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.7.4.jar:2.7.4]
...
The @Async annotation works fine for the find methods, but not for save and saveAll. Could someone please point out what I'm doing wrong?
Thanks,
CodePudding user response：
To get a bulk insert with Spring Boot and Spring Data JPA you need only two things:
- set the option spring.jpa.properties.hibernate.jdbc.batch_size to appropriate value you need (for example: 20)
- use saveAll() method of your repo with the list of entities prepared for inserting.
Working example is here.