how can i pull an object out of S3 bucket, a file. and read it into a object in spring-boot and java. So i have a file in S3 that contains multiple json rows, i want to read it from there into a list of objects.

now im following this example:

https://codetinkering.com/aws-s3-select-api-java-example/

which works if i want to stream it, but i want to read into a list of objects.

The file contains multiple rows of data as such, each row is a json object, in AWS ui this works fine with a s3 select

{
   "name":"ABC"
   "surname":"DEF"
}
{
   "name":"ABC"
   "surname":"DEF"
}

CodePudding user response：

Here is the implementation, first add this dependency in pom.xml

         <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>2.17.285</version>
         </dependency>

File content - This file (filename.txt) is present in AWS S3 bucket

{
   "name":"rohit",
   "surname":"sharma"
}
{
   "name":"virat",
   "surname":"kohli"
}

Java Classes

public class Person {
    private String name;
    private String surname;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getSurname() {
        return surname;
    }

    public void setSurname(String surname) {
        this.surname = surname;
    }

    @Override
    public String toString() {
        return "Person{"  
                "name='"   name   '\''  
                ", surname='"   surname   '\''  
                '}';
    }
}

@Configuration
public class AwsS3ClientConfig {

    @Bean
    public S3Client s3Client(){
        AwsBasicCredentials awsBasicCredentials = AwsBasicCredentials.create("ACCESS_KEY_ID", "SECRET_ACCESS_KEY");
        return S3Client
                .builder()
                .region(Region.US_EAST_1)
                .credentialsProvider(StaticCredentialsProvider.create(awsBasicCredentials))
                .build();
    }
}

AwsS3Service V1 - Reading S3 file records and create object list

@Service
public class AwsS3Service {

    private final S3Client s3Client;

    @Autowired
    public AwsS3Service(S3Client s3Client) {
        this.s3Client = s3Client;
    }


    public List<Person> readFileAndCreateList(String bucketName, String keyName) throws IOException {
        final Path file = readFile(bucketName, keyName);
        return convertFileToList(file);
    }

    private Path readFile(String bucketName, String keyName) throws IOException {
        GetObjectRequest getObjectRequest = GetObjectRequest
                .builder()
                .bucket(bucketName)
                .key(keyName)
                .build();

        final byte[] bytes = s3Client
                .getObject(getObjectRequest)
                .readAllBytes();
        final Path path = Paths.get("demo.txt");
        Files.write(path, bytes);
        return path;
    }

    private List<Person> convertFileToList(Path path) throws IOException {
        final List<String> lines = Files.readAllLines(path);
        StringBuilder json = new StringBuilder();
        List<Person> persons=new ArrayList<>();
        for (String line : lines) {
            if ("{".equals(line)) {
                json = new StringBuilder("{");
            } else if ("}".equals(line)) {
                json.append("}");
               persons.add(new ObjectMapper()
                        .readValue(json.toString(), Person.class));
            } else {
                json.append(line.trim());
            }
        }
        return persons;
    }
}

AwsS3Service V2 - Reading S3 file records in-memory and create object list

@Service
public class AwsS3Service {

    private final S3Client s3Client;

    @Autowired
    public AwsS3Service(S3Client s3Client) {
        this.s3Client = s3Client;
    }


    public List<Person> readFileAndCreateObjectList(String bucketName, String keyName) throws IOException {
        final List<String> lines = readFile(bucketName, keyName);
        return convertFileLinesToObjectList(lines);
    }

    private List<String> readFile(String bucketName, String keyName) throws IOException {
        GetObjectRequest getObjectRequest = GetObjectRequest
                .builder()
                .bucket(bucketName)
                .key(keyName)
                .build();

        byte[] bytes;
        try (ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()) {
            s3Client
                    .getObject(getObjectRequest)
                    .transferTo(byteArrayOutputStream);
            bytes = byteArrayOutputStream.toByteArray();
        }

        List<String> lines=new ArrayList<>();
        try(ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
            InputStreamReader inputStreamReader = new InputStreamReader(byteArrayInputStream);
            BufferedReader bufferedReader=new BufferedReader(inputStreamReader)){
           while (bufferedReader.ready()){
              lines.add(bufferedReader.readLine());
           }
        }
        return lines;
    }

    private List<Person> convertFileLinesToObjectList(List<String> lines) throws IOException {
        StringBuilder json = new StringBuilder();
        List<Person> persons = new ArrayList<>();
        for (String line : lines) {
            if ("{".equals(line)) {
                json = new StringBuilder("{");
            } else if ("}".equals(line)) {
                json.append("}");
                persons.add(new ObjectMapper()
                        .readValue(json.toString(), Person.class));
            } else {
                json.append(line.trim());
            }
        }
        return persons;
    }
}

@SpringBootApplication
public class DemoApplication implements CommandLineRunner {

    private final AwsS3Service awsS3Service;

    @Autowired
    public DemoApplication(AwsS3Service awsS3Service) {
        this.awsS3Service = awsS3Service;
    }

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class);
    }

    @Override
    public void run(String... args) throws Exception {
        //KEY_NAME==filename.txt
        final List<Person> peoples =
                awsS3Service
                        .readFileAndCreateList("BUCKET_NAME", "KEY_NAME");
        System.out.println(peoples);
    }
}

Output

[Person{name='rohit', surname='sharma'}, Person{name='virat', surname='kohli'}]