Handle big, but not too big files from S3 buckets

(29. November 2020, en)

So you have files up to 1 GiB on your S3 storage and you need to read them in on a regular basis?

read them in the memory if small.
download them to temporary files, read them and delete afterwards.

In particular the »delete afterwards« can be tricky in Java:

public InputStream getContentStream(String key) throws IOException {
  S3Object object = s3Client.getObject(bucket, key);
  long size = object.getObjectMetadata().getContentLength();
  if (size > MAX_IN_MEMORY_SIZE) {
    return downloadAndOpen(object);
  } else {
    return object.getObjectContent();
  }
}

private InputStream downloadAndOpen(S3Object object) throws IOException {
  Path tempFile = Files.createTempFile(bucket + "-", null);
  try {
    s3Client.getObject(new GetObjectRequest(bucket, object.getKey()), tempFile.toFile());
  } catch (Exception ex) {
    Files.delete(tempFile);
    throw ex;
  }

  return Files.newInputStream(tempFile, StandardOpenOption.DELETE_ON_CLOSE);
}

Use getContentStream with try with resources.