Handle big, but not too big files from S3 buckets

(, en)

So you have files up to 1 GiB on your S3 storage and you need to read them in on a regular basis?

  1. read them in the memory if small.
  2. download them to temporary files, read them and delete afterwards.

In particular the »delete afterwards« can be tricky in Java:

public InputStream getContentStream(String key) throws IOException {
  S3Object object = s3Client.getObject(bucket, key);
  long size = object.getObjectMetadata().getContentLength();
  if (size > MAX_IN_MEMORY_SIZE) {
    return downloadAndOpen(object);
  } else {
    return object.getObjectContent();

private InputStream downloadAndOpen(S3Object object) throws IOException {
  Path tempFile = Files.createTempFile(bucket + "-", null);
  try {
    s3Client.getObject(new GetObjectRequest(bucket, object.getKey()), tempFile.toFile());
  } catch (Exception ex) {
    throw ex;

  return Files.newInputStream(tempFile, StandardOpenOption.DELETE_ON_CLOSE);

Use getContentStream with try with resources.