9. Resource handling

Spring Cloud AWS does not modify the default resource loader unless it encounters an explicit configuration with an XML namespace element. The configuration consists of one element for the whole application context that is shown below:

<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	   xmlns:aws-context="http://www.springframework.org/schema/cloud/aws/context"
	   xsi:schemaLocation="http://www.springframework.org/schema/cloud/aws/context
	   http://www.springframework.org/schema/cloud/aws/context/spring-cloud-aws-context.xsd">

	<aws-context:context-credentials>
    		...
        </aws-context:context-credentials>

	<aws-context:context-resource-loader/>
</beans>

Downloading files can be done by using the s3 protocol to reference Amazon S3 buckets and objects inside their bucket. The typical pattern is s3://<bucket>/<object> where bucket is the global and unique bucket name and object is a valid object name inside the bucket. The object name can be a file in the root folder of a bucket or a nested file within a directory inside a bucket.

The next example demonstrates the use of the resource loader to load different resources.

public class SimpleResourceLoadingBean {

	@Autowired
	private ResourceLoader resourceLoader;

	public void resourceLoadingMethod() throws IOException {
		Resource resource = this.resourceLoader.getResource("s3://myBucket/rootFile.log");
		Resource secondResource = this.resourceLoader.getResource("s3://myBucket/rootFolder/subFile");

		InputStream inputStream = resource.getInputStream();
		//read file
	}
}

Since Spring Framework 3.1 the resource loader can also be used to upload files with the org.springframework.core.io.WritableResource interface which is a specialization of the org.springframework.core.io.ResourceLoader interface. Clients can upload files using the WritableResource interface. The next example demonstrates an upload of a resource using the resource loader.

public class SimpleResourceLoadingBean {

    @Autowired
    private ResourceLoader resourceLoader;

    public void writeResource() throws IOException {
        Resource resource = this.resourceLoader.getResource("s3://myBucket/rootFile.log");
        WritableResource writableResource = (WritableResource) resource;
        try (OutputStream outputStream = writableResource.getOutputStream()) {
            outputStream.write("test".getBytes());
        }
    }
}

9.3.1 Uploading multi-part files

Amazon S3 supports multi-part uploads to increase the general throughput while uploading. Spring Cloud AWS by default only uses one thread to upload the files and therefore does not provide parallel upload support. Users can configure a custom org.springframework.core.task.TaskExecutor for the resource loader. The resource loader will queue multiple threads at the same time to use parallel multi-part uploads.

The configuration for a resource loader that uploads with 10 Threads looks like the following

<beans ...>
  <aws-context:context-resource-loader task-executor="executor" />
  <task:executor id="executor" pool-size="10" queue-capacity="0" rejection-policy="CALLER_RUNS" />
</beans>

Warning

Spring Cloud AWS consumes up to 5 MB (at a minimum) of memory per thread. Therefore each parallel thread will incur a memory footprint of 5 MB in the heap, and a thread size of 10 will consume therefore up to 50 mb of heap space. Spring Cloud AWS releases the memory as soon as possible. Also, the example above shows that there is no queue-capacity configured, because queued requests would also consume memory.

public class SimpleResourceLoadingBean {

	@Autowired
	private AmazonS3 amazonS3;

	public void withTransferManager() {
		TransferManager transferManager = new TransferManager(this.amazonS3);
		transferManager.upload("myBucket","filename",new File("someFile"));
	}
}

The Spring resource loader also supports collecting resources based on an Ant-style path specification. Spring Cloud AWS offers the same support to resolve resources within a bucket and even throughout buckets. The actual resource loader needs to be wrapped with the Spring Cloud AWS one in order to search for s3 buckets, in case of non s3 bucket the resource loader will fall back to the original one. The next example shows the resource resolution by using different patterns.

public class SimpleResourceLoadingBean {

	private ResourcePatternResolver resourcePatternResolver;

 	@Autowired
    public void setupResolver(ApplicationContext applicationContext, AmazonS3 amazonS3){
        this.resourcePatternResolver = new PathMatchingSimpleStorageResourcePatternResolver(amazonS3, applicationContext);
    }

 	public void resolveAndLoad() throws IOException {
 		Resource[] allTxtFilesInFolder =  this.resourcePatternResolver.getResources("s3://bucket/name/*.txt");
 		Resource[] allTxtFilesInBucket =  this.resourcePatternResolver.getResources("s3://bucket/**/*.txt");
 		Resource[] allTxtFilesGlobally =  this.resourcePatternResolver.getResources("s3://**/*.txt");
 	}
}

	Warning
	Resolving resources throughout all buckets can be very time consuming depending on the number of buckets a user owns.

CloudFormation also allows to create buckets during stack creation. These buckets will typically have a generated name that must be used as the bucket name. In order to allow application developers to define static names inside their configuration, Spring Cloud AWS provides support to resolve the generated bucket names. Application developers can use the org.springframework.cloud.aws.core.env.ResourceIdResolver interface to resolve the physical names that are generated based on the logical names.

The next example shows a bucket definition inside a CloudFormation stack template. The bucket will be created with a name like integrationteststack-sampleBucket-23qysofs62tc2

{
	"Resources": {
		"sampleBucket": {
			"Type": "AWS::S3::Bucket"
		}
	}
}

Application developers can resolve that name and use it to load resources as shown in the next example below.

public class SimpleResourceLoadingBean {

	private final ResourceLoader loader;
	private final ResourceIdResolver idResolver;

	@Autowired
	public SimpleResourceLoadingBean(ResourceLoader loader, ResourceIdResolver idResolver) {
		this.loader = loader;
		this.idResolver = idResolver;
	}

	public void resolveAndLoad() {
		String sampleBucketName = this.idResolver.
			resolveToPhysicalResourceId("sampleBucket");
		Resource resource = this.loader.
			getResource("s3://" + sampleBucketName + "/test");
	}
}