FetchS3Object

Description

Retrieves the contents of an S3 Object and writes it to the content of a FlowFile

Tags

AWS, Amazon, Fetch, Get, S3

Properties

In the list below required Properties are shown with an asterisk (*). Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display Name	API Name	Default Value	Allowable Values	Description
Bucket *	Bucket	${s3.bucket}		The S3 Bucket to interact with Supports Expression Language, using FlowFile attributes and Environment variables.
Object Key *	Object Key	${filename}		The S3 Object Key to use. This is analogous to a filename for traditional file systems. Supports Expression Language, using FlowFile attributes and Environment variables.
Region *	Region	US West (Oregon)	AWS GovCloud (US) AWS GovCloud (US-East) US East (N. Virginia) US East (Ohio) US West (N. California) US West (Oregon) EU (Ireland) EU (London) EU (Paris) EU (Frankfurt) EU (Zurich) EU (Stockholm) EU (Milan) EU (Spain) Asia Pacific (Hong Kong) Asia Pacific (Mumbai) Asia Pacific (Hyderabad) Asia Pacific (Singapore) Asia Pacific (Sydney) Asia Pacific (Jakarta) Asia Pacific (Melbourne) Asia Pacific (Tokyo) Asia Pacific (Seoul) Asia Pacific (Osaka) South America (Sao Paulo) China (Beijing) China (Ningxia) Canada (Central) Canada West (Calgary) Middle East (UAE) Middle East (Bahrain) Africa (Cape Town) US ISO East US ISOB East (Ohio) US ISO West Israel (Tel Aviv) Use 's3.region' Attribute	The AWS Region to connect to.
AWS Credentials Provider Service *	AWS Credentials Provider service		Controller Service: AWSCredentialsProviderService Implementations: AWSCredentialsProviderControllerService	The Controller Service that is used to obtain AWS credentials provider
Communications Timeout *	Communications Timeout	30 secs		The amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out.
Version	Version			The Version of the Object to download Supports Expression Language, using FlowFile attributes and Environment variables.
SSL Context Service	SSL Context Service		Controller Service: SSLContextService Implementations: StandardRestrictedSSLContextService StandardSSLContextService	Specifies an optional SSL Context Service that, if provided, will be used to create connections
Endpoint Override URL	Endpoint Override URL			Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. Supports Expression Language, using Environment variables.
Signer Override	Signer Override	Default Signature	Default Signature Signature Version 4 Signature Version 2 Custom Signature	The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation.
Custom Signer Class Name *	custom-signer-class-name			Fully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth.Signer interface. Supports Expression Language, using Environment variables. This property is only considered if: the property Signer Override has a value of CustomSignerType
Custom Signer Module Location	custom-signer-module-location			Comma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any). Supports Expression Language, using Environment variables. This property is only considered if: the property Signer Override has a value of CustomSignerType
Encryption Service	encryption-service		Controller Service: AmazonS3EncryptionService Implementations: StandardS3EncryptionService	Specifies the Encryption Service Controller used to configure requests. PutS3Object: For backward compatibility, this value is ignored when 'Server Side Encryption' is set. FetchS3Object: Only needs to be configured in case of Server-side Customer Key, Client-side KMS and Client-side Customer Key encryptions.
Proxy Configuration Service	proxy-configuration-service		Controller Service: ProxyConfigurationService Implementations: StandardProxyConfigurationService	Specifies the Proxy Configuration Controller Service to proxy network requests. Supported proxies: HTTP + AuthN
Requester Pays *	requester-pays	False	True False	If true, indicates that the requester consents to pay any charges associated with retrieving objects from the S3 bucket. This sets the 'x-amz-request-payer' header to 'requester'.
Range Start	range-start			The byte position at which to start reading from the object. An empty value or a value of zero will start reading at the beginning of the object. Supports Expression Language, using FlowFile attributes and Environment variables.
Range Length	range-length			The number of bytes to download from the object, starting from the Range Start. An empty value or a value that extends beyond the end of the object will read to the end of the object. Supports Expression Language, using FlowFile attributes and Environment variables.

Dynamic Properties

This component does not support dynamic properties.

Relationships

Name	Description
failure	If the Processor is unable to process a given FlowFile, it will be routed to this Relationship.
success	FlowFiles are routed to this Relationship after they have been successfully processed.

Reads Attributes

This processor does not read attributes.

Writes Attributes

Name	Description
absolute.path	The path of the file
filename	The name of the file
hash.algorithm	MD5
hash.value	The MD5 sum of the file
mime.type	If S3 provides the content type/MIME type, this attribute will hold that file
path	The path of the file
s3.additionalDetails	The S3 supplied detail from the failed operation
s3.bucket	The name of the S3 bucket
s3.encryptionStrategy	The name of the encryption strategy that was used to store the S3 object (if it is encrypted)
s3.errorCode	The S3 moniker of the failed operation
s3.errorMessage	The S3 exception message from the failed operation
s3.etag	The ETag that can be used to see if the file has changed
s3.exception	The class name of the exception thrown during processor execution
s3.expirationTime	If the file has an expiration date, this attribute will be set, containing the milliseconds since epoch in UTC time
s3.expirationTimeRuleId	The ID of the rule that dictates this object's expiration time
s3.sseAlgorithm	The server side encryption algorithm of the object
s3.statusCode	The HTTP error code (if available) from the failed operation
s3.url	The URL that can be used to access the S3 object
s3.version	The version of the S3 object

State Management

This component does not store state.

Restricted

This component is not restricted.

Input Requirement

This component requires an incoming relationship.

Example Use Cases

Use Case 1

Fetch a specific file from S3

Configuration

The "Bucket" property should be set to the name of the S3 bucket that contains the file. Typically this is defined as an attribute on an incoming FlowFile, so this property is set to ${s3.bucket}.
The "Object Key" property denotes the fully qualified filename of the file to fetch. Typically, the FlowFile's filename attribute is used, so this property is set to ${filename}.
The "Region" property must be set to denote the S3 region that the Bucket resides in. If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_REGION}.

The "AWS Credentials Provider service" property should specify an instance of the AWSCredentialsProviderControllerService in order to provide credentials for accessing the file.

Example Use Cases Involving Other Components

Multiprocessor Use Case 1

Retrieve all files in an S3 bucket

Components Involved

• ListS3
  1. The "Bucket" property should be set to the name of the S3 bucket that files reside in. If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_SOURCE_BUCKET}.
  2. The "Region" property must be set to denote the S3 region that the Bucket resides in. If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_SOURCE_REGION}.
  3. The "AWS Credentials Provider service" property should specify an instance of the AWSCredentialsProviderControllerService in order to provide credentials for accessing the bucket.
  4. The 'success' Relationship of this Processor is then connected to FetchS3Object.
• FetchS3Object
  1. "Bucket" = "${s3.bucket}"
  2. "Object Key" = "${filename}"
  3. The "AWS Credentials Provider service" property should specify an instance of the AWSCredentialsProviderControllerService in order to provide credentials for accessing the bucket.
  4. The "Region" property must be set to the same value as the "Region" property of the ListS3 Processor.

Multiprocessor Use Case 2

Retrieve only files from S3 that meet some specified criteria

Components Involved

• ListS3
  1. The "Bucket" property should be set to the name of the S3 bucket that files reside in. If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_SOURCE_BUCKET}.
  2. The "Region" property must be set to denote the S3 region that the Bucket resides in. If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_SOURCE_REGION}.
  3. The "AWS Credentials Provider service" property should specify an instance of the AWSCredentialsProviderControllerService in order to provide credentials for accessing the bucket.
  4. The 'success' Relationship of this Processor is then connected to RouteOnAttribute.
• RouteOnAttribute
  1. If you would like to "OR" together all of the conditions (i.e., the file should be retrieved if any of the conditions are met), set "Routing Strategy" to "Route to 'matched' if any matches".
  2. If you would like to "AND" together all of the conditions (i.e., the file should only be retrieved if all of the conditions are met), set "Routing Strategy" to "Route to 'matched' if all match".
  3. For each condition that you would like to filter on, add a new property. The name of the property should describe the condition. The value of the property should be an Expression Language expression that returns true if the file meets the condition or false if the file does not meet the condition.
  4. Some attributes that you may consider filtering on are:
     • filename (the name of the file)
     • s3.length (the number of bytes in the file)
     • s3.tag.<tag name> (the value of the s3 tag with the name tag name)
     • s3.user.metadata.<key name> (the value of the user metadata with the key named key name)
  5. For example, to fetch only files that are at least 1 MB and have a filename ending in .zip we would set the following properties:
     • "Routing Strategy" = "Route to 'matched' if all match"
     • "At least 1 MB" = "${s3.length:ge(1000000)}"
     • "Ends in .zip" = "${filename:endsWith('.zip')}"
  6. Auto-terminate the unmatched Relationship.
  7. Connect the matched Relationship to the FetchS3Object processor.
• FetchS3Object
  1. "Bucket" = "${s3.bucket}"
  2. "Object Key" = "${filename}"
  3. The "AWS Credentials Provider service" property should specify an instance of the AWSCredentialsProviderControllerService in order to provide credentials for accessing the bucket.
  4. The "Region" property must be set to the same value as the "Region" property of the ListS3 Processor.

Multiprocessor Use Case 3

Retrieve new files as they arrive in an S3 bucket

This method of retrieving files from S3 is more efficient than using ListS3 and more cost effective. It is the pattern recommended by AWS. However, it does require that the S3 bucket be configured to place notifications on an SQS queue when new files arrive. For more information, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html

Components Involved

• GetSQS
  1. The "Queue URL" must be set to the appropriate URL for the SQS queue. It is recommended that this property be parameterized, using a value such as #{SQS_QUEUE_URL}.
  2. The "Region" property must be set to denote the SQS region that the queue resides in. It's a good idea to parameterize this property by setting it to something like #{SQS_REGION}.
  3. The "AWS Credentials Provider service" property should specify an instance of the AWSCredentialsProviderControllerService in order to provide credentials for accessing the bucket.
  4. The 'success' relationship is connected to EvaluateJsonPath.
• EvaluateJsonPath
  1. "Destination" = "flowfile-attribute"
  2. "s3.bucket" = "$.Records[0].s3.bucket.name"
  3. "filename" = "$.Records[0].s3.object.key"
  4. The 'success' relationship is connected to FetchS3Object.
• FetchS3Object
  1. "Bucket" = "${s3.bucket}"
  2. "Object Key" = "${filename}"
  3. The "Region" property must be set to the same value as the "Region" property of the GetSQS Processor.
  4. The "AWS Credentials Provider service" property should specify an instance of the AWSCredentialsProviderControllerService in order to provide credentials for accessing the bucket.

System Resource Considerations

This component does not specify system resource considerations.

See Also

CopyS3Object, DeleteS3Object, GetS3ObjectMetadata, ListS3, PutS3Object, TagS3Object