S3
Amazon's S3 is a bucket-based system for storing data in the cloud.
Example locators
Source locators:
s3://bucket/dir/file.csv
s3://bucket/dir/
Destination locators:
s3://bucket/dir/
At this point, we do not support single-file output to a cloud bucket. This is relatively easy to add, but has not yet been implemented.
Configuration & authentication
The following environment variables are used to authenticate:
AWS_ACCESS_KEY_ID
(required): The ID for your AWS credentials.AWS_SECRET_ACCESS_KEY
(required): The secret part of your AWS credentials.AWS_SESSION_TOKEN
(optional): Set this to use temporary AWS crdentials.AWS_DEFAULT_REGION
(required): Set this to your AWS region.
Supported features
s3 features:
- cp FROM:
- cp TO:
--if-exists=overwrite