S3

Amazon's S3 is a bucket-based system for storing data in the cloud.

Example locators

Source locators:

  • s3://bucket/dir/file.csv
  • s3://bucket/dir/

Destination locators:

  • s3://bucket/dir/

At this point, we do not support single-file output to a cloud bucket. This is relatively easy to add, but has not yet been implemented.

Configuration & authentication

The following environment variables are used to authenticate:

  • AWS_ACCESS_KEY_ID (required): The ID for your AWS credentials.
  • AWS_SECRET_ACCESS_KEY (required): The secret part of your AWS credentials.
  • AWS_SESSION_TOKEN (optional): Set this to use temporary AWS crdentials.
  • AWS_DEFAULT_REGION (required): Set this to your AWS region.

Supported features

s3 features:
- cp FROM:
- cp TO:
  --if-exists=overwrite