Hevo lets you load data from files in an S3 bucket into your data warehouse.
Provide S3 connection details on S3 Connection Settings page. You will have the following options in the connection details block:
- Source Name - A unique name for this source
- Access Key ID - AWS access key ID which has permissions to read from the given bucket
- Secret Access Key - AWS Secret Access Key for the above Access Key ID
- Bucket - The name of the bucket from which you want to ingest data.
- Prefix - Path Prefix for the data directory. By default, the files are listed from the root of the directory.
- Bucket Region - Choose the AWS region where the bucket is located.
- File Format - Choose a file format. Hevo currently supports JSON and CSV formats. Let us know if you need support for a different format.
- Files GZipped - Select this option if the files in your S3 bucket are GZipped.
- Create Event Types from folders - Select this option when your prefix path has subdirectories containing files in different formats. Hevo, in that case, will read each of the subdirectories as a separate event type. Please note, that any files lying at the prefix path(and not in any of the subdirectories) will be ignored.
Order of Ingesting files
- The files are read in the lexical order of their names and files once read will not be re-read until the ingestion job is restarted.
- Only the files at the path mentioned as Prefix are read. The subdirectories are ignored.