AWS S3 - Goldsky Docs

Pipeline configuration

{
  "sources": [],
  "transforms": [],
  "sinks": [
    {
      "description": "Type.Optional(Type.String())",
      "type": "file",
      "sourceStreamName": "Type.String()",
      "secretName": "Type.String()",
      "path": "Type.String()",
      "format": "Type.Union([Type.Literal('csv'), Type.Literal('parquet')])",
      "partitionColumns": "Type.Optional(Type.String())"
    }
  ]
}

Secrets

Create an Elasticsearch secret with the following CLI command:

goldsky secret create AN_AWS_S3_SECRET --type s3 --value '{
  "accessKeyId": "Type.String()",
  "secretAccessKey": "Type.String()",
  "region": "Type.String()"
}'