Skip to content

Commit af64811

Browse files
authored
Merge pull request #258 from VirtualMetric/DT-420-aws-security-lake-documentation
DT-420-aws-security-lake-documentation
2 parents 80e302f + e68a6cd commit af64811

File tree

1 file changed

+40
-39
lines changed

1 file changed

+40
-39
lines changed

docs/configuration/targets/aws-s3.mdx

Lines changed: 40 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -45,75 +45,76 @@ targets:
4545
4646
### AWS Credentials
4747
48-
| Parameter | Type | Required | Default | Description |
48+
|Parameter|Type|Required|Default|Description|
4949
|---|---|---|---|---|
50-
| **key** | string | Y* | - | AWS access key ID for authentication |
51-
| **secret** | string | Y* | - | AWS secret access key for authentication |
52-
| **session** | string | N | - | Optional session token for temporary credentials |
53-
| **region** | string | Y | - | AWS region (e.g., `us-east-1`, `eu-west-1`) |
54-
| **endpoint** | string | N | - | Custom S3-compatible endpoint URL (for non-AWS S3 services) |
50+
|`key`|string|Y*|-|AWS access key ID for authentication|
51+
|`secret`|string|Y*|-|AWS secret access key for authentication|
52+
|`session`|string|N|-|Optional session token for temporary credentials|
53+
|`region`|string|Y|-|AWS region (e.g., `us-east-1`, `eu-west-1`)|
54+
|`endpoint`|string|N|-|Custom S3-compatible endpoint URL (for non-AWS S3 services)|
5555

56-
\* = Conditionally required. AWS credentials (**key** and **secret**) are required unless using IAM role-based authentication on AWS infrastructure.
56+
\* = Conditionally required. AWS credentials (`key` and `secret`) are required unless using IAM role-based authentication on AWS infrastructure.
5757

5858
### Connection
5959

60-
| Parameter | Type | Required | Default | Description |
60+
|Parameter|Type|Required|Default|Description|
6161
|---|---|---|---|---|
62-
| **name** | string | Y | - | Unique identifier for the target |
63-
| **type** | string | Y | `awss3` | Target type identifier (must be `awss3`) |
64-
| **part_size** | integer | N | `5242880` | Multipart upload part size in bytes (minimum 5MB) |
62+
|`name`|string|Y|-|Unique identifier for the target|
63+
|`type`|string|Y|`awss3`|Target type identifier (must be `awss3`)|
64+
|`part_size`|integer|N|`5242880`|Multipart upload part size in bytes (minimum 5MB)|
6565

6666
### Files
6767

68-
| Parameter | Type | Required | Default | Description |
68+
|Parameter|Type|Required|Default|Description|
6969
|---|---|---|---|---|
70-
| **buckets** | array | Y | - | Array of bucket configurations for file distribution |
71-
| **buckets.bucket** | string | Y | - | S3 bucket name |
72-
| **buckets.name** | string | Y | - | File name template (supports variables: `{date}`, `{time}`, `{unix}`, `{tag}`) |
73-
| **buckets.format** | string | Y | - | Output format: `json`, `multijson`, `avro`, `parquet` |
74-
| **buckets.compression** | string | N | - | Compression algorithm: `gzip`, `snappy`, `deflate` |
75-
| **buckets.extension** | string | N | - | File extension override (defaults to format-specific extension) |
76-
| **buckets.schema** | string | N* | - | Schema definition file path (required for Avro and Parquet formats) |
77-
| **buckets.size** | integer | N | `10485760` | Maximum file size in bytes before rotation (10MB default) |
78-
| **buckets.batch** | integer | N | `1000` | Maximum number of events per file |
79-
80-
\* = Conditionally required. **schema** field is required when **format** is set to `avro` or `parquet`.
70+
|`buckets`|array|Y|-|Array of bucket configurations for file distribution|
71+
|`buckets.bucket`|string|Y|-|S3 bucket name|
72+
|`buckets.name`|string|Y|-|File name template (supports variables: `{date}`, `{time}`, `{unix}`, `{tag}`)|
73+
|`buckets.format`|string|Y|-|Output format: `json`, `multijson`, `avro`, `parquet`|
74+
|`buckets.compression`|string|N|-|Compression algorithm: `gzip`, `snappy`, `deflate`|
75+
|`buckets.extension`|string|N|-|File extension override (defaults to format-specific extension)|
76+
|`buckets.schema`|string|N*|-|Schema definition file path (required for Avro and Parquet formats)|
77+
|`buckets.size`|integer|N|`10485760`|Maximum file size in bytes before rotation (10MB default)|
78+
|`buckets.batch`|integer|N|`1000`|Maximum number of events per file|
79+
80+
\* = Conditionally required. `schema` field is required when `format` is set to `avro` or `parquet`.
8181

8282
### AWS Security Lake
8383

84-
| Parameter | Type | Required | Default | Description |
84+
|Parameter|Type|Required|Default|Description|
8585
|---|---|---|---|---|
86-
| **source** | string | N* | - | Security Lake source identifier |
87-
| **account** | string | N* | - | AWS account ID for Security Lake |
86+
|`source`|string|N*|-|Security Lake source identifier|
87+
|`account`|string|N*|-|AWS account ID for Security Lake|
8888

89-
\* = Conditionally required. When **source**, **region**, and **account** are all provided, files use Security Lake path structure: `ext/{source}/region={region}/accountId={account}/eventDay={date}/{file}`
89+
\* = Conditionally required. When `source`, `region`, and `account` are all provided, files use Security Lake path structure: `ext/{source}/region={region}/accountId={account}/eventDay={date}/{file}`
9090

9191
### Azure Function App Integration
9292

93-
| Parameter | Type | Required | Default | Description |
93+
|Parameter|Type|Required|Default|Description|
9494
|---|---|---|---|---|
95-
| **function.url** | string | N | - | Azure Function App endpoint URL for indirect uploads |
96-
| **function.method** | string | N | `POST` | HTTP method for function app requests |
95+
|`function.url`|string|N|-|Azure Function App endpoint URL for indirect uploads|
96+
|`function.method`|string|N|`POST`|HTTP method for function app requests|
9797

9898
### Debug
9999

100-
| Parameter | Type | Required | Default | Description |
100+
|Parameter|Type|Required|Default|Description|
101101
|---|---|---|---|---|
102-
| **description** | string | N | - | Optional description of target purpose |
103-
| **tag** | string | N | - | Target identifier tag for routing and filtering |
104-
| **status** | boolean | N | `true` | Enable or disable target processing |
102+
|`description`|string|N|-|Optional description of target purpose|
103+
|`tag`|string|N|-|Target identifier tag for routing and filtering|
104+
|`status`|boolean|N|`true`|Enable or disable target processing|
105105

106106
## Details
107107

108-
AWS S3 target provides enterprise-grade cloud storage integration with comprehensive file format support and AWS Security Lake compatibility.
108+
The target provides enterprise-grade cloud storage integration with comprehensive file format support and AWS Security Lake compatibility.
109109

110110
**Authentication Methods**: Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.
111111

112112
**File Formats**: Supports four output formats with distinct use cases:
113-
- **json**: Single JSON object per file (human-readable, suitable for small datasets)
114-
- **multijson**: Newline-delimited JSON objects (streaming format, efficient for large datasets)
115-
- **avro**: Schema-based binary serialization (compact, schema evolution support)
116-
- **parquet**: Columnar storage format (optimized for analytics, compression-friendly)
113+
114+
- `json`: Single JSON object per file (human-readable, suitable for small datasets)
115+
- `multijson`: Newline-delimited JSON objects (streaming format, efficient for large datasets)
116+
- `avro`: Schema-based binary serialization (compact, schema evolution support)
117+
- `parquet`: Columnar storage format (optimized for analytics, compression-friendly)
117118

118119
**Compression Options**: All formats support optional compression (`gzip`, `snappy`, `deflate`) to reduce storage costs and transfer times. Compression is applied before upload.
119120

0 commit comments

Comments
 (0)