Skip to content

Commit 38929b4

Browse files
andreacioniandreacioni
committed
multiple updates after PR review:
- Builders moved to a dedicated package - project README.adoc updated; - root README.md updated: add reference to new module; - use org.springframework.util.unit.DataSize to express data sizes; - removed apache-client exclusion in pom.xml to allow user to not specify an actual implementation in their dependencies - updated spring-javaformat-checkstyle to match the latest version - update gitignore to match the one of Spring Batch Notion - Add Maven Wrapper executables - Add Github Action workflow - Removed maven-wrapper.jar - Code cleanup Signed-off-by: Andrea Cioni <[email protected]>
1 parent 3fe11bb commit 38929b4

File tree

20 files changed

+641
-490
lines changed

20 files changed

+641
-490
lines changed

‎.github/workflows/spring-batch-s3.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
name: Spring Batch S3
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- 'spring-batch-s3/**'
7+
push:
8+
paths:
9+
- 'spring-batch-s3/**'
10+
11+
env:
12+
MAVEN_ARGS: -B -V -ntp -e -Djansi.passthrough=true -Dstyle.color=always
13+
14+
jobs:
15+
build:
16+
runs-on: ubuntu-latest
17+
steps:
18+
- uses: actions/checkout@v4
19+
- name: Set up Java
20+
uses: actions/setup-java@v4
21+
with:
22+
distribution: 'temurin'
23+
java-version: '21'
24+
cache: 'maven'
25+
- name: Build with Maven
26+
run: ./mvnw $MAVEN_ARGS verify javadoc:javadoc
27+
working-directory: spring-batch-s3

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ This project is part of the [Spring organization][] on GitHub.
1414
| [`spring-batch-geode`](spring-batch-geode) | Support for [Apache Geode] | TBA | [![Maven Central](https://img.shields.io/maven-central/v/org.springframework.batch.extensions/spring-batch-geode?label)](https://central.sonatype.com/artifact/org.springframework.batch.extensions/spring-batch-geode) | [![Spring Batch Geode](https://github.com/spring-projects/spring-batch-extensions/actions/workflows/spring-batch-geode.yml/badge.svg)](https://github.com/spring-projects/spring-batch-extensions/actions/workflows/spring-batch-geode.yml?query=branch%3Amain) |
1515
| [`spring-batch-neo4j`](spring-batch-neo4j) | Support for [Neo4j] | [@michael-simons](https://github.com/michael-simons) | [![Maven Central](https://img.shields.io/maven-central/v/org.springframework.batch.extensions/spring-batch-neo4j?label)](https://central.sonatype.com/artifact/org.springframework.batch.extensions/spring-batch-neo4j) | [![Spring Batch Neo4j](https://github.com/spring-projects/spring-batch-extensions/actions/workflows/spring-batch-neo4j.yml/badge.svg)](https://github.com/spring-projects/spring-batch-extensions/actions/workflows/spring-batch-neo4j.yml?query=branch%3Amain) |
1616
| [`spring-batch-notion`](spring-batch-notion) | Support for [Notion] | [@scordio](https://github.com/scordio) | [![Maven Central](https://img.shields.io/maven-central/v/org.springframework.batch.extensions/spring-batch-notion?label)](https://central.sonatype.com/artifact/org.springframework.batch.extensions/spring-batch-notion) | [![Spring Batch Notion](https://github.com/spring-projects/spring-batch-extensions/actions/workflows/spring-batch-notion.yml/badge.svg?branch=main)](https://github.com/spring-projects/spring-batch-extensions/actions/workflows/spring-batch-notion.yml?query=branch%3Amain) |
17+
| [`spring-batch-s3`](spring-batch-s3) | Support for [Amazon S3] | [@andreacioni](https://github.com/andreacioni) | [![Maven Central](https://img.shields.io/maven-central/v/org.springframework.batch.extensions/spring-batch-s3?label)](https://central.sonatype.com/artifact/org.springframework.batch.extensions/spring-batch-s3) | [![Spring Batch S3](https://github.com/spring-projects/spring-batch-extensions/actions/workflows/spring-batch-s3.yml/badge.svg?branch=main)](https://github.com/spring-projects/spring-batch-extensions/actions/workflows/spring-batch-s3.yml?query=branch%3Amain) |
1718

1819
## Getting support
1920

‎spring-batch-s3/.editorconfig‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
root = true
22

3-
[*.{adoc,bat,groovy,html,java,js,jsp,kt,kts,md,properties,py,rb,sh,sql,svg,txt,xml,xsd}]
3+
# Unix-style newlines with a newline ending every file
4+
[*]
45
charset = utf-8
5-
6-
[*.{groovy,java,kt,kts,xml,xsd}]
7-
indent_style = tab
8-
indent_size = 4
9-
continuation_indent_size = 8
106
end_of_line = lf
7+
insert_final_newline = true
8+
9+
# Maven POM code convention
10+
[pom.xml]
11+
indent_size = 2
12+
indent_style = space
13+
max_line_length = 205

‎spring-batch-s3/.gitignore‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,32 @@
1+
### Maven ###
2+
target/
3+
.mvn/wrapper/maven-wrapper.jar
14
.flattened-pom.xml
5+
6+
### STS ###
7+
.apt_generated
8+
.classpath
9+
.factorypath
10+
.project
11+
.settings
12+
.springBeans
13+
.sts4-cache
14+
15+
### IntelliJ IDEA ###
16+
.idea
17+
*.iws
18+
*.iml
19+
*.ipr
20+
21+
### NetBeans ###
22+
/nbproject/private/
23+
/nbbuild/
24+
/dist/
25+
/nbdist/
26+
/.nb-gradle/
27+
build/
28+
!**/src/main/**/build/
29+
!**/src/test/**/build/
30+
31+
### VS Code ###
32+
.vscode/

‎spring-batch-s3/.mvn/maven.config‎

Lines changed: 0 additions & 2 deletions
This file was deleted.

‎spring-batch-s3/.mvn/wrapper/maven-wrapper.jar‎

-48.4 KB
Binary file not shown.

‎spring-batch-s3/.mvn/wrapper/maven-wrapper.properties‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,19 @@
1-
distributionUrl=https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.6.3/apache-maven-3.6.3-bin.zip
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
wrapperVersion=3.3.2
18+
distributionType=only-script
19+
distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.8/apache-maven-3.9.8-bin.zip

‎spring-batch-s3/README.adoc‎

Lines changed: 6 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,7 @@
33
:icons: font
44
:source-highlighter: highlightjs
55

6-
Spring Batch extension for Amazon S3 (even other S3 compatible may be supported) which contains `S3ItemReader` and `S3ItemWriter` implementations
7-
for reading from and writing to S3 buckets, including support for multipart uploads.
6+
https://spring.io/projects/spring-batch[Spring Batch] extension for https://aws.amazon.com/it/s3/[Amazon S3] which contains `S3ItemReader` and `S3ItemWriter` implementations for reading from and writing to S3 buckets, including support for multipart uploads. Other S3-compatible technologies (like as Google Cloud Storage, MinIO, etc.) may be supported, but they are not guaranteed to work.
87

98
*Note*: these writers are based on the *AWS SDK V2*.
109

@@ -21,19 +20,13 @@ To use the `spring-batch-s3` extension, you need to add the following dependency
2120
<artifactId>spring-batch-s3</artifactId>
2221
<version>${spring-batch-extensions.version}</version>
2322
</dependency>
24-
<dependency>
25-
<groupId>software.amazon.awssdk</groupId>
26-
<artifactId>apache-client</artifactId>
27-
<version>${aws.sdk.version}</version>
28-
</dependency>
2923
----
3024

3125
=== Gradle
3226

3327
[source,groovy]
3428
----
3529
implementation 'org.springframework.batch.extensions:spring-batch-s3:${springBatchExtensionsVersion}'
36-
implementation 'software.amazon.awssdk:apache-client:${awsSdkVersion}'
3730
----
3831

3932
== Known limitations
@@ -57,20 +50,7 @@ More details in the JavaDocs of the classes.
5750

5851
=== Configuration of the `S3Client`
5952

60-
To use the `S3ItemReader` and `S3ItemWriter`, you need to configure the AWS S3 client.
61-
This can be done using Java configuration or XML configuration.
62-
63-
So far only this synchronous client is supported, you can't use the `S3AsyncClient` with these components.
64-
65-
==== Java Config
66-
67-
[source,java]
68-
----
69-
@Bean
70-
public S3Client s3Client() {
71-
return S3Client.builder().build();
72-
}
73-
----
53+
To use the `S3ItemReader` and `S3ItemWriter`, you need to configure the AWS S3 client by providing a `S3Client`. Checkout the https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/http-configuration-apache.html[AWS SDK for Java] documentation for more details on how to configure this.
7454

7555
=== Configure `S3Serializer`
7656

@@ -118,7 +98,7 @@ ItemReader<String> downloadItemReader() throws Exception {
11898
.bucketName("bucket_name")
11999
.objectKey("object_key")
120100
.deserializer(s3Deserializer())
121-
.bufferSize(1024 * 1024) // Default 128 Bytes
101+
.bufferSize((int) DataSize.ofMegabytes(1).toBytes()) // Default 128 Bytes
122102
.build();
123103
}
124104
----
@@ -163,20 +143,15 @@ ItemWriter<String> uploadItemWriter() throws IOException {
163143
.bucketName("bucket_name")
164144
.objectKey("object_key")
165145
.multipartUpload(true) // Default is false
166-
.partSize(10 * 1024 * 1024) // Default is 5 MB
146+
.partSize((int) DataSize.ofMegabytes(10).toBytes()) // Default is 5 MB
167147
.contentType("text/csv") // Default is application/octet-stream
168148
.serializer(s3Serializer())
169149
.build();
170150
}
171151
----
172152

173153
There are several additional options you can set for the `S3ItemWriter`:
154+
174155
* `multipartUpload`: If set to `true`, the writer will use multipart upload for large files. The default is `false`.
175156
* `partSize`: The size of each part in a multipart upload. The default is 5 MB.
176-
* `contentType`: The content type of the uploaded file. The default is `application/octet-stream`.
177-
178-
== Links
179-
180-
* https://github.com/spring-projects/spring-batch-extensions
181-
* https://spring.io/projects/spring-batch
182-
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html
157+
* `contentType`: The content type of the uploaded file. The default is `application/octet-stream`.

0 commit comments

Comments
 (0)