|
6 | 6 | -- |
7 | 7 | This section goes into more detail about Spring Cloud Task's integration with Spring |
8 | 8 | Batch. Tracking the association between a job execution and the task in which it was |
9 | | -executed as well as remote partitioning through Spring Cloud Deployer are covered in |
10 | | -this section. |
| 9 | +executed. |
11 | 10 |
|
12 | 11 | -- |
13 | 12 |
|
@@ -53,143 +52,6 @@ NOTE: You can find a sample batch application in the samples module of the Sprin |
53 | 52 | Task Project, |
54 | 53 | https://github.com/spring-cloud/spring-cloud-task/tree/master/spring-cloud-task-samples/batch-job[here]. |
55 | 54 |
|
56 | | - |
57 | | -[[batch-partitioning]] |
58 | | -== Remote Partitioning |
59 | | - |
60 | | -Spring Cloud Deployer provides facilities for launching Spring Boot-based applications on |
61 | | -most cloud infrastructures. The `DeployerPartitionHandler` and |
62 | | -`DeployerStepExecutionHandler` delegate the launching of worker step executions to Spring |
63 | | -Cloud Deployer. |
64 | | - |
65 | | -To configure the `DeployerStepExecutionHandler`, you must provide a `Resource` |
66 | | -representing the Spring Boot Uber-jar to be executed, a `TaskLauncherHandler`, and a |
67 | | -`JobExplorer`. You can configure any environment properties as well as the max number of |
68 | | -workers to be executing at once, the interval to poll for the results (defaults to 10 |
69 | | -seconds), and a timeout (defaults to -1 or no timeout). The following example shows how |
70 | | -configuring this `PartitionHandler` might look: |
71 | | - |
72 | | -NOTE: This feature is now end-of-life and will be removed in a future release. |
73 | | - |
74 | | -[source,java] |
75 | | ----- |
76 | | -@Bean |
77 | | -public PartitionHandler partitionHandler(TaskLauncher taskLauncher, |
78 | | - JobExplorer jobExplorer) throws Exception { |
79 | | -
|
80 | | - MavenProperties mavenProperties = new MavenProperties(); |
81 | | - mavenProperties.setRemoteRepositories(new HashMap<>(Collections.singletonMap("springRepo", |
82 | | - new MavenProperties.RemoteRepository(repository)))); |
83 | | -
|
84 | | - Resource resource = |
85 | | - MavenResource.parse(String.format("%s:%s:%s", |
86 | | - "io.spring.cloud", |
87 | | - "partitioned-batch-job", |
88 | | - "1.1.0.RELEASE"), mavenProperties); |
89 | | -
|
90 | | - DeployerPartitionHandler partitionHandler = |
91 | | - new DeployerPartitionHandler(taskLauncher, jobExplorer, resource, "workerStep"); |
92 | | -
|
93 | | - List<String> commandLineArgs = new ArrayList<>(3); |
94 | | - commandLineArgs.add("--spring.profiles.active=worker"); |
95 | | - commandLineArgs.add("--spring.cloud.task.initialize.enable=false"); |
96 | | - commandLineArgs.add("--spring.batch.initializer.enabled=false"); |
97 | | -
|
98 | | - partitionHandler.setCommandLineArgsProvider( |
99 | | - new PassThroughCommandLineArgsProvider(commandLineArgs)); |
100 | | - partitionHandler.setEnvironmentVariablesProvider(new NoOpEnvironmentVariablesProvider()); |
101 | | - partitionHandler.setMaxWorkers(2); |
102 | | - partitionHandler.setApplicationName("PartitionedBatchJobTask"); |
103 | | -
|
104 | | - return partitionHandler; |
105 | | -} |
106 | | ----- |
107 | | - |
108 | | -NOTE: When passing environment variables to partitions, each partition may |
109 | | -be on a different machine with different environment settings. |
110 | | -Consequently, you should pass only those environment variables that are required. |
111 | | - |
112 | | -Notice in the example above that we have set the maximum number of workers to 2. |
113 | | -Setting the maximum of workers establishes the maximum number of |
114 | | -partitions that should be running at one time. |
115 | | - |
116 | | -The `Resource` to be executed is expected to be a Spring Boot Uber-jar with a |
117 | | -`DeployerStepExecutionHandler` configured as a `CommandLineRunner` in the current context. |
118 | | -The repository enumerated in the preceding example should be the remote repository in |
119 | | -which the Spring Boot Uber-jar is located. Both the manager and worker are expected to have visibility |
120 | | -into the same data store being used as the job repository and task repository. Once the |
121 | | -underlying infrastructure has bootstrapped the Spring Boot jar and Spring Boot has |
122 | | -launched the `DeployerStepExecutionHandler`, the step handler executes the requested |
123 | | -`Step`. The following example shows how to configure the `DeployerStepExecutionHandler`: |
124 | | - |
125 | | -[source,java] |
126 | | ----- |
127 | | -@Bean |
128 | | -public DeployerStepExecutionHandler stepExecutionHandler(JobExplorer jobExplorer) { |
129 | | - DeployerStepExecutionHandler handler = |
130 | | - new DeployerStepExecutionHandler(this.context, jobExplorer, this.jobRepository); |
131 | | -
|
132 | | - return handler; |
133 | | -} |
134 | | ----- |
135 | | - |
136 | | -NOTE: You can find a sample remote partition application in the samples module of the |
137 | | -Spring Cloud Task project, |
138 | | -https://github.com/spring-cloud/spring-cloud-task/tree/master/spring-cloud-task-samples/partitioned-batch-job[here]. |
139 | | - |
140 | | -[[asynchronously-launch-remote-batch-partitions]] |
141 | | -=== Asynchronously launch remote batch partitions |
142 | | - |
143 | | -By default batch partitions are launched sequentially. However, in some cases this may affect performance as each launch will block until the resource (For example: provisioning a pod in Kubernetes) is provisioned. |
144 | | -In these cases you can provide a `ThreadPoolTaskExecutor` to the `DeployerPartitionHandler`. This will launch the remote batch partitions based on the configuration of the `ThreadPoolTaskExecutor`. |
145 | | -For example: |
146 | | - |
147 | | -[source,java] |
148 | | ----- |
149 | | - @Bean |
150 | | - public ThreadPoolTaskExecutor threadPoolTaskExecutor() { |
151 | | - ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); |
152 | | - executor.setCorePoolSize(4); |
153 | | - executor.setThreadNamePrefix("default_task_executor_thread"); |
154 | | - executor.setWaitForTasksToCompleteOnShutdown(true); |
155 | | - executor.initialize(); |
156 | | - return executor; |
157 | | - } |
158 | | -
|
159 | | - @Bean |
160 | | - public PartitionHandler partitionHandler(TaskLauncher taskLauncher, JobExplorer jobExplorer, |
161 | | - TaskRepository taskRepository, ThreadPoolTaskExecutor executor) throws Exception { |
162 | | - Resource resource = this.resourceLoader |
163 | | - .getResource("maven://io.spring.cloud:partitioned-batch-job:2.2.0.BUILD-SNAPSHOT"); |
164 | | -
|
165 | | - DeployerPartitionHandler partitionHandler = |
166 | | - new DeployerPartitionHandler(taskLauncher, jobExplorer, resource, |
167 | | - "workerStep", taskRepository, executor); |
168 | | - ... |
169 | | - } |
170 | | ----- |
171 | | - |
172 | | -NOTE: We need to close the context since the use of `ThreadPoolTaskExecutor` leaves a thread active thus the app will not terminate. To close the application appropriately, we will need to set `spring.cloud.task.closecontextEnabled` property to `true`. |
173 | | - |
174 | | - |
175 | | -[[notes-on-developing-a-batch-partitioned-application-for-the-kubernetes-platform]] |
176 | | -=== Notes on Developing a Batch-partitioned application for the Kubernetes Platform |
177 | | - |
178 | | -* When deploying partitioned apps on the Kubernetes platform, you must use the following |
179 | | -dependency for the Spring Cloud Kubernetes Deployer: |
180 | | -+ |
181 | | -[source,xml] |
182 | | ----- |
183 | | -<dependency> |
184 | | - <groupId>org.springframework.cloud</groupId> |
185 | | - <artifactId>spring-cloud-starter-deployer-kubernetes</artifactId> |
186 | | -</dependency> |
187 | | ----- |
188 | | -* The application name for the task application and its partitions need to follow |
189 | | -the following regex pattern: `[a-z0-9]([-a-z0-9]*[a-z0-9])`. |
190 | | -Otherwise, an exception is thrown. |
191 | | - |
192 | | - |
193 | 55 | [[batch-informational-messages]] |
194 | 56 | == Batch Informational Messages |
195 | 57 |
|
|
0 commit comments