-
Notifications
You must be signed in to change notification settings - Fork 734
Open
Labels
Description
Bug report
Attempting to leverage GPU resources in GCP batch.
Identical error to closed issue: Issue #575
Steps to reproduce the problem
Configuration Profile:
process {
label = 'mywf'
executor = 'google-batch'
container = env('GCP_CONTAINER')
errorStrategy = 'retry'
maxRetries = 0
withName: 'EMBED_SHARD' {
machineType = 'n1-standard-16'
cpus = 4
memory = '24 GB'
accelerator = [type: 'nvidia-tesla-t4', count: 1]
}
Program output
ERROR ~ Error executing process > 'EMBED_SHARD (1)'
Caused by:
Cannot cast object 'null' with class 'null' to class 'long'. Try 'java.lang.Long' instead
Sep-16 03:12:52.148 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf -c nextflow.demo.config -profile aif -resume -with-trace
Sep-16 03:12:52.453 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.04.6
Sep-16 03:12:52.534 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/.nextflow/plugins; core-plugins: nf-amazon@2.15.0,[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Sep-16 03:12:52.611 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Sep-16 03:12:52.613 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Sep-16 03:12:52.619 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Sep-16 03:12:52.647 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Sep-16 03:12:52.700 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /home/user/nextflow.config
Sep-16 03:12:52.708 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /home/user/nextflow.demo.config
Sep-16 03:12:52.714 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/user/nextflow.config
Sep-16 03:12:52.716 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/user/nextflow.demo.config
Sep-16 03:12:52.793 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/user/.nextflow/secrets/store.json
Sep-16 03:12:52.802 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@6999cd39] - activable => nextflow.secret.LocalSecretsProvider@6999cd39
Sep-16 03:12:52.817 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `aif`
Sep-16 03:12:54.248 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `aif`
Sep-16 03:12:54.473 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, aif]
Sep-16 03:12:54.571 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Sep-16 03:12:54.611 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [prickly_wright] DSL2 - revision: fdd5640116
Sep-16 03:12:54.616 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[nf-google@1.21.1]
Sep-16 03:12:54.617 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[nf-google@1.21.1]
Sep-16 03:12:54.622 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-google version: 1.21.1
Sep-16 03:12:54.651 [main] INFO org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Sep-16 03:12:54.652 [main] INFO org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Sep-16 03:12:54.701 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-google@1.21.1
Sep-16 03:12:54.804 [main] DEBUG nextflow.Session - Session UUID: ceed8985-d772-4bc9-a1ee-aa871325abb3
Sep-16 03:12:54.806 [main] DEBUG nextflow.Session - Run name: prickly_wright
Sep-16 03:12:54.807 [main] DEBUG nextflow.Session - Executor pool size: 4
Sep-16 03:12:55.153 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Sep-16 03:12:55.167 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=12; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Sep-16 03:12:55.208 [main] DEBUG nextflow.cli.CmdRun -
Version: 25.04.6 build 5954
Created: 01-07-2025 11:27 UTC
System: Linux 5.10.0-35-cloud-amd64
Runtime: Groovy 4.0.26 on OpenJDK 64-Bit Server VM 17.0.10+7
Encoding: UTF-8 (UTF-8)
Process: 1277687@davids-vm-20250706 [10.4.0.208]
CPUs: 4 - Mem: 14.6 GB (734.3 MB) - Swap: 0 (0)
Sep-16 03:12:55.348 [main] DEBUG nextflow.file.FileHelper - Unable to determine FS type (1): gs://shared-aif-bucket-8b96/jobs-tmpfiles/wf
...
Sep-16 03:13:12.024 [Actor Thread 12] INFO nextflow.processor.TaskProcessor - [be/6577fb] Cached process > NORMALIZE_CHANNELS (22)
Sep-16 03:13:12.149 [FileTransfer-1] DEBUG nextflow.file.FilePorter - Local cache found for foreign file /home/Sep-16 03:13:12.024 [Actor Thread 12] INFO nextflow.processor.TaskProcessor - [be/6577fb] Cached process > NORMALIZE_CHANNELS (22)
Sep-16 03:13:12.149 [FileTransfer-1] DEBUG nextflow.file.FilePorter - Local cache found for foreign file /home/user/config/channel_mapping.C-4.ome.json at gs://shared-aif-bucket-8b96/jobs-tmpfiles/wf/stage-ceed8985-d772-4bc9-a1ee-aa871325abb3/cd/2c888c31398318771d054f9943a28d/channel_mapping.C-4.ome.json
Sep-16 03:13:13.068 [Actor Thread 9] DEBUG nextflow.processor.TaskProcessor - Process `BUILD_STACK_AND_INDEX (1)` is unable to find [CloudStoragePath]: `/jobs-tmpfiles/wf/6e/10ba26317e6d022792b3fd010d6a4e/meta/oob_annotations.jsonl` (pattern: `meta/oob_annotations.jsonl`)
Sep-16 03:13:13.070 [Actor Thread 9] INFO nextflow.processor.TaskProcessor - [6e/10ba26] Cached process > BUILD_STACK_AND_INDEX (1)
Sep-16 03:13:13.197 [Actor Thread 9] DEBUG nextflow.processor.TaskProcessor - Process BUILD_STACK_AND_INDEX > Skipping output binding because one or more optional files are missing: fileoutparam<4:1>
Sep-16 03:13:13.768 [Actor Thread 7] INFO nextflow.processor.TaskProcessor - [2a/f2bc77] Cached process > SHARD_CELLS (1)
Sep-16 03:13:15.048 [Task submitter] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=EMBED_SHARD (1); work-dir=gs://shared-aif-bucket-8b96/jobs-tmpfiles/wf/db/e614349c6462870c7726d211ed7a6d
error [org.codehaus.groovy.runtime.typehandling.GroovyCastException]: Cannot cast object 'null' with class 'null' to class 'long'. Try 'java.lang.Long' instead
Sep-16 03:13:15.066 [Task submitter] ERROR nextflow.processor.TaskProcessor - Error executing process > 'EMBED_SHARD (1)'
Caused by:
Cannot cast object 'null' with class 'null' to class 'long'. Try 'java.lang.Long' instead
org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object 'null' with class 'null' to class 'long'. Try 'java.lang.Long' instead
at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.castToNumber(DefaultTypeTransformation.java:175)
at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.longUnbox(DefaultTypeTransformation.java:90)
at nextflow.cloud.google.batch.GoogleBatchTaskHandler.newSubmitRequest(GoogleBatchTaskHandler.groovy:355)
at nextflow.cloud.google.batch.GoogleBatchTaskHandler.submit(GoogleBatchTaskHandler.groovy:176)
at nextflow.processor.TaskPollingMonitor.submit(TaskPollingMonitor.groovy:225)
at nextflow.processor.TaskPollingMonitor.submitPendingTasks(TaskPollingMonitor.groovy:609)
at nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:420)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
at groovy.lang.Closure.call(Closure.java:433)
at groovy.lang.Closure.call(Closure.java:412)
at groovy.lang.Closure.run(Closure.java:505)
at java.base/java.lang.Thread.run(Thread.java:840)
Sep-16 03:13:15.080 [Task submitter] DEBUG nextflow.Session - Session aborted -- Cause: Cannot cast object 'null' with class 'null' to class 'long'. Try 'java.lang.Long' instead
Sep-16 03:13:15.115 [Task submitter] DEBUG nextflow.Session - The following nodes are still active:
[process] MERGE_SHARDS
status=ACTIVE
port 0: (queue) OPEN ; channel: -
port 1: (cntrl) - ; channel: $
Sep-16 03:13:15.125 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: google-batch) - terminating tasks monitor poll loop
Sep-16 03:13:15.125 [main] DEBUG nextflow.Session - Session await > all processes finished
Sep-16 03:13:15.125 [main] DEBUG nextflow.Session - Session await > all barriers passed
Sep-16 03:13:15.151 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=25; pendingCount=1; submittedCount=0; runningCount=-1; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=11m 43s;loadCpus=-4; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
Sep-16 03:13:15.151 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Sep-16 03:13:15.312 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Sep-16 03:13:15.352 [main] INFO org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Sep-16 03:13:15.352 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-google
Sep-16 03:13:15.358 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
Environment
- Nextflow version: [ 25.04.6 ]
- Java version: [ openjdk 17.0.10 2024-01-16 ]
- Operating system: [GCP + Docker = Ubuntu 24.04]
- Bash version: GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)