System VMs fail to start – “Unable to create a deployment for VM instance

[yann1ck-oder-so]

Hi everyone,

I’m running a management server and currently have two KVM hosts, both on Ubuntu.
The basic configuration is complete, and both hosts are successfully added to the cluster.

The problem:
My System VMs (Console Proxy & Secondary Storage VM) won’t start.
In the CloudStack UI, the message just says “Could not be started.”

When checking the log (/var/log/cloudstack/management/management-server.log), I get the following error:

com.cloud.exception.InsufficientServerCapacityException:
Unable to create a deployment for VM instance {"id":852,"instanceName":"s-852-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"447f0e3d-dae7-4f06-885c-fe262274c46d"}
Scope=interface com.cloud.dc.DataCenter; id=1

Meaning (according to docs):
CloudStack could not find a host with sufficient resources (CPU, RAM, or storage) to deploy the Secondary Storage VM.

However, both hosts are green (Up) in the UI and appear to be available.
I’ve checked the following:

Hosts show “Up” status

Plenty of free CPU and RAM

Networking appears correct

Storage is connected and accessible

Question:
What could cause CloudStack to fail finding a suitable host, even though both are active?
Are there any additional requirements (e.g., network setup, storage tags, or system offering settings) I might have missed?

Thanks in advance!

[rajujith]

@yann1ck-oder-so 'InsufficientServerCapacityException' is a generic error; there must be a specific error in the logs preceding this error.

[yann1ck-oder-so]

Is this more usefull?

2025-10-24 06:51:38,464 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-24:[ctx-11397d36, job-236/job-2298]) (logid:9ca0f174) Publish async job-2298 complete on message bus
2025-10-24 06:51:38,464 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-24:[ctx-11397d36, job-236/job-2298]) (logid:9ca0f174) Wake up jobs related to job-2298
2025-10-24 06:51:38,464 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-24:[ctx-11397d36, job-236/job-2298]) (logid:9ca0f174) Update db status for job-2298
2025-10-24 06:51:38,465 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-24:[ctx-11397d36, job-236/job-2298]) (logid:9ca0f174) Wake up jobs joined with job-2298 and disjoin all subjobs created from job- 2298
2025-10-24 06:51:38,473 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (Work-Job-Executor-24:[ctx-11397d36, job-236/job-2298]) (logid:9ca0f174) Done executing com.cloud.vm.VmWorkStart for job-2298
2025-10-24 06:51:38,473 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-24:[ctx-11397d36, job-236/job-2298]) (logid:9ca0f174) Remove job-2298 from job monitoring
2025-10-24 06:51:38,481 WARN  [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-78241083]) (logid:8d474bb9) Unable to start proxy [874] due to [Unable to create a deployment for VM instance {"id":874,"instanceName":"v-874-VM","state":"Starting","type":"ConsoleProxy","uuid":"5856057e-97e7-41f9-90f7-e524d6a661f9"}]. com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":874,"instanceName":"v-874-VM","state":"Starting","type":"ConsoleProxy","uuid":"5856057e-97e7-41f9-90f7-e524d6a661f9"}Scope=interface com.cloud.dc.DataCenter; id=1
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1451)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

[weizhouapache]

the full log of job-2298 would be better

[rajujith]

https://docs.cloudstack.apache.org/en/latest/adminguide/troubleshooting.html

[yann1ck-oder-so]

I have no output, if I execute this command: grep "job-2298)" management-server.log

[rajujith]

@yann1ck-oder-so make sure you grep the log file for 2025-10-24