Cubestore memory usage

[jtcies]

We currently use self-hosted Cubestore on our Kubernetes cluster. We're trying to reduce Kubernetes costs. Typically our Cubestore workers memory requirements are relatively low, but they spike each day when refreshing our pre-aggregations.

Are the any suggestions for parameters to tune that might help keep our memory usage low without worrying about OOM on the pods during these refreshes? We've already reduced CUBEJS_REFRESH_WORKER_CONCURRENCY but wondering if there are other recommendations here. Increased latency/slower refreshes are ok for this trade-off.

[igorlukanin]

@jtcies What is your current Cube Store cluster configuration? How many worker nodes? How much memory provisioned to each? Also, how many partitions do you have to rebuild every day? What are their sizes?

It's hard to be giving recommendations without seeing the full picture. One advice might be to reduce partition sizes (using partition_granularity) so that less data is processed during each partition build. Another (rather weird) advice might be to build pre-aggs with a certain number of Cube Store workers in a cluster, then spin it down, then run one with a reduced number of workers during "business hours" (if you feel like you need less workers for serving queries than for building partitions).