Update quickpool (#56)

* pull new quickpool

* adapt library code

* update benchmarks

* save plots in benchmarks folder

* log scale on x axis

* use global pool

* update quickpool

* adapt thread pool

* prettify benchmarks

* add free push/async/wait

* better benchmarks

* actually benchmark quickpool

* change plot size everywhere

* affinity

* better benchmarks

* update benchmarks

* backoff algorithm

* less iter for imbalanced

* update benchmarks

* fix nested parallel for

* inline free functions

* don't move into parallel for

* bump version

* update docs

* update parallel for doc/args

* fix single threaded loops

* backwards compatible batched loops

* update quickpool

* update docs

* try to get codecov running

* try on linux

* yeah that won't work

* keep distr as it was

* update quickpool (resizing)

* allow parallel loops to limit threads

* reduce number of tests

* finalize

* safe exception tests

* do-while in wait()

* safer resize

* update NEWS and README

Author: tnagler

.Rbuildignore

@@ -15,3 +15,4 @@ revdep/
 .vscode/
 new-benchmarks.R
 bench*
+revdep/

.github/workflows/R-CMD-check.yaml

@@ -80,7 +80,7 @@ jobs:
           _R_CHECK_CRAN_INCOMING_: false
         run: rcmdcheck::rcmdcheck(args = c("--no-manual", "--as-cran"), error_on = "warning", check_dir = "check")
         shell: Rscript {0}
-        # run: cd .. && R CMD build RcppThread && R CMD check RcppThread_1.1.0.tar.gz 
+        # run: cd .. && R CMD build RcppThread && R CMD check RcppThread_1.1.0.tar.gz
         # shell: bash
 
       - name: Show testthat output
@@ -94,9 +94,3 @@ jobs:
         with:
           name: ${{ runner.os }}-r${{ matrix.config.r }}-results
           path: check
-
-      # - name: Test coverage
-      #   if: matrix.config.os == 'macOS-latest' && matrix.config.r == 'release'
-      #   run: |
-      #     Rscript -e 'remotes::install_github("r-lib/covr@gh-actions")'
-      #     Rscript -e 'covr::codecov(token = "${{secrets.CODECOV_TOKEN}}")'

DESCRIPTION

@@ -1,6 +1,6 @@
 Package: RcppThread
 Title: R-Friendly Threading in C++
-Version: 1.1.0
+Version: 2.0.0
 Authors@R: c(
     person("Thomas", "Nagler",, "mail@tnagler.com", role = c("aut", "cre"),
            comment = c(ORCID = "0000-0003-1855-0046"))
@@ -13,7 +13,7 @@ Encoding: UTF-8
 SystemRequirements: C++11
 URL: https://github.com/tnagler/RcppThread
 BugReports: https://github.com/tnagler/RcppThread/issues
-RoxygenNote: 7.1.1
+RoxygenNote: 7.1.2
 Suggests: 
     testthat,
     R.rsp,

NEWS.md

@@ -1,4 +1,4 @@
-# RcppThread 1.1.0
+# RcppThread 2.0.0
 
 * Add R function `detectCores()` (#48).
 
@@ -10,6 +10,11 @@
 * Free-standing `parallelFor()` and `parallelForEach()` functions now dispatch 
   to a global thread pool that persists for the entire session. This 
   significantly speeds up programs that repeatedly call these functions. (#54)
+  
+* New free-standing `push()`, `pushReturn()`/`async()`, and `wait()`, mirroring
+  functionality from `ThreadPool`. (#56)
+
+* Option to resize a thread pool (#56).
 
 
 # RcppThread 1.0.0

README.md

@@ -27,13 +27,15 @@ For a detailed description of its functionality and examples, see the associated
 [JSS paper](https://doi.org/10.18637/jss.v097.c01)
 or the [API documentation](https://tnagler.github.io/RcppThread/).
 
-Since then, the following features have been added:
+Since then, the following **new features** have been added:
 
-- Free-standing `parallelFor()` and `parallelForEach()` functions now dispatch 
+- Free-standing functions like `parallelFor()` now dispatch 
   to a global thread pool that persists for the entire session. This 
   significantly speeds up programs that repeatedly call these functions.
 
-- Faster runtimes due to a work stealing task queue with lock-free pops (from [quickpool](https://github.com/tnagler/quickpool)).
+- Faster runtimes due to lock-free work stealing queue and loops (from [quickpool](https://github.com/tnagler/quickpool)).
+
+- Option to resize a thread pool.
 
 - An R function `RcppThread::detectCores()` to determine the number of (logical)
   cores on your machine.

benchmarks/benchmarks.R

@@ -0,0 +1,85 @@
+library("tidyverse")
+library("Rcpp")
+library("RcppParallel")
+library("RcppThread")
+library("ggthemes")
+
+
+Rcpp::sourceCpp(here::here("benchmarks/benchmarks.cpp"))
+
+
+wait_for <- 5
+
+plot_df <- function(df, title = NULL) {
+  p <- df %>%
+    pivot_longer(-n, "call") %>%
+    ggplot(aes(n, value, color = call)) +
+    geom_line(size = 0.6) +
+    expand_limits(y = 0) +
+    labs(color = "", linetype = "") +
+    naglr::theme_naglr(plot_title_size = 12) +
+    theme(legend.margin = margin(1, 1, 1, 1)) +
+    theme(legend.position = "bottom") +
+    scale_x_log10() +
+    ylab("speedup") +
+    labs(title = title)
+  print(p)
+}
+
+
+ns <- 10^(2:6)
+res <- benchEmpty(rev(ns), wait_for)
+df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+plot_df(df, "empty jobs")
+ggsave(here::here("benchmarks/benchEmptyThread.pdf"), width = 7.5, height = 3)
+
+
+ns <- 10^(2:6)
+res <- benchSqrt(rev(ns), wait_for)
+df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+plot_df(df, "1000x sqrt")
+ggsave(here::here("benchmarks/benchSqrt.pdf"), width = 7.5, height = 3)
+
+
+# ns <- 10^(2:6)
+# res <- benchSqrtWrite(rev(ns), wait_for)
+# df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+# plot_df(df, "1000x sqrt modify inplace")
+# ggsave(here::here("benchmarks/benchSqrtWrite.pdf"), width = 7.5, height = 3)
+
+ns <- 10^(2:4)
+res <- benchSqrtImbalanced(rev(ns), wait_for)
+df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+plot_df(df, "n times sqrt (imbalanced)")
+ggsave(here::here("benchmarks/benchSqrtImbalanced.pdf"), width = 7.5, height = 3)
+
+# ns <- 10^(2:4)
+# res <- benchSqrtWriteImbalanced(rev(ns), wait_for)
+# df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+# plot_df(df, "n times sqrt modify inplace (imbalanced)")
+# ggsave(here::here("benchmarks/benchSqrtWriteImbalanced.pdf"), width = 7.5, height = 3)
+
+ns <- 10^(2:5)
+res <- benchKDE(rev(ns), 100, wait_for)
+df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+plot_df(df, "kernel density d = 10")
+ggsave(here::here("benchmarks/benchKDE-10.pdf"), width = 7.5, height = 3)
+
+
+ns <- 10^(2:5)
+res <- benchKDE(rev(ns), 100, wait_for)
+df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+plot_df(df, "kernel density d = 100")
+ggsave(here::here("benchmarks/benchKDE-100.pdf"), width = 7.5, height = 3)
+
+ns <- 10^(2:4)
+res <- benchKendall(rev(ns), 10, wait_for)
+df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+plot_df(df, "Kendall matrix (unbalanced) d = 10")
+ggsave(here::here("benchmarks/benchKendall-10.pdf"), width = 7.5, height = 3)
+
+ns <- 10^(2:4)
+res <- benchKendall(rev(ns), 100, wait_for)
+df <- cbind(data.frame(n = rev(ns)), as.data.frame(res[, -1]))
+plot_df(df, "Kendall matrix (unbalanced) d = 100")
+ggsave(here::here("benchmarks/benchKendall-100.pdf"), width = 7.5, height = 3)