时间戳

首页 > 解决方案 > Cumulative sum with restart, optimization in R

r - Cumulative sum with restart, optimization in R

问题描述

I have 34 rasters (nrow: 17735, ncol: 11328, ncell: 200902080) with values 0 and 1, of 4Mb each. I want the cumulative sum of those values with zero reset.

I tried several alternatives based on: Cumulative sum that resets when 0 is encountered

library(microbenchmark)
library(compiler)
library(dplyr)
library(data.table)

x <- c(0,0,0,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0)

fun = function(x)
{ cs = cumsum(x)
  cs - cummax((x == 0) * cs)
}
funC <- cmpfun(fun)

microbenchmark(
funcioEx = fun(x),
funComEx = funC(x),
lapplyEx = unname(unlist(lapply(split(x,cumsum(c(0,diff(x) != 0))), cumsum))),
dataTaEx = data.table(x)[, whatiwant := cumsum(x), by = rleid(x==0L)],
reduceEx = Reduce(function(x, y) if (y == 0) 0 else x+y, x, accumulate=TRUE)
)

I would like to optimize this procedure for my data, because with the second option (funComEx, the fastest) it takes about 3 hours.

标签： roptimizationcumsummicrobenchmark

解决方案

Rcpp可能会有所帮助

library(Rcpp)
cppFunction(
    "IntegerVector foo(NumericVector vect){
    int N = vect.size();
    IntegerVector ans(N);
    ans[0] = vect[0];
    for (int i = 1; i < N; i++){
      if(vect[i] > 0){
        ans[i] = ans[i-1] + vect[i];
      } else {
        ans[i] = 0;
      }
    }
    return(ans);
  }"
)

set.seed(42)
x = sample(0:1, 1e4, TRUE)

identical(foo(x), fun(x))
#[1] TRUE

microbenchmark(
    funcioEx = fun(x),
    funComEx = funC(x),
    lapplyEx = unname(unlist(lapply(split(x,cumsum(c(0,diff(x) != 0))), cumsum))),
    dataTaEx = data.table(x)[, whatiwant := cumsum(x), by = rleid(x==0L)],
    reduceEx = Reduce(function(x, y) if (y == 0) 0 else x+y, x, accumulate=TRUE),
    foo_RCPP = foo(x)
)
#Unit: microseconds
#     expr       min         lq        mean     median         uq       max neval
# funcioEx    98.238   104.2495   118.81500   113.1935   121.1110   280.637   100
# funComEx    97.358   103.2230   113.81515   112.1670   118.1785   220.522   100
# lapplyEx 17810.638 18888.9055 20130.20765 19399.7415 20641.0550 28073.981   100
# dataTaEx  3435.387  3832.0025  4468.77932  4023.6395  4347.3840 17053.181   100
# reduceEx  7472.515  8174.4020  9614.23122  8634.7985 10177.1305 15719.788   100
# foo_RCPP    52.491    62.6085    80.65777    66.5670    72.4320  1102.315   100

推荐阅读