matlab - 在 MATLAB 中运行 PARFOR 循环时内存不足

Question

我和一些同事在集群上运行以下函数时内存不足，其中 F_scattered 是使用scatterInterpolant 生成的插值集合。任何人都可以建议我们可以采取的步骤来减少 parfor 循环的内存密集度吗？我担心编写的代码可能会将所有 F_scattered 发送给每个工作人员，但我不确定如何检查是否发生这种情况。如果有帮助，此消息底部会提供更多上下文。预先感谢您的帮助。

函数 [V1q,V2q] = interpolate(F_gridded,X_gridded,F_scattered,X_scattered)

w = F_gridded(X_gridded);
k = fix(w);
w = w-k;
[n,m] = size(F_scattered);
assert(n==2)
F1 = cell(m,1);
F2 = cell(m,1);

parfor j=1:m
    kj = k==j;
    X = X_scattered(kj,:);
    if j<m
        W = w(kj);
        F1{j} = (1-W).*F_scattered{1,j}(X)+W.*F_scattered{1,j+1}(X);
        F2{j} = (1-W).*F_scattered{2,j}(X)+W.*F_scattered{2,j+1}(X);
    else
        F1{j} = F_scattered{1,j}(X);
        F2{j} = F_scattered{2,j}(X);
    end
end

Vq = nan(size(w));

for j=1:m
    kj = k==j;
    V1q(kj) = F1{j};
    V2q(kj) = F2{j};
end

结尾

附加上下文如下。我和一些同事正在尝试使用内生网格方法在七维网格上进行搭配。相关详情如下：

• 我们的维度是(b1,dstate,omega,u,b2,b3,eps_pi)。b1、b2、b3 和 eps 是连续的。dstate、omega 和 u 是离散的。eps_pi 由于内生网格算法而分散，但其他变量的网格是使用 ndgrid 从向量构建的。

• 因为scatteredInterpolant 最多接受三个维度的参数，所以我们通过在(b1,dstate,omega,u) 中“切片”我们的搭配网格来构造我们的插值，然后在与每个切片相关联的(b2,b3,eps_pi) 值上运行scatterInterpolant .

• 由于 b1 是 (b1,dstate,omega,u) 中唯一的连续变量，因此我们通过 (i) 进行插值，找到包含 (b1,dstate,omega,u) 的查询值的两个切片，然后 (ii)在 (b2,b3,eps_pi) 的查询值上运行与这些切片相关联的分散插值，然后 (iii) 对输出进行适当加权平均。

以下是相对最小形式的代码的关键部分：

%%% 设置网格向量

    b1_t_grid = linspace(0,1,8); 
    dstate_t_grid  = [-1:12];
    omega_t_grid = [-2,2];
    u_t_grid = [0,1];
    b2_t_grid = linspace(0,1,5); 
    b3_t_grid = linspace(0,1,5);
    quasidiff_t_grid = linspace(-9*0.05*4*2,9*0.05*4*2,8);   % ``target’’ variable for the endogenous grid method

%%% 将切片维度与其他维度分开

    gridded_dims_TSnow        = [numel(b1_t_grid),numel(dstate_t_grid),numel(omega_t_grid),numel(u_t_grid)];
n_gridded_substates_TSnow = prod(gridded_dims_TSnow);

scattered_dims_TSnow        = [numel(b2_t_grid),numel(b3_t_grid),numel(quasidiff_t_grid)];
n_scattered_substates_TSnow = prod(scattered_dims_TSnow);

[b1_ts_TSnow,dstate_ts_TSnow,omega_ts_TSnow,u_ts_TSnow,b2_ts_TSnow,b3_ts_TSnow,quasidiff_ts_TSnow] = ndgrid(b1_t_grid,dstate_t_grid,omega_t_grid,u_t_grid,b2_t_grid,b3_t_grid,quasidiff_t_grid);

b1_ts_TSnow        = reshape(b1_ts_TSnow       ,n_gridded_substates_TSnow,n_scattered_substates_TSnow);
dstate_ts_TSnow    = reshape(dstate_ts_TSnow   ,n_gridded_substates_TSnow,n_scattered_substates_TSnow);
omega_ts_TSnow     = reshape(omega_ts_TSnow    ,n_gridded_substates_TSnow,n_scattered_substates_TSnow);
u_ts_TSnow         = reshape(u_ts_TSnow        ,n_gridded_substates_TSnow,n_scattered_substates_TSnow);
b2_ts_TSnow        = reshape(b2_ts_TSnow       ,n_gridded_substates_TSnow,n_scattered_substates_TSnow);
b3_ts_TSnow        = reshape(b3_ts_TSnow       ,n_gridded_substates_TSnow,n_scattered_substates_TSnow);
quasidiff_ts_TSnow = reshape(quasidiff_ts_TSnow,n_gridded_substates_TSnow,n_scattered_substates_TSnow);

grid_size_TSnow = numel(b1_ts_TSnow);
grid_dims_TSnow =  size(b1_ts_TSnow); 

%%% 我们最终将应用内生网格算法的维度的初始猜测：

Gamma = @(yhat,omega) 0.05*(yhat-omega) + 0.10*(max(0,yhat-omega)).^2;
CP = @(omega,u) ((omega == 2).*(u == 1) - (omega == -2).*(u == 0))*0.05*4;    
CP_ts_TSnow    = CP(omega_ts_TSnow,u_ts_TSnow);

    yhat_ts_TSnow_UNCnow  = (ergoprob_L*omega_L + (1-ergoprob_L)*omega_H)*ones(grid_dims_TSnow);
    eps_pi_ts_TSnow = quasidiff_ts_TSnow - Gamma(yhat_ts_TSnow_UNCnow,omega_ts_TSnow) - CP_ts_TSnow;

%%% 预先计算一些对插值有用的东西

substate_finder_TSnow = griddedInterpolant({b1_t_grid,dstate_t_grid,omega_t_grid,u_t_grid},reshape(1:n_gridded_substates_TSnow,gridded_dims_TSnow));

yhat_t_TSnow_fxns  = cell(1,n_gridded_substates_TSnow); pihat_t_TSnow_fxns = cell(1,n_gridded_substates_TSnow);

parfor iii=1:n_gridded_substates_TSnow
    yhat_t_TSnow_fxns{iii}  = scatteredInterpolant(b2_ts_TSnow(iii,:)',b3_ts_TSnow(iii,:)',eps_pi_ts_TSnow(iii,:)', yhat_ts_TSnow_UNCnow(iii,:)');
   pihat_t_TSnow_fxns {iii} = scatteredInterpolant(b2_ts_TSnow(iii,:)',b3_ts_TSnow(iii,:)',eps_pi_ts_TSnow(iii,:)',pihat_ts_TSnow_UNCnow(iii,:)');

结尾

%%% 插值示例，给定查询点的任意网格 (b1s,dstates,omegas,us,b2s,b3s,epses)

        yhats  = NaN(size(b1s));
        pihats= NaN(size(b1s));

            [yhats(:),pihats(:)] = interpolate(substate_finder_TSnow ,[b1s(:),dstates(:), omegas(:),us(:)],...
                                                                      [yhat_t_TSnow_fxns;pihat_t_TSnow_fxns],[b2s(:),b3(:),epses (:)]);

标签： matlabmemorymemory-managementmemory-leaksparfor