时间戳

首页 > 解决方案 > 如何在使用 GBM 算法期间估计 ram 使用情况

python - 如何在使用 GBM 算法期间估计 ram 使用情况

问题描述

我想估计我的算法 XGBOOST 和 LightGBM 如何使用 python 消耗内存

我的意思是这样的：

我用这段代码来估计速度，我问是否有类似的记忆方法：

start_time = time.time()
xg_cl.fit(X_train, y_train, eval_metric="auc", early_stopping_rounds=300, eval_set=eval_set, verbose=True)
XgBoost_time_training = (time.time() - start_time)
print("--- %s seconds ---" % (time.time() - start_time))

PS：我需要一个可以在 WINDOWS 而不是 linux 中执行的代码

标签： python

解决方案

你可以使用memory_profiler.profile，这里是一个例子

import numpy as np

from xgboost import XGBRegressor
from sklearn.model_selection import train_test_split

from memory_profiler import profile


precision = 10
fp = open('memory.log', 'w+')
@profile(precision=precision, stream=fp)
def xgb_test():

    X1 = np.linspace(0, np.radians(360), 1000).reshape(-1, 1)
    X2 = np.linspace(0, np.radians(360), 1000).reshape(-1, 1)
    Y = np.sin(X1) + 0.0001 * X2
    X = np.concatenate((X1, X2), axis = 1)

    x_train, x_val, y_train, y_val = train_test_split(X, Y, test_size = 0.2, random_state = 69)
    eval_set = [(x_train, y_train), (x_val, y_val)]
    XGBModel = XGBRegressor(max_depth = 10)
    XGBModel.fit(x_train,y_train,eval_set=eval_set,early_stopping_rounds=100,eval_metric="mae",verbose=2)

if __name__ == '__main__':
    xgb_test()

通过运行脚本，每行（步骤）的内存消耗将被写入memory.log文件，如下所示：

Filename: test.py

Line #    Mem usage    Increment   Line Contents
================================================
    12 104.6992187500 MiB 104.6992187500 MiB   @profile(precision=precision, stream=fp)
    13                             def xgb_test():
    14
    15 104.6992187500 MiB   0.0000000000 MiB       X1 = np.linspace(0, np.radians(360), 1000).reshape(-1, 1)
    16 104.6992187500 MiB   0.0000000000 MiB       X2 = np.linspace(0, np.radians(360), 1000).reshape(-1, 1)
    17 104.7226562500 MiB   0.0234375000 MiB       Y = np.sin(X1) + 0.0001 * X2
    18 104.7382812500 MiB   0.0156250000 MiB       X = np.concatenate((X1, X2), axis = 1)
    19
    20 104.8320312500 MiB   0.0937500000 MiB       x_train, x_val, y_train, y_val = train_test_split(X, Y, test_size = 0.2, random_state = 69)
    21 104.8320312500 MiB   0.0000000000 MiB       eval_set = [(x_train, y_train), (x_val, y_val)]
    22 104.8320312500 MiB   0.0000000000 MiB       XGBModel = XGBRegressor(max_depth = 10)
    23 105.6718750000 MiB   0.8398437500 MiB       XGBModel.fit(x_train,y_train,eval_set=eval_set,early_stopping_rounds=100,eval_metric="mae",verbose=2)

其中Increment列显示每个步骤占用的内存量。

推荐阅读