php - Large amount of data. Best way to iterate over, without getting memory exhaustion?
问题描述
Using Laravel 6, and so Eloquent Collection classes.
So I have a "lot" of data to process. Roughly 5000 rows, and when fetching it, this generates a Collection of 5000 models. Now each of those models has maybe 20 attributes that need to be read.
Is there a fast way to do this? I currently have an array of the attributes I want to read, and then loops set up like this:
\fopen()...
foreach ($models as $model) {
$row = [];
foreach ($this->attributes as $attr) {
$row[] = \data_get($model, $attr);
}
\fputcsv($fh, $row);
}
\fclose()...
$models
is a Laravel Collection, created by EloquentModel::find($ids);
where $ids
is an array of integers. (5000 ids from the db)
$this
refers to the class which contains the foreach loops. There is nothing else in this class besides the function that contains the code above, and the attributes
property which is just an array of strings.
For 5000 rows, each row loops 20 attributes, this can take a very long time to process, and in every case this actually throws a FatalErrorException: Allowed memory size of 134217728 bytes exhausted
So what is the fastest way to retrieve the set of attributes for each row? I can't think of a faster one than this nested loop personally.
Additionally, seeing as \fputcsv()
is writing each line to file, and the $row
variable is being overwritten each loop, why am I still getting Allowed memory size exhausted
?
Would a LazyCollection be the solution here?
Thanks!
解决方案
This would process the models in the chunks, 200 at a time, thereby saving a lot of memory.
Model::whereIn('id', $ids)->chunk(200, function($models){
foreach ($models as $model) {
$row = [];
foreach ($this->attributes as $attr) {
$row[] = \data_get($model, $attr);
}
\fputcsv($fh, $row);
}
});
推荐阅读
- excel - Excel Power 查询逻辑使用逻辑获取未出现的数据
- html - 使用 css 旋转的文本未正确对齐
- python - 如何让网页仅从 html 子集中抓取链接
- javascript - 从单击的特定按钮获取ID?
- php - 使用codeigniter根据级别创建动态菜单
- timer - Swiftui - 定时器在一段时间内漂移
- java - 用于高级修剪字符串的 Java 程序
- javascript - 有没有办法让纯 React App(没有破折号)使用 plotly.js 制作一个带有两个下拉列表的图表,这样两者都会影响显示的图表?
- json - 如何配置 kafka 反序列化器,以便它只传递带有值的键?
- html - 从 vs 代码运行 html 文件不包括驱动器路径