python - 使用 madvise 加速 python mmap

Question

我使用 pythonmmap相当快地访问非常大文件的随机位置。现在我阅读madvise()并尝试进一步加快内存映射文件的随机访问。我现在看到的行为令人困惑，所以我请求帮助。我在文件中有一个位置数组（字节偏移），我想在其中读取以下多行。对于这些字节偏移中的每一个，我都会调用mm.madvise(mmap.MADV_WILLNEED, ..)以告诉内核（？）它们将很快被访问。如果我抓住一条线，这会加快速度。但是，如果我尝试访问多条线路，那么时间就会很奇怪。下面的一些示例代码和时序：

import mmap
import numpy as np

source = "path/to/file"
offsets = np.array([28938058915, 12165253255,  3363686649,  2337907709, 18321471207,
                    3043986123, 29547707866, 23431772405, 14399201212,  8695070697], dtype="uint64")


def get_data_batch(source, offsets):
    # open the big file in byte mode and initiate the mmap
    with open(source, 'rb') as f:
        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
        pagesize = 4096

        # prepare starts sites for madvise call
        # start of WILLNEED has to be multiple of pagesize
        new_offsets = offsets - (offsets % pagesize)

        # tell kernel which positions are needed
        for new_offset in new_offsets:
            # preload say 20 pages of data following each offset
            mm.madvise(mmap.MADV_RANDOM)
            mm.madvise(mmap.MADV_WILLNEED, int(new_offset), 20)

        # now actually read the data
        for offset in offsets:
            # Use offset to jump to position in file
            mm.seek(offset)
            # read several lines at that position
            chunk1 = mm.readline()
            chunk2 = mm.readline()
            chunk3 = mm.readline()
            chunk4 = mm.readline()
            chunk5 = mm.readline()
            chunk6 = mm.readline()

如果我在不使用 的情况下执行该函数madvise()，我可以看到访问第一行需要相当长的时间，但以下几行非常快：

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
...

    22       100    1360035.0  13600.4     98.1              chunk1 = mm.readline()
    23       100      12004.0    120.0      0.9              chunk2 = mm.readline()
    24       100         94.0      0.9      0.0              chunk3 = mm.readline()
    25       100        282.0      2.8      0.0              chunk4 = mm.readline()
    26       100         63.0      0.6      0.0              chunk5 = mm.readline()
    27       100      11785.0    117.8      0.8              chunk6 = mm.readline()

如果我包含madvise()，我会获得整体加速，并且第一行的阅读速度非常快。但是以下一些现在需要一些时间，尽管要少得多：

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
...
    22       100       4514.0     45.1      1.7              chunk1 = mm.readline()
    23       100     107462.0   1074.6     39.7              chunk2 = mm.readline()
    24       100         89.0      0.9      0.0              chunk3 = mm.readline()
    25       100      79073.0    790.7     29.2              chunk4 = mm.readline()
    26       100         91.0      0.9      0.0              chunk5 = mm.readline()
    27       100      71475.0    714.8     26.4              chunk6 = mm.readline()

有人可以解释发生了什么吗？有没有办法让所有readline()电话都像最快的电话一样快？

谢谢

标签： pythonlinuxperformanceoptimizationmmap