I wrote simple code which occur page-fault via python. the process is as below.
i) make a temp file and mmap via np.memmap() api
ii) fill the memory
iii) flush memory
iv) access memory by index
I expected page fault in the iv) step. I used perf trace to monitor page fault. (sudo perf trace -F maj --no-syscalls) But page fault didn't happened. Below is my full python code.
import mmap
import os
from time import sleep
import timeit
import torch
import torch.nn as nn
from torch.optim import Optimizer
import torch.nn.functional as F
import numpy as np
from collections import namedtuple
import random
import math
from time import time
*#ref: np.memmap usage: https://pythonspeed.com/articles/mmap-vs-zarr-hdf5/*
from tempfile import mkdtemp
import os.path as path
filename = path.join(mkdtemp(), 'newfile.dat') #make tmp file in the /tmp/tmp*/newfile.dat
data_size = 10240*10240
data = np.arange(data_size, dtype='float32')
data.resize((10240,10240))
#step i)
fp = np.memmap(filename, dtype='float32', mode = 'w+', shape=(10240,10240))
#step ii)
fp[:] = data[:] #this occur page fault to file /tmp/tmp*/newfile.dat
#step iii)
fp.flush()
os.system('sudo sh -c "sync; echo 3 /proc/sys/vm/drop_caches"')
sleep(5)
#step iv)
'''
Bellow four access accur 4 page fault to file /tmp/tmp*/newfile.dat
if do 'fp[:] = data[:]' before these, fault doesn't occur though I flush cache before access.
'''
print(fp[0])
print(fp[100])
print(fp[1000])
print(fp[10000])
is there anything I missed?
I tried access array 'fp' by index without filling(e.g., fp[:] = data[:]). If I access array 'fp' without filling, page fault occur.
If I skip process ii) and iii), page fault occur in the step iv) as bellow
14353.230 ( 0.000 ms): sudo/63776 majfault [0x7fc101575004] => /tmp/tmpz00lweix/newfile.dat@0x4 (d.)
14355.719 ( 0.000 ms): sudo/63776 majfault [0x7fc10195d004] => /tmp/tmpz00lweix/newfile.dat@0x3e8004 (d.)
14357.331 ( 0.000 ms): sudo/63776 majfault [0x7fc103c85004] => /tmp/tmpz00lweix/newfile.dat@0x2710004 (d.)
14357.658 ( 0.000 ms): sudo/63776 majfault [0x7fc119c15004] => /tmp/tmpz00lweix/newfile.dat@0x186a0004 (d.)