0

I wrote simple code which occur page-fault via python. the process is as below.

i) make a temp file and mmap via np.memmap() api

ii) fill the memory

iii) flush memory

iv) access memory by index

I expected page fault in the iv) step. I used perf trace to monitor page fault. (sudo perf trace -F maj --no-syscalls) But page fault didn't happened. Below is my full python code.

import mmap
import os
from time import sleep
import timeit
import torch
import torch.nn as nn
from torch.optim import Optimizer
import torch.nn.functional as F
import numpy as np
from collections import namedtuple
import random
import math
from time import time


*#ref: np.memmap usage: https://pythonspeed.com/articles/mmap-vs-zarr-hdf5/*
from tempfile import mkdtemp
import os.path as path
filename = path.join(mkdtemp(), 'newfile.dat') #make tmp file in the /tmp/tmp*/newfile.dat


data_size = 10240*10240
data = np.arange(data_size, dtype='float32')
data.resize((10240,10240))

#step i)
fp = np.memmap(filename, dtype='float32', mode = 'w+', shape=(10240,10240))

#step ii)
fp[:] = data[:] #this occur page fault to file /tmp/tmp*/newfile.dat

#step iii)
fp.flush()
os.system('sudo sh -c "sync; echo 3 /proc/sys/vm/drop_caches"')
sleep(5)

#step iv)
'''
Bellow four access accur 4 page fault to file /tmp/tmp*/newfile.dat
if do 'fp[:] = data[:]' before these, fault doesn't occur though I flush cache before access.
'''
print(fp[0])
print(fp[100])
print(fp[1000])
print(fp[10000])

is there anything I missed?

I tried access array 'fp' by index without filling(e.g., fp[:] = data[:]). If I access array 'fp' without filling, page fault occur.

If I skip process ii) and iii), page fault occur in the step iv) as bellow

 14353.230 ( 0.000 ms): sudo/63776 majfault [0x7fc101575004] => /tmp/tmpz00lweix/newfile.dat@0x4 (d.)
 14355.719 ( 0.000 ms): sudo/63776 majfault [0x7fc10195d004] => /tmp/tmpz00lweix/newfile.dat@0x3e8004 (d.)
 14357.331 ( 0.000 ms): sudo/63776 majfault [0x7fc103c85004] => /tmp/tmpz00lweix/newfile.dat@0x2710004 (d.)
 14357.658 ( 0.000 ms): sudo/63776 majfault [0x7fc119c15004] => /tmp/tmpz00lweix/newfile.dat@0x186a0004 (d.)
이두솔
  • 1
  • 1
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. – Community Nov 08 '22 at 08:56
  • You are better off testing this kind of low level stuff in C, not Python + Numpy, which add one thousand different layers of abstractions on top of the OS. – Marco Bonelli Nov 09 '22 at 14:22

0 Answers0