I found this book: https://sites.stat.washington.edu/jaw/COURSES/580s/582/HO/Lehmann_and_Romano-TestingStatisticalHypotheses.pdf
Chapter 15 lays out how to construct statistical test by subsampling. It seems a very convincing method, and can at least serve as an alternative for testing on the original sample. I have talked with quite a few PhDs in statistical area, but none of them heard of the idea - construct statistical test by subsampling.
Why it's not popular?
