Ilmun Kim is a research associate at the Statistical Laboratory at the University of Cambridge, mentored by Richard Samworth and Rajen Shah. He completed his PhD at Carnegie Mellon University in 2020 under the joint supervision of Larry Wasserman and Sivaraman Balakrishnan. His research interests are broadly in the areas of nonparametric inference and high-dimensional statistics. Currently, his research focuses on developing statistical methods for nonparametric testing problems. He is also interested in asymptotic theory, concentration of measure and minimax theory as tools for understanding modern statistical problems.

Ilmun will deliver his Lawrence Brown PhD Student Award lecture at the online JSM, August 8–12, 2021.

Statistical power of permutation tests

A permutation test is a nonparametric approach to hypothesis testing, routinely used in a variety of scientific and engineering applications. The permutation test constructs the resampling distribution of a test statistic by permuting the labels of the observations. The resampling distribution, also called the permutation distribution, serves as a reference from which to assess the significance of the observed test statistic. A key property of the permutation test is that it provides exact control of the type I error rate for any test statistic whenever the labels are exchangeable under the null hypothesis. Due to this attractive non-asymptotic property, the permutation test has received considerable attention, and has been applied to a wide range of statistical tasks including testing independence, two-sample testing, change point detection, classification and so on.

Once the type I error is controlled, the next concern is the type II error, or equivalently the power of the resulting test. Despite its increasing popularity and empirical success, the power of the permutation test has not been fully explored beyond simple cases. While some progress has been made, existing results are often restricted to conventional asymptotic settings where underlying parameters are held fixed as the sample size increases. The goal of this work is to attempt to fill this gap by developing a general framework for studying the power of the permutation test under finite-sample scenarios.

To this end, we introduce a simple method for analyzing the non-asymptotic power of the permutation test based on the first two moments of a general test statistic. The utility of the proposed method is illustrated in the context of two-sample and independence testing under both discrete and continuous settings. In each setting, we introduce permutation tests and investigate their minimax performance. Specific examples of test statistics that we analyze include weighted U-statistics for multinomial testing and Gaussian kernel-based statistics for density testing. We also introduce exponential concentration bounds for permuted U-statistics, which allow us to obtain a sharper condition for the power analysis. Building on these exponential bounds, we propose permutation tests that are adaptive to unknown smoothness parameters without losing much power. This talk is based on joint work with Sivaraman Balakrishnan and Larry Wasserman.