Topic Modeling: Optimal Estimation, Statistical Inference, And Beyond

With the development of computer technology and the internet, increasingly large amounts of textual data are generated and collected every day. It is a significant challenge to analyze and extract meaningful and actionable information from vast amounts of unstructured textual data. This thesis explores several problems in topic modelings and provides new algorithms with theoretical guarantees. The first part of this thesis aims to develop an optimality theory for unsupervised topic modeling under the probabilistic latent semantic indexing (pLSI) model. Novel and computationally fast algorithms for estimation and inference of both the word-topic matrix and the topic-document matrix are proposed and their theoretical properties are investigated. Moreover, a refitting algorithm is proposed to establish asymptotic normality and construct valid confidence intervals for the individual entries of the word-topic and topic-document matrices. In the second part, we study supervised topic modeling, which jointly considers a collection of documents and their paired side information. To take account of the compositional nature of the topic-document matrix, we adapt the log-contrast model and introduce a novel bias-adjusted algorithm to investigate the regression coefficients in the generalized linear model. In addition, a de-biased procedure is proposed to establish an asymptotically unbiased and normally distributed estimator, and hence valid confidence intervals are constructed for the individual entries of regression coefficients. We also investigate the errors-in-variables models under the generalized linear model framework in the third part. We proposed an estimator when the measurement error is small.