EVALUATING THE GENERALIZATION ABILITY OF AI-GENERATED TEXT DETECTORS TO UNSEEN GENERATORS

Abstract

Many AI-generated text detectors demonstrate high performance on datasets constructed within typical evaluation protocols. In particular, classical models based on stylometric features, such as text length, punctuation patterns, and aggregated formality indicators, can effectively capture statistical regularities of machine generation. However, their performance decreases substantially when texts produced by previously unseen generators are encountered. Under such conditions, feature distributions shift, which leads to a decline in classification quality, primarily due to an increase in false negative errors. This paper investigates the generalization ability of detection models under conditions involving an unseen generator. The study compares classical stylometric models and transformer-based approaches using the LOGO (Leave-One-Generator-Out) evaluation protocol. The task is formulated as binary text classification across two domains, Reddit and Wikipedia, and involves three generators, namely ChatGPT, Davinci, and Dolly. The classical models include Random Forest and Gradient Boosting, whereas the transformer-based approaches are represented by DistilBERT and RoBERTa. Model performance is evaluated using Accuracy, Precision, Recall, F1, and Macro-F1, with the final results averaged across multiple random initializations. The results show that transformer-based models demonstrate a higher ability to generalize to texts produced by unseen generators. In contrast, stylometric approaches exhibit a substantial degradation in performance, particularly depending on the domain and text length. Error analysis indicates that the main factor behind this decline is the increase in false negative errors. An additional analysis of feature importance shows that classical models rely heavily on surface-level textual characteristics, which do not ensure stable generalization across different generators. Therefore, the findings highlight the importance of evaluating AI-generated text detectors under the LOGO protocol to ensure robust performance in the presence of new language models.