Transaction-Gateway Security Monitoring Through Real-Time Communication Behavior Analysis

Abstract

Online payment gateway systems process high-frequency communication among user devices, merchant servers, banking interfaces, fraud-control services, and third-party APIs. Abnormal network traffic in these systems may reflect credential stuffing, transaction replay, API abuse, bot-driven request bursts, or coordinated denial-of-service activities. This study proposes a SHAP-based ensemble anomaly detection framework for online payment gateway network traffic. The proposed model combines Balanced Random Forest, LightGBM, and AdaBoost using a cost-aware voting strategy to reduce missed detection of high-risk abnormal traffic. SHAP is used to explain how transaction-related network features and API communication patterns influence model decisions. Experiments are conducted on a payment gateway traffic dataset generated from a controlled transaction-processing environment with 1,200 merchant endpoints, 9 banking API interfaces, and 5 fraud-screening services. The dataset contains 7.18 million labeled request-flow records over 42 days, covering normal payment authorization, refund requests, chargeback queries, credential stuffing, replay attempts, abnormal API polling, and bot-based request flooding. After preprocessing, 55 features are retained, including request burst rate, failed authentication ratio, API endpoint switching frequency, transaction session duration, repeated token usage, response-code entropy, and source IP rotation density. The proposed model achieves 98.34% accuracy, 97.27% macro-F1, and 99.02% AUC. For replay and credential-stuffing traffic, recall reaches 95.86% and 96.44%, respectively. Compared with standalone LightGBM, the ensemble method reduces the false negative rate from 2.58% to 1.49% and improves detection of low-volume API abuse by 3.63%. SHAP analysis identifies repeated token usage, high failed authentication ratio, abnormal endpoint switching, and source IP rotation density as the most important features. The results indicate that explainable ensemble learning can strengthen real-time security monitoring for online payment gateway infrastructures.