Incorporating BERT-based NLP and Transformer for An Ensemble Model and its Application to Personal Credit Prediction
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Tree-based algorithms have been the dominant methods used build a prediction model for tabular data. This also includes personal credit data. However, they are limited to compatibility with categorical and numerical data only, and also do not capture information of the relationship between other features. In this work, we proposed an ensemble model using the Transformer architecture that includes text features and harness the self-attention mechanism to tackle the feature relationships limitation. We describe a text formatter module, that converts the original tabular data into sentence data that is fed into FinBERT along with other text features. Furthermore, we employed FT-Transformer that train with the original tabular data. We evaluate this multi-modal approach with two popular tree-based algorithms known as, Random Forest and Extreme Gradient Boosting, XGBoost and TabTransformer. Our proposed method shows superior Default Recall, F1 score and AUC results across two public data sets. Our results are significant for financial institutions to reduce the risk of financial loss regarding defaulters.
Abstract
Tree-based algorithms have been the dominant methods used build a prediction model for tabular data. This also includes personal credit data. However, they are limited to compatibility with categorical and numerical data only, and also do not capture information of the relationship between other features. In this work, we proposed an ensemble model using the Transformer architecture that includes text features and harness the self-attention mechanism to tackle the feature relationships limitation. We describe a text formatter module, that converts the original tabular data into sentence data that is fed into FinBERT along with other text features. Furthermore, we employed FT-Transformer that train with the original tabular data. We evaluate this multi-modal approach with two popular tree-based algorithms known as, Random Forest and Extreme Gradient Boosting, XGBoost and TabTransformer. Our proposed method shows superior Default Recall, F1 score and AUC results across two public data sets. Our results are significant for financial institutions to reduce the risk of financial loss regarding defaulters.
참고문헌 (21)
- Wang, Chongren, "A Deep Learning Approach for Credit Scoring Using Feature Embedded Transformer", Applied Sciences, 12(21) (2022)
- Markov, Anton, "Credit scoring methods : Latest trends and points to consider", The Journal of Finance and Data Science, 8, pp.180~201 (2022)
- Hayashi, Yoichi, "Emerging trends in deep learning for credit scoring : A review", Electronics, 11(19) (2022)
- Crook, Jonathan N., "Recent developments in consumer credit risk assessment", European Journal of Operational Research, 185(3), pp.1447~1465 (2007)
- Li, Yu, "Credit risk prediction based on machine learning methods", 2019 14th International Conference on Computer Science & Education(ICCSE) (2019)
- Li, Hua, "XGBoost model and its application to personal credit evaluation", IEEE Intelligent Systems, 35(3), pp.52~61 (2020)
- Misra, Siddharth, "Noninvasive fracture characterization based on the classification of sonic wave travel times", Machine learning for subsurface characterization, pp.243~287 (2020)
- L Breiman, "Random Forests", pp.5~32 (2001), Springer
- Kotu, Vijay, "Data science", pp.19~37 (2019), Morgan Kaufmann
- Deshpande, V. K., "Predictive Analytics and Data Mining" (2015), Elsevier Inc
- Vaswani, Ashish, "Attention is all you need", Advances in Neural Information Processing Systems, 30 (2017)
- Zhu, Yitan, "Converting tabular data into images for deep learning with convolutional neural networks", Scientific reports, (11325) (2021)
- Gorishniy, Yury, "Revisiting deep learning models for tabular data", Advances in Neural Information Processing Systems, 34, pp.18932~18943 (2021)
- Erickson, Nick, "Multimodal automl for image, text and tabular data", Proceeding of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.4786~4787 (2022)
- Huang, Allen H., "FinBERT : A large language model for extracting information from financial text", Contemporary Accounting Research, 40(2) (2023)
- Ju, Cheng, "The relative performance of ensemble methods with deep convolutional neural networks for image classification", Journal of Applied Statistics, 45(15) (2018)
- Sun, Chi, "How to fine-tune bert for text classification?", Chinese Computational Linguistics: 18th China National Conference, CCL 2019 (2019), Springer International Publishing
- Kotsiantis. Sotiris, "Handling imbalanced datasets : A review", GEST international transactions on computer science and engineering, 30(1), pp.25~36 (2006)