Anomaly detection is the process of identifying data points that significantly deviate from the normal behavior of the dataset. The methods used for anomaly detection vary depending on the researcher's background and the nature of the data. Typically, these methods involve fitting a model to the data to define the expected behavior, and then applying statistical tests to determine if a given data point follows this behavior.
{tocify} $title={Table of Contents}
Introduction
Definition of Terms
Anomaly detection refers to the process of identifying behaviors and patterns that significantly deviate from the norm [1], [2]. These anomalies can serve as early warnings, allowing policymakers to take proactive measures to mitigate potential damages. Financial crises have profound impacts on global economies, leading countries into severe economic recessions, increased unemployment, and significant losses in welfare. Therefore, the ability to detect anomalies that may indicate the onset of a financial crisis is of utmost importance.
The process of identifying patterns that do not follow a defined behavior is also referred to as novelty detection [3], outlier detection [4], or one-class learning [5]. Anomaly Detection (AD) finds applications in many fields, including closed-circuit monitoring systems, evaluation of bank loan applications, medical diagnosis, pharmaceutical research, time series analysis, and early detection systems for financial crises. In Graph 1, we observe that over time, the daily changes in the S&P 500 index fluctuate around 0, except during specific periods of economic crises, such as the 2007-09 crisis and the 2020-2021 pandemic period [7], creating the phenomenon of "anomalies" in the consistency of the data.
Types of Anomalies
It is easy to understand that there are many types of anomalies. However, they can be categorized into three main categories:
- Point Anomalies: When an individual data point deviates significantly from the rest of the dataset, it is called a point anomaly [2]. An example of a point anomaly could be an unusually high expenditure on a credit card.
- Conditional Anomalies: When a dataset is anomalous within a specific context, it is called a conditional anomaly or contextual anomaly. The concept of context is derived from the structure of the dataset and must be specified as part of the problem formulation. For example, snowfall in a mountain village on Naxos in December would be considered normal, but the same phenomenon in July would be an "anomaly" due to the different context.
- Collective Anomalies: When a collection of data points exhibits anomalous behavior collectively, it is called a collective anomaly. Individual data instances in a collective anomaly may not be anomalous by themselves, but their presence together creates the anomaly phenomenon [2]. This type of deviation is common in financial data. For instance, in Graph 2, we observe that the adjusted closing prices of the S&P 500 index showed significant deviation over a short period, which individually would not be considered anomalies.
Main Anomaly Detection Methods
Statistical Methods
These methods utilize statistical tests and models to identify outliers and anomalies in the data. Statistical methods are simple and fast, but they may not capture complex and non-linear patterns in the data. Statistical methods were among the first approaches used for anomaly detection. These methods typically involve defining specific acceptable limits for various indicators. When a financial measurement exceeds these limits, it is considered an anomaly. The most commonly used statistical techniques include:
Z-Score Analysis
This involves standardizing data points so they have a mean of zero and a standard deviation of one. A data point is considered anomalous if its Z-score exceeds a specific threshold, indicating it is several standard deviations away from the mean [9].
- `Z` is the Z-score
- `X_i` is the `i` data point
- `μ` is the mean of the dataset
- `σ` is the standard deviation
Moving Average and Bollinger Bands
- `SMA` is the Simple Moving Average
- `p_i` is the `i` data point
- `n` is the 'moving' dataset
- `SMSTD` is the standard deviation
Extreme Value Theory
In Graph 2, an application of the major statistical methods on the S&P 500 index is illustrated. The index data were sourced from Yahoo Finance and processed using Python. Specifically, the first image of the graph depicts the Z-Score method for two standard deviations. It successfully detects part of the collective anomaly that resulted from the COVID-19 pandemic [7].
Next, the second image of the graph shows the Bollinger Bands method using a moving average. The moving average is based on 20-day data, and the Bollinger Bands correspond to two standard deviations. We observe that this method identifies more anomalies than the Z-Score method over a larger portion of the time series, not exclusively during the months of the significant drop in the index. Consequently, it fails to reveal the collective anomaly of the recession.
Finally, the last image of Graph 2 presents the Extreme Value Theory method, which does not use absolute values but quantifies the data with the median as the base value. The lower limit used is 90%, and the upper limit is 110%, meaning the acceptable values lie within a 20% range (90% - 110%). In conclusion, we observe that the statistical method that best captures the collective anomaly is the Extreme Value Theory.
Machine Learning Methods
These methods use algorithms and models that learn from the data and detect anomalies based on learned patterns. Machine learning methods are powerful and flexible but require a lot of data and computational resources, and the interpretability of the results is often challenging [13]. With advances in computational capabilities, machine learning has become a key tool for anomaly detection. Machine learning methods can handle large, complex datasets and uncover hidden patterns that traditional statistical methods may miss.
- Supervised Learning: Algorithms in this category require labeled datasets where anomalies are predefined. Models such as logistic regression, decision trees, LSTM networks, and support vector machines (SVM) are trained to classify new data points as normal or anomalous. However, in the financial context, obtaining labeled data can be difficult and expensive [13], [14].
- Unsupervised Learning: Algorithms in this category do not require labeled data and are particularly useful for detecting anomalies in financial data [15]. Techniques such as clustering (e.g., K-means, DBSCAN) and dimensionality reduction (e.g., Principal Component Analysis, t-SNE) help identify data points that do not conform to the overall pattern of the dataset [11], [16].
- Semi-Supervised Learning: This approach uses a combination of a small amount of labeled data and a large volume of unlabeled data. It is particularly useful in situations where labeled data are scarce, as is often the case in the study of financial crises [17].
- Deep Learning: Neural networks, particularly recurrent neural networks (RNN) and convolutional neural networks (CNN), can model complex temporal and spatial relationships in financial data. Autoencoders, a type of neural network used for unsupervised learning, can be trained to reconstruct input data. Significant reconstruction errors may indicate anomalies [18], [19], [20].
Hybrid Methods
These methods combine statistical and machine learning techniques to leverage the strengths of both approaches. For example, the isolation forest can be used, which is a machine learning algorithm that isolates anomalies by randomly partitioning the data into different features and measuring how easy it is to separate a data point from the rest of the data [21]. The easier the isolation, the more likely it is an anomaly. The isolation forest can also perform statistical tests to determine the threshold for anomaly detection.
Applications in Financial Economics
Early Warning Systems
Anomaly detection models form the backbone of early warning systems designed to predict financial crises. By monitoring key financial indicators and highlighting potential anomalies, these systems can provide timely alerts to policymakers [22], [23]. This enables them to take preventive measures such as adjusting monetary policies, implementing regulatory changes, or making corrections in financial markets [16]. The effectiveness of these models significantly depends on the selection and processing of relevant indicators. Commonly monitored indicators include macroeconomic indicators such as GDP, inflation, and unemployment, as well as financial indicators like spread variables and short-term debt to reserves ratios, among others [24].
Risk Management
Financial institutions and investors can also leverage anomaly detection methods to improve risk management practices. By identifying unusual patterns in index activities, company transaction flows, or profitability, investors can diversify investments by isolating financial vehicles that exhibit significant fluctuations—anomalies [22], [25]. For example, during the financial crisis of 2007-09, institutions that employed robust risk management systems based on anomaly detection were better positioned to mitigate the impending collapse [15].
Fraud Detection
Anomaly detection is also crucial at the microeconomic level for identifying fraudulent activities within the economic system. Fraudulent transactions often deviate from usual patterns, and early detection can prevent significant financial losses [26]. Techniques such as clustering, grouping, and machine learning models can help flag suspicious transactions for further investigation [27]. These techniques group transactions based on the similarity of their characteristics, allowing the detection of outliers that may indicate illicit behavior. For example, transactions that significantly deviate from an employee's usual spending patterns may indicate the theft of banking information.
Implementation Challenges
Data Quality and Availability
Detecting anomalies related to financial crises faces many significant challenges. The accuracy of anomaly detection depends heavily on the quality and availability of real-time data. This requires specialized equipment, which can be particularly expensive to acquire. Additionally, economic data are often noisy, incomplete, and subject to revisions [29], meaning that managing them can only be done by specialized researchers. Moreover, data for predicting crises may cover various sectors, geographical areas, and multiple time periods, making their integration a particularly complex process [30].
Dynamic Nature of Financial Markets
The economies of countries are interconnected, creating the global economy, resulting in international markets being extremely dynamic and volatile. They are heavily influenced by many exogenous factors, including political events, regulatory changes, and environmental conditions. This dynamic nature makes it difficult to define a stable baseline for normal behavior, complicating anomaly detection.
Interpretability of Results
In economics, understanding the rationale behind an emerging anomaly is vital for decision-making. Therefore, there is a need for models that not only detect anomalies but also enable the researcher to explain the phenomenon. Machine learning models, particularly deep learning models, although highly accurate in recognizing anomalies, offer limited interpretability.
Errors
A significant issue arising from anomaly detection methods is errors. These errors can lead to incorrect regulatory policies by financial institutions, thereby affecting the entire economic system. The likelihood of false positives (incorrectly marking normal behavior as anomalous) and false negatives (failing to detect actual anomalies) is not zero. False positives can lead to unnecessary efforts in detection, intervention, and correction, while false negatives can result in incorrect categorization and ineffective prediction [31].
Conclusion
It is clear that anomaly detection in the financial sector is an essential tool for all stakeholders. Identifying early warning signals through statistical methods and machine learning contributes to the formation of a transparent and stable economic system. However, the complex nature of international economies and financial markets presents significant challenges, requiring continuous improvement in both data quality and the interpretability of the models used.
The results of successful anomaly detection are substantial and include improved early warning systems, risk management practices, enhanced regulatory compliance of companies, and the detection of financial fraud. In conclusion, although anomaly detection is not a panacea, it is an additional powerful tool in the arsenal of mechanisms for financial stability and economic transparency. Ongoing research and collaboration between academia, industry, and regulatory authorities will be crucial for harnessing the full potential of anomaly detection in the financial sector.
References
[1] V. Chandola, A. Banerjee, and V. Kumar,
‘Anomaly detection: A survey’, ACM Comput. Surv., vol. 41, no. 3, pp.
1–58, Jul. 2009, doi: 10.1145/1541880.1541882.
[2] J. D. Parmar and J. T.
Patel, ‘Anomaly Detection in Data Mining: A Review’, Int. J. Adv. Res.
Comput. Sci. Softw. Eng., 2017.
[3] A. Purarjomandlangrudi,
A. H. Ghapanchi, and M. Esmalifalak, ‘A data mining approach for fault
diagnosis: An application of anomaly detection algorithm’, Measurement,
vol. 55, pp. 343–352, Sep. 2014, doi: 10.1016/j.measurement.2014.05.029.
[4] S. Marsland, ‘Novelty
Detection in Learning Systems’, 2003.
[5] K.-R. Muller, S. Mika,
G. Ratsch, K. Tsuda, and B. Scholkopf, ‘An introduction to kernel-based
learning algorithms’, IEEE Trans. Neural Netw., vol. 12, no. 2, pp.
181–201, Mar. 2001, doi: 10.1109/72.914517.
[6] Standard & Poor’s
Corp, ‘S&P U.S. Indices Methodology’. Accessed: May 09, 2024. [Online].
Available: https://www.spglobal.com/spdji/en/methodology/article/sp-us-indices-methodology/
[7] H. Yilmazkuday,
‘COVID-19 effects on the S&P 500 index’, Appl. Econ. Lett., vol. 30,
no. 1, pp. 7–13, Jan. 2023, doi: 10.1080/13504851.2021.1971607.
[8] Yahoo Finance, ‘S&P
500 (^GSPC) Stock Historical Prices & Data - Yahoo Finance’. Accessed: May
09, 2024. [Online]. Available: https://finance.yahoo.com/quote/%5EGSPC/history/
[9] P. J. Rousseeuw and M.
Hubert, ‘Anomaly detection by robust statistics’, WIREs Data Min. Knowl.
Discov., vol. 8, no. 2, p. e1236, Mar. 2018, doi: 10.1002/widm.1236.
[10] S. Vergura, ‘Bollinger
Bands Based on Exponential Moving Average for Statistical Monitoring of
Multi-Array Photovoltaic Systems’, Energies, vol. 13, no. 15, p. 3992,
Aug. 2020, doi: 10.3390/en13153992.
[11] X. Dong, Y. Li, D. E. Rapach,
and G. Zhou, ‘Anomalies and the Expected Market Return’, J. Finance,
vol. 77, no. 1, pp. 639–681, 2022, doi: 10.1111/jofi.13099.
[12] E. Vignotto and S.
Engelke, ‘Extreme value theory for anomaly detection – the GPD classifier’, Extremes,
vol. 23, no. 4, pp. 501–520, Dec. 2020, doi: 10.1007/s10687-020-00393-0.
[13] W. Jia, R. M. Shukla, and
S. Sengupta, ‘Anomaly Detection using Supervised Learning and Multiple
Statistical Methods’, in 2019 18th IEEE International Conference On Machine
Learning And Applications (ICMLA), Sep. 2019, pp. 1291–1297. doi:
10.1109/ICMLA.2019.00211.
[14] H. S. Dhiman, D. Deb, S.
M. Muyeen, and I. Kamwa, ‘Wind Turbine Gearbox Anomaly Detection Based on
Adaptive Threshold and Twin Support Vector Machines’, IEEE Trans. Energy Convers.,
vol. 36, no. 4, pp. 3462–3469, Sep. 2021, doi: 10.1109/TEC.2021.3075897.
[15] K. L. Tran, H. A. Le, C.
P. Lieu, and D. T. Nguyen, ‘Machine Learning to Forecast Financial Bubbles in
Stock Markets: Evidence from Vietnam’, Int. J. Financ. Stud., vol. 11,
no. 4, Art. no. 4, Dec. 2023, doi: 10.3390/ijfs11040133.
[16] K. D. Edmonds, ‘THE
EFFECTIVENESS OF MACHINE LEARNING-BASED ANOMALY DETECTION ALGORITHMS APPLIED TO
DEFENSE CONTRACT FINANCIAL DATA’, 2020.
[17] M. E. Villa-Pérez, M. Á.
Álvarez-Carmona, O. Loyola-González, M. A. Medina-Pérez, J. C. Velazco-Rossell,
and K.-K. R. Choo, ‘Semi-supervised anomaly detection algorithms: A comparative
summary and future research directions’, Knowl.-Based Syst., vol. 218,
p. 106878, Apr. 2021, doi: 10.1016/j.knosys.2021.106878.
[18] Z. Chen, C. K. Yeo, B. S.
Lee, and C. T. Lau, ‘Autoencoder-based network anomaly detection’, in 2018
Wireless Telecommunications Symposium (WTS), Apr. 2018, pp. 1–5. doi:
10.1109/WTS.2018.8363930.
[19] M.-S. Cheong, M.-C. Wu,
and S.-H. Huang, ‘Interpretable Stock Anomaly Detection Based on
Spatio-Temporal Relation Networks With Genetic Algorithm’, IEEE Access,
vol. 9, pp. 68302–68319, 2021, doi: 10.1109/ACCESS.2021.3077067.
[20] I. Ullah and Q. H.
Mahmoud, ‘Design and Development of RNN Anomaly Detection Model for IoT
Networks’, IEEE Access, vol. 10, pp. 62722–62750, 2022, doi:
10.1109/ACCESS.2022.3176317.
[21] H. Xu, G. Pang, Y. Wang,
and Y. Wang, ‘Deep Isolation Forest for Anomaly Detection’, IEEE Trans.
Knowl. Data Eng., vol. 35, no. 12, pp. 12591–12604, Sep. 2023, doi:
10.1109/TKDE.2023.3270293.
[22] A. S. Koyuncugil and N.
Ozgulbas, ‘Financial early warning system model and data mining application for
risk detection’, Expert Syst. Appl., vol. 39, no. 6, pp. 6238–6253, May
2012, doi: 10.1016/j.eswa.2011.12.021.
[23] T. Park, ‘Enhancing
Anomaly Detection in Financial Markets with an LLM-based Multi-Agent
Framework’. arXiv, Mar. 28, 2024. doi: 10.48550/arXiv.2403.19735.
[24] M. Bussiere and M.
Fratzscher, ‘Towards a new early warning system of financial crises’, J.
Int. Money Finance, vol. 25, no. 6, pp. 953–973, Oct. 2006, doi:
10.1016/j.jimonfin.2006.07.007.
[25] P. Vanini, S. Rossi, E.
Zvizdic, and T. Domenig, ‘Online payment fraud: from anomaly detection to risk
management’, Financ. Innov., vol. 9, no. 1, p. 66, Mar. 2023, doi:
10.1186/s40854-023-00470-w.
[26] T. Pourhabibi, K.-L. Ong,
B. H. Kam, and Y. L. Boo, ‘Fraud detection: A systematic literature review of
graph-based anomaly detection approaches’, Decis. Support Syst., vol.
133, p. 113303, Jun. 2020, doi: 10.1016/j.dss.2020.113303.
[27] D. Huang, D. Mu, L. Yang,
and X. Cai, ‘CoDetect: Financial Fraud Detection With Anomaly Feature
Detection’, IEEE Access, vol. 6, pp. 19161–19174, 2018, doi:
10.1109/ACCESS.2018.2816564.
[28] M. Ul Hassan, M. H.
Rehmani, and J. Chen, ‘Anomaly Detection in Blockchain Networks: A
Comprehensive Survey’, IEEE Commun. Surv. Tutor., vol. 25, no. 1, pp.
289–318, 2023, doi: 10.1109/COMST.2022.3205643.
[29] Y. Aït-Sahalia, J. Fan,
and D. Xiu, ‘High-Frequency Covariance Estimates With Noisy and Asynchronous
Financial Data’, J. Am. Stat. Assoc., vol. 105, no. 492, pp. 1504–1517,
Dec. 2010, doi: 10.1198/jasa.2010.tm10163.
[30] V. Garcia-Font, C.
Garrigues, and H. Rifà-Pous, ‘Difficulties and Challenges of Anomaly Detection
in Smart Cities: A Laboratory Analysis’, Sensors, vol. 18, no. 10, p.
3198, Sep. 2018, doi: 10.3390/s18103198.
[31] M. Grill, T. Pevný, and
M. Rehak, ‘Reducing false positives of network anomaly detection by local
adaptive multivariate smoothing’, J. Comput. Syst. Sci.,
vol. 83, no. 1, pp. 43–57, Feb. 2017, doi: 10.1016/j.jcss.2016.03.007.