In recent years, advancements in artificial neural networks (ANNs) have played a pivotal role in solving complex problems across various domains. However, traditional ANNs often face challenges in capturing intricate features, especially when dealing with non-linear data patterns. To address these limitations, the ensemble deep random vector functional link (edRVFL) neural network has emerged as a promising solution. This blog delves into the recent paper titled "Ensemble Deep Random Vector Functional Link Neural Network Based on Fuzzy Inference System," which introduces a novel approach that combines the strengths of edRVFL and fuzzy inference systems (FIS) to enhance feature learning capabilities.

Introduction to Neural Networks and RVFL

Neural networks have been at the forefront of machine learning due to their ability to model complex relationships between inputs and outputs. These networks aim to approximate a target function by learning a hypothesis function within a hypothesis space . The goal is to minimize the generalization error, i.e., the difference between the predicted output and the true output. Traditional neural networks rely on backpropagation (BP) to optimize parameters, which often leads to challenges such as slow convergence, susceptibility to local optima, and high computational demands.

To overcome these challenges, randomized neural networks (RNNs) were introduced. Unlike BP-based networks, RNNs fix certain parameters during training and compute output layer parameters using iterative processes or closed-form solutions. Among RNNs, the random vector functional link (RVFL) network stands out. RVFL is a shallow feed-forward network characterized by randomly initialized hidden layer parameters that remain fixed during training. This architecture allows for efficient learning with fewer adjustable settings, making it a practical choice for various applications.

However, the shallow RVFL model has limitations, such as its inability to effectively capture intricate hidden relationships due to its reliance on a single hidden layer. To address these limitations, the deep RVFL (dRVFL) and ensemble deep RVFL (edRVFL) networks were proposed. The dRVFL introduces multiple hidden layers, enabling the capture of more complex relationships, while the edRVFL leverages ensemble learning to enhance stability and effectiveness.

The Role of Fuzzy Inference Systems (FIS)

Fuzzy logic provides a mathematical framework for handling uncertainty and imprecise information. Within this framework, the fuzzy inference system (FIS) employs fuzzy logic principles to emulate human-like reasoning through the integration of fuzzy sets, linguistic variables, and a set of rules. The fundamental components of FIS include fuzzification, rule evaluation, and defuzzification. Fuzzification transforms precise input values into fuzzy sets, rule evaluation applies conditional statements to generate fuzzy output values, and defuzzification converts these fuzzy outputs into precise outcomes.

FIS has found applications in various domains, including control systems, pattern recognition, finance, and classification. Given the dynamic nature of feature enhancement capabilities in FIS, the paper proposes integrating FIS with the edRVFL architecture to create a hybrid model—edRVFL-FIS.

The Proposed edRVFL-FIS Model

The paper introduces the ensemble deep random vector functional link neural network based on a fuzzy inference system (edRVFL-FIS). The core idea behind this model is to leverage the capabilities of both deep learning and fuzzy logic to produce rich feature representations for training the ensemble model. The edRVFL-FIS model incorporates diverse clustering methods, including R-means, K-means, and Fuzzy C-means, to establish fuzzy layer rules, resulting in three model variations: edRVFL-FIS-R, edRVFL-FIS-K, and edRVFL-FIS-C.

Architecture of the edRVFL-FIS Model

The architecture of the edRVFL-FIS model is modular and structured, allowing for a systematic processing of data. Input samples undergo a fuzzification process, forming a fuzzy layer. This layer generates enhanced fuzzified features by combining original input features with fuzzy features. Each hidden layer in the model operates as an independent base model within the ensemble framework. These base models utilize both hidden and defuzzified features to make predictions.

Each base model in the edRVFL-FIS framework can be viewed as an RVFL model that incorporates distinct input features. The ensemble structure leverages diverse supervised defuzzified features for each base model, enhancing the richness of feature extraction during output prediction.

Mathematical Framework of edRVFL-FIS

To understand the edRVFL-FIS model, let's dive into its mathematical framework. The model's architecture is defined by several key equations that describe the formation of hidden layers, fuzzification, defuzzification, and output prediction.

1. Formation of Hidden Layers:
   The first hidden layer is calculated as:

   where is the input matrix, is the fuzzified matrix, is the randomly initialized weights matrix, is the bias vector, and is the activation function.

   For subsequent hidden layers (where ):

2. Output Calculation:
   The output of the first hidden layer is computed as:

   where is the unknown weight matrix, and is the defuzzified matrix.

   For the -th hidden layer:

3. Fuzzification:
   The fuzzification process generates a fuzzy layer matrix by fuzzifying each training sample using the TSK FIS. The weighted fire strength of the -th fuzzy rule is given by:

   where is the membership function corresponding to the fuzzy set .

4. Defuzzification:
   The defuzzification process converts the fuzzified vectors into defuzzified outputs that contribute to the final output layers. The defuzzified output of a sample is defined as:

   where are the parameters used in the THEN part of the fuzzy inference system.

5. Final Output:
   For an unknown sample , the output of the -th base model is:

   The final output is obtained by combining the outputs of all base models through majority voting:

Experimental Results and Analysis of the edRVFL-FIS Model

The paper titled "Ensemble Deep Random Vector Functional Link Neural Network Based on Fuzzy Inference System" conducts a series of experiments to validate the performance of the proposed ensemble deep random vector functional link neural network based on a fuzzy inference system (edRVFL-FIS). This section provides a detailed overview of these experiments, highlighting the datasets used, the compared models, the experimental setup, and the results.

1. Datasets

The experiments were conducted using two main categories of datasets:

• Category-1 UCI Datasets: These datasets include relatively smaller datasets from the UCI Machine Learning Repository, with sample sizes ranging from 100 to 1000.
• Category-2 UCI Datasets: These datasets also come from the UCI repository but feature larger sample sizes, ranging from 1000 to 10,000.
• NDC Datasets: The NDC datasets, generated using David Musicant’s NDC Data Generator, are large-scale datasets with sample sizes ranging from 10,000 to 1 million.

Each dataset varies in terms of domain, size, and complexity, providing a comprehensive evaluation of the model's performance.

2. Compared Models

The performance of the proposed edRVFL-FIS models was compared against several baseline models, including both shallow and deep variants of randomized neural networks (RNNs). The models compared in the experiments include:

1. RVFL: Standard shallow RVFL model.
2. ELM: Extreme Learning Machine, a fast learning algorithm for single-layer feedforward neural networks.
3. BLS: Broad Learning System, which expands the feature space with various feature mapping units.
4. H-ELM: Hierarchical Extreme Learning Machine, which enhances ELM by stacking multiple layers.
5. dRVFL: Deep RVFL model with multiple hidden layers.
6. edRVFL: Ensemble deep RVFL model, which leverages ensemble learning techniques.
7. Fuzzy BLS: A neuro-fuzzy-based variant of the Broad Learning System.
8. edEGERVFL: Ensemble deep extended graph embedded RVFL, which incorporates graph structures in the feature space.

The proposed models evaluated were:

1. edRVFL-FIS-R: edRVFL-FIS model with R-means clustering.
2. edRVFL-FIS-K: edRVFL-FIS model with K-means clustering.
3. edRVFL-FIS-C: edRVFL-FIS model with Fuzzy C-means clustering.

3. Experimental Setup

The experiments were conducted with a focus on evaluating the classification accuracy, standard deviation, ranking, and statistical significance of the models' performance. The models were trained and tested on each dataset, and their performance was compared using several statistical tests, including the Friedman test and the Wilcoxon-signed-rank test. These tests were employed to determine the statistical significance of the differences in performance between the proposed models and the baseline models.

The key metrics used for comparison were:

• Classification Accuracy: The percentage of correct predictions made by the model.
• Standard Deviation: The variability in model accuracy across different runs.
• Average Rank: The average ranking of the model across all datasets, with lower ranks indicating better performance.

4. Results on Category-1 UCI Datasets

Table 1 presents the classification accuracies of the proposed edRVFL-FIS models and the baseline models on Category-1 UCI datasets. The proposed models consistently outperformed the baseline models, with edRVFL-FIS-R achieving the highest average accuracy of 82.08%, followed by edRVFL-FIS-C and edRVFL-FIS-K.

Table 1: Classification Accuracies on Category-1 UCI Datasets

Explanation of Results in Table 1:

• RVFL, ELM, BLS, H-ELM, dRVFL, edRVFL, Fuzzy BLS, edEGERVFL: These are the baseline models against which the proposed edRVFL-FIS models are compared.
• edRVFL-FIS-R, edRVFL-FIS-K, edRVFL-FIS-C: These represent the proposed ensemble deep random vector functional link neural network models using R-means, K-means, and Fuzzy C-means clustering, respectively.
• Bold values: Indicate the highest accuracy for a given dataset.

The proposed models (edRVFL-FIS variants) consistently show higher average accuracies and lower average ranks compared to the baseline models, indicating their superior performance across a range of datasets.

The Wilcoxon-signed-rank test was used to compare the performance of the proposed models with baseline models. The p-values indicate that the proposed models significantly outperform the baseline models.

Table 2: Wilcoxon-signed-rank Test on Category-1 UCI Datasets

Explanation of Table 2:

• p-value: The p-value represents the probability of obtaining the observed results assuming the null hypothesis is true. A lower p-value indicates stronger evidence against the null hypothesis.
• Null Hypothesis: The null hypothesis in this context is that there is no significant difference in the performance of the proposed edRVFL-FIS models and the baseline models.

In Table 2, all the p-values are below 0.05, leading to the rejection of the null hypothesis in each case. This indicates that the proposed edRVFL-FIS models (R, K, and C variants) significantly outperform all the baseline models across the Category-1 UCI datasets.

5. Results on Category-2 UCI Datasets

The Category-2 UCI datasets, which include larger datasets, further validated the robustness of the proposed models. Table 3 shows that the edRVFL-FIS models continued to outperform the baseline models, with edRVFL-FIS-C achieving the highest average accuracy of 72.32%.

Table 3: Classification Accuracies on Category-2 UCI Datasets

Explanation of Table 3:

• RVFL, ELM, BLS, H-ELM, dRVFL, edRVFL, Fuzzy BLS: These are the baseline models against which the proposed edRVFL-FIS models are compared.
• edRVFL-FIS-R, edRVFL-FIS-K, edRVFL-FIS-C: These represent the proposed ensemble deep random vector functional link neural network models using R-means, K-means, and Fuzzy C-means clustering, respectively.
• Bold values: Indicate the highest accuracy for a given dataset.

The proposed edRVFL-FIS models (R, K, and C variants) generally achieve higher classification accuracies across the datasets, reflecting their ability to better capture the underlying data patterns compared to the baseline models. The edRVFL-FIS-C model, in particular, shows the highest average accuracy and ranks well across the datasets.

6. Results on NDC Datasets

The NDC datasets were used to evaluate the performance of the proposed models on large-scale data. These datasets ranged from 10,000 to 1 million samples. As shown in Table 4, the proposed edRVFL-FIS-K model achieved the highest average accuracy of 97.93%, demonstrating its effectiveness in handling big data.

Table 4: Classification Accuracies on NDC Datasets

The results from these tables indicate that the proposed edRVFL-FIS models consistently outperform traditional models across different datasets, showcasing their ability to generalize effectively across small, medium, and large-scale datasets. The incorporation of fuzzy inference systems within the deep RVFL framework allows the models to capture more intricate patterns, leading to superior performance.

7. Statistical Significance

To ensure that the observed differences in performance are statistically significant, the authors employed the Friedman test followed by the Wilcoxon-signed-rank test. The Friedman test was used to rank the models based on their performance across all datasets, while the Wilcoxon-signed-rank test compared the proposed models against each baseline model in a pairwise manner.

The Friedman test results confirmed that the differences in rankings were significant, leading to the rejection of the null hypothesis (no difference between the models). The Wilcoxon-signed-rank test further supported these findings, showing that the proposed edRVFL-FIS models significantly outperformed the baseline models.

Macro level conclusion of the Experiments

The paper presents extensive experimental results to demonstrate the efficacy of the proposed edRVFL-FIS models. The models were evaluated on benchmark datasets from the UCI repository and the NDC datasets, which vary in size and complexity. The proposed models consistently outperformed existing baseline models in terms of accuracy, standard deviation, and ranking.

For instance, on Category-1 UCI datasets (sample sizes ranging from 100 to 1000), the edRVFL-FIS-R model achieved an average accuracy of 82.08%, while the edRVFL-FIS-C and edRVFL-FIS-K models achieved 81.69% and 81.61%, respectively. These results were further validated using statistical tests such as the Friedman and Wilcoxon-signed-rank tests, which confirmed the statistical superiority of the proposed models over baseline models.

Similarly, on larger NDC datasets (with sample sizes ranging from 10,000 to 1 million), the edRVFL-FIS-K model achieved the highest average accuracy of 97.93%, followed by edRVFL-FIS-C (97.69%) and edRVFL-FIS-R (97.45%). These results underscore the robustness of the proposed models in handling large and complex datasets.

Conclusion and Future Directions

The proposed edRVFL-FIS model represents a significant advancement in the field of neural networks by integrating the strengths of deep learning and fuzzy logic. The use of fuzzy inference systems within the edRVFL framework enables the model to capture intricate patterns and relationships in data, resulting in superior performance across various datasets.

The paper's findings suggest that the proposed edRVFL-FIS models offer a promising solution for addressing the limitations of traditional neural networks, particularly in scenarios involving complex and non-linear data patterns. Future research could explore further enhancements to the model, such as

the incorporation of more advanced clustering techniques or the application of the model to other domains.

In conclusion, the edRVFL-FIS model represents a powerful tool for data-driven decision-making, offering a robust and versatile approach to feature learning and prediction.

References

1. Sajid, M., Tanveer, M., & Suganthan, P. N. (2024). "Ensemble Deep Random Vector Functional Link Neural Network Based on Fuzzy Inference System." IEEE Transactions on Fuzzy Systems. DOI: 10.1109/TFUZZ.2024.3411614.

2. Huang, G. B., Zhu, Q. Y., & Siew, C. K. (2006). "Extreme Learning Machine: Theory and Applications." Neurocomputing, 70(1-3), 489-501. DOI: 10.1016/j.neucom.2005.12.126.

3. Chen, C. P., & Liu, Z. (2017). "Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture." IEEE Transactions on Neural Networks and Learning Systems, 29(1), 10-24. DOI: 10.1109/TNNLS.2017.2671881.

4. Zhou, J., Zhang, L., & Tang, C. (2017). "Hierarchical Extreme Learning Machine With Applications in EEG Classification." IEEE Transactions on Cybernetics, 48(12), 3291-3303. DOI: 10.1109/TCYB.2017.2771492.

5. Shi, L., Tanveer, M., & Kumar, P. (2021). "Deep Random Vector Functional Link Networks." Neural Computing and Applications, 33(11), 5925-5937. DOI: 10.1007/s00521-020-05383-5.

6. Tanveer, M., Tiwari, A., & Suganthan, P. N. (2022). "Ensemble Deep Extended Graph Embedded Random Vector Functional Link Network." IEEE Transactions on Neural Networks and Learning Systems, 33(10), 5247-5261. DOI: 10.1109/TNNLS.2021.3134696.

7. Zadeh, L. A. (1965). "Fuzzy Sets." Information and Control, 8(3), 338-353. DOI: 10.1016/S0019-9958(65)90241-X.

8. Takagi, T., & Sugeno, M. (1985). "Fuzzy Identification of Systems and its Applications to Modeling and Control." IEEE Transactions on Systems, Man, and Cybernetics, SMC-15(1), 116-132. DOI: 10.1109/TSMC.1985.6313399.

9. Sajid, M., Tanveer, M., & Suganthan, P. N. (2024). Ensemble deep random vector functional link neural network based on fuzzy inference system. IEEE Transactions on Fuzzy Systems. https://doi.org/10.1109/TFUZZ.2024.3411614