407 lines
23 KiB
TeX
407 lines
23 KiB
TeX
\subsection{Evaluation Metrics}
|
||
|
||
\textit{Precision}, \textit{recall} and \textit{F-measure} are widely used standard metrics in related work, such as issue assignment \cite{zanetti2013categorizing}, bug prediction \cite{d2010extensive,neuhaus2007predicting} and reviewer recommendation \cite{lee2013patch,jeong2009improving}.
|
||
These metrics can measure the performance of models from different perspectives.
|
||
For instance, \textit{precision} is used to measure the exactness of the prediction, whereas \textit{recall} evaluates the completeness.
|
||
|
||
\textit{F-measure} denotes the balance and discrepancy between \textit{precision} and \textit{recall},
|
||
which can be interpreted as the weighted average of \textit{precision} and \textit{recall}:
|
||
|
||
\begin{equation}
|
||
F-measure = 2*\frac{precision*recall}{precision+recall}
|
||
\end{equation}
|
||
|
||
In~\cite{zhou2014combining}, the weighted average value of \textit{F-measure} for both categories
|
||
is used to evaluate the classification model.
|
||
This metric considers the performance of both categories and provides an overall performance of the classification model.
|
||
Thus, in this paper, metrics similar to~\cite{zhou2014combining}, which are defined in Equation \ref{equation:performance}, were selected,
|
||
in which the \textit{average F-measure} as $f_{avg}$, \textit{F-measure} of bug (nonbug) as ${f_{bug}}$ (${f_{nonbug}}$), and number of bug (nonbug) as ${n_{bug}}$(${n_{nonbug}}$).
|
||
|
||
\begin{equation}
|
||
{f_{avg}} = \frac{n_{bug}*f_{bug}+n_{nonbug}*f_{nonbug}}{n_{bug}+n_{nonbug}}
|
||
\label{equation:performance}
|
||
\end{equation}
|
||
|
||
|
||
\subsection{RQ1:Performance of Text-based Classification}
|
||
|
||
\begin{table*}[htbp]
|
||
\centering
|
||
\caption{\label{tab:compareML} Comparisons of different text-based classifiers}
|
||
\begin{tabular}{lccccc}
|
||
\toprule
|
||
Group A vs. Group B & Estimator & Lower & Upper & Statistic & p-value \\
|
||
\midrule
|
||
NB vs. Base Line & 0.028 & 0.010 & 0.074 & -9.61727834 & 0.000000e+00 \\
|
||
LR vs. Base Line & 0.005 & 0.001 & 0.025 & -8.85487385 & 0.000000e+00 \\
|
||
RF vs. Base Line & 0.001 & 0.000 & 0.016 & -6.53029671 & 3.446741e-10 \\
|
||
SVM vs. Base Line & 0.001 & 0.001 & 0.002 & -43.36487241 & 0.000000e+00 \\
|
||
SVM vs. NB & 0.127 & 0.067 & 0.226 & -7.62243934 & 0.1.173506e-13 \\
|
||
SVM vs. LR & 0.310 & 0.207 & 0.437 & -4.04251177 & 5.212218e-04 \\
|
||
SVM vs. RF & 0.296 & 0.195 & 0.421 & -4.36384024 & 1.115098e-04 \\
|
||
% \bottomrule
|
||
% Combined method vs. SVM & 0.348 & 0.246 & 0.466 & -3.277566 & 0.005725885 \\
|
||
% Combined method vs. developer information & 0.398 & 0.290 & 0.517 & -2.207413 & 0.012257345 \\
|
||
% Combined method vs. perplexity information & 0.448 & 0.336 & 0.567 & -1.113729 & 0.048434745 \\
|
||
|
||
\bottomrule
|
||
\end{tabular}
|
||
\end{table*}
|
||
|
||
For issue classification,
|
||
four different machine learning classifiers (``\textit{Naive Bayes}'', ``\textit{Logistic Regression}'', ``\textit{Random Tree}'', ``\textit{Support Vector Machine}'') were used on our dataset,
|
||
and four classification models were built for each project
|
||
to discuss which text-based classifier performs best.
|
||
A baseline method was set, a grep-based method, which uses a simple grep with keywords like ``bug'',``defect'', or ``fix'' to determine whether an issue is bug.
|
||
Boxplot was used to exhibit the result of each classifier and acquire the overall performance of all projects for each classifier.
|
||
The \textit{average F-measure} (\ie $f_{avg}$) was used to evaluate these classifiers and calculate the average value of ten-fold results as performance.
|
||
Figure~\ref{figure:performanceML} shows the $f_{avg}$ of baseline method and four different classifiers,
|
||
where the y-axis is $f_{avg}$.
|
||
|
||
|
||
Figure \ref{figure:performanceML} shows that all four classifier approaches outperform the baseline method.
|
||
\textit{Logistic Regression} and \textit{Random Tree} reach a close performance,
|
||
which are better than that of \textit{Naive Bayes} but slightly worse than \textit{SVM}.
|
||
|
||
Statistical analysis was used to verify our conclusions about the difference between text-based classifiers and the baseline method.
|
||
Traditionally, the comparison of multiple groups follows a two-step approach:
|
||
first, a global null hypothesis is tested,
|
||
and then multiple comparisons are used to test the sub-hypotheses pertaining to each pair of groups.
|
||
However, the global test null hypothesis may be rejected, whereas none of the sub-hypotheses are rejected, or vice versa \cite{gabriel1969simultaneous}.
|
||
Therefore, the one-step approach, multiple contrast test procedure $\widetilde{\textbf{T}}$~\cite{konietschke2012rank,yu2016IST},
|
||
is preferred in this study.
|
||
The procedure $\widetilde{\textbf{T}}$ by \textit{nparcomp} package \cite{fraenkel1993design} in R was implemented to evaluate the F-Measure of all the approaches operating on 80 projects in our dataset.
|
||
The \textit{Tukey} (all-pairs) was set to contrast to compare all groups pairwise.
|
||
For each pair of groups, the 95\% confidence interval was analyzed to test whether the corresponding null sub-hypothesis can be rejected.
|
||
If the lower boundary of the interval is greater than zero for groups A and B,
|
||
then the metric value is higher in A than in B.
|
||
Similarly, if the upper boundary of the interval is less than zero for groups A and B,
|
||
then the metric value is lower in A than in B.
|
||
Finally, if the lower boundary of the interval is less than zero and the upper boundary is greater than zero,
|
||
then the data do not provide sufficient evidence to reject the null hypothesis.
|
||
|
||
|
||
|
||
% \begin{table*}[htbp]
|
||
% \centering
|
||
% \caption{\label{tab:compareImprove}Performance comparison of different approaches}
|
||
% \begin{tabular}{llcccc}
|
||
% \toprule
|
||
% Case & Group A vs. Group B & Avg. Imp & Min. Imp & Max. Imp & p.Value \\
|
||
% \midrule
|
||
% Performance < 0.75 & DI + SVM vs. SVM & 0.003 & 0.003 & 0.033 & 0.8743 \\
|
||
% & PI + DI + SVM vs. SVM & 0.024 & 0.002 & 0.061 & 0.0041 \\
|
||
% & PI + DI + SVM vs. DI + SVM & 0.021 & 0 & 0.076 & 0.0168 \\
|
||
% \midrule
|
||
% Performance < 0.8 & DI + SVM vs. SVM & 0.001 & -0.031 & 0.033 & 0.8163 \\
|
||
% & PI + DI + SVM vs. SVM & 0.014 & -0.008 & 0.061 & 0.0124 \\
|
||
% & PI + DI + SVM vs. DI + SVM & 0.013 & 0 & 0.076 & 0.0299 \\
|
||
% \bottomrule
|
||
% \end{tabular}
|
||
% \end{table*}
|
||
|
||
Table~\ref{tab:compareML} shows results of procedure $\widetilde{\textbf{T}}$
|
||
(the last three rows are the results of Section~\ref{sec:2stage}).
|
||
All the \textit{p-values} are less than 0.05 in the first four rows.
|
||
Thus, a significant difference among the four text-based classifications and base line method is observed,
|
||
which implies that text-based classifications are useful for the issue classification in ITS of GitHub.
|
||
Similarly, all the \textit{p-values} are less than 0.05 in the next four rows and the lower and upper boundaries are greater than zero,
|
||
which means that the performance of \textit{SVM} is significantly better than the other three classifications.
|
||
|
||
% \fbox{
|
||
% \parbox{
|
||
% \begin{center}
|
||
% aaa\\
|
||
% bbb
|
||
% \end{center}
|
||
% }
|
||
% }
|
||
|
||
\begin{framed}
|
||
\noindent
|
||
\textbf{Result 1:}
|
||
\textit{
|
||
In the context of GitHub's ITS, text-based classification approaches can achieve
|
||
69.7\% to 98.9\% of average F-measure (calculated as Equation~\ref{equation:performance}) on our large-scale dataset,
|
||
and the SVM classifier is the most effective approach compared to other typical classifiers.
|
||
%For most of projects in GitHub, MLTs still work on classifying issues.
|
||
%And SVM performs significant better than other 3 MLTs for 101 projects.
|
||
}
|
||
\end{framed}
|
||
|
||
\begin{figure}[!htb]
|
||
\centering
|
||
% \includegraphics[width=8.5cm]{classprocess}
|
||
\includegraphics[width=8.5cm]{figure/boxplot}
|
||
\caption{$f_{avg}$ of different ML methods}
|
||
\label{figure:performanceML}
|
||
\end{figure}%picture
|
||
|
||
\subsection{RQ2:Regression Analysis}
|
||
\label{sec:regressionResult}
|
||
|
||
Table~\ref{tab:fixedmodel} shows the result of regression analysis.
|
||
The model achieves an amazing fit (${R_c}^2 = 92.5\%$).
|
||
In our model, all the remaining factors are well below three,
|
||
thereby indicating the absence of multicollinearity~\cite{gharehyazie2014developer}.
|
||
Moreover, no interaction among the variables in the models is observed,
|
||
making the interpretation of our results easy and maintaining the cleanliness of the models.
|
||
For project-level measures, the number of issues ($log(issue\_num)$) is highly significant,
|
||
which means that the number of issues in train set is far from sufficient. Based on current data set, the more training sets used,
|
||
the higher is the \textit{average F-measure} that the classification achieves.
|
||
Moreover, no statistical significance is observed in other project-level measures with regard to the influencing the performance of the classification model.
|
||
For issue-level measures, the number of confused issues ($log(confuse\_count + 0.5)$) contained in the dataset is highly significant.
|
||
When the dataset contains many confused issues, too many instances locate closely at the hyperplane of the classification model,
|
||
which complicates the construction of an effective model.
|
||
The median number of words ($log(med\_word\_count)$) in issues is insignificant,
|
||
thereby suggesting that providing many textual summaries of the issues does not help in distinguishing between bug-prone or non-bug-prone,
|
||
and some key words may be enough to build an effective classification model.
|
||
|
||
|
||
|
||
% \begin{table}[htbp]
|
||
% \centering
|
||
% \caption{\label{tab:compareML}Performance comparison of different ML techniques}
|
||
% \begin{tabular}{lrrll}
|
||
% \toprule
|
||
% & Estimate & t Value & p value & \\
|
||
% \midrule
|
||
% $log(star + watch)$ & 0.006207 & 3.216 & 0.00173 & ** \\
|
||
% $log(post_num)$ & 0.078255 & 13.754 & <2e-16 & *** \\
|
||
% $log(confuse_count + 0.5)$ & -0.064934 & -18.108 & <2e-16 & *** \\
|
||
% $log(med_word_count)$ & 0.009829 & 1.265 & 0.20883 & \\
|
||
% \bottomrule
|
||
% \end{tabular}
|
||
% \end{table}
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\caption{\label{tab:fixedmodel}Regression result of fixed effect}
|
||
\begin{tabular}{l|r@{}lr@{}l}
|
||
\hline
|
||
& Coeffs & & Sum Sq. & \\ \hline
|
||
|
||
(intercept) & -3.37543 &* & & \\
|
||
$log(star + watch)$ & 0.06316 & & 0.197 & \\
|
||
$log(issue\_num)$ & 1.90440 &***& 30.484 &*** \\
|
||
$log(contributors)$ & -0.03135 & & 0.022 & \\
|
||
$log(age + 0.5)$ & -0.22421 &* & 0.288 & \\
|
||
$log(commits)$ & -0.34256 & & 0.842 &* \\ \hline
|
||
|
||
$log(confuse\_count + 0.5)$ & -1.83346 &***& 134.623 &*** \\
|
||
$log(med\_word\_count)$ & 0.12505 & & 0.067 & \\ \hline
|
||
|
||
marginal R-squared & \multicolumn{4}{c}{0.6798150} \\
|
||
conditional R-squared & \multicolumn{4}{c}{0.9251896} \\ \hline
|
||
|
||
\multicolumn{5}{l}{signif.: $p<0.001$ `***', $p<0.01$ `**', $p<0.05$ `*'} \\
|
||
\end{tabular}
|
||
\end{table}
|
||
|
||
\begin{framed}
|
||
\textbf{Result 2:}
|
||
\textit{
|
||
% 增加训练集的数量能够有效的提升模型的性能，除此之外，训练数据中如果包含太多的confused issue，会严重影响训练集的性能。
|
||
Increasing the size of the training set can effectively improve the performance of the classification model.
|
||
Furthermore, too many confused issues in the training set will seriously affect its performance.
|
||
}
|
||
\end{framed}
|
||
|
||
\subsection{RQ3:two-stage Classification}
|
||
\label{sec:2stage}
|
||
|
||
% \begin{table*}[htbp]
|
||
% \centering
|
||
% \caption{Classification results of different methods}
|
||
% \label{table:detailresult}
|
||
% \begin{tabular}{|l|l|ccc|ccc|ccc|}
|
||
% \hline
|
||
% & & \multicolumn{3}{c|}{1st baseline} & \multicolumn{3}{c|}{2nd baseline} & \multicolumn{3}{c|}{our approach} \\
|
||
% & & Precision & Recall & F-Meature & Precision & Recall & F-Meature & Precision & Recall & F-Meature \\ \hline
|
||
% \multirow{3}{*}{First Quartile} & Bug & 0.720 & 0.672 & 0.670 & 0.718 & 0.680 & 0.676 & 0.734 & 0.715 & 0.706 \\
|
||
% & Nonbug & 0.747 & 0.708 & 0.708 & 0.747 & 0.708 & 0.711 & 0.754 & 0.737 & 0.730 \\
|
||
% & Average & 0.734 & 0.690 & 0.689 & 0.733 & 0.694 & 0.694 & \textbf{0.744} & \textbf{0.726} & \textbf{0.718} \\ \hline
|
||
% \multirow{3}{*}{Median} & Bug & 0.748 & 0.719 & 0.715 & 0.748 & 0.722 & 0.719 & 0.761 & 0.746 & 0.731 \\
|
||
% & Nonbug & 0.765 & 0.727 & 0.733 & 0.765 & 0.729 & 0.735 & 0.770 & 0.759 & 0.750 \\
|
||
% & Average & 0.757 & 0.723 & 0.724 & 0.757 & 0.726 & 0.727 & \textbf{0.766} & \textbf{0.753} & \textbf{0.741} \\ \hline
|
||
|
||
% \end{tabular}
|
||
% \end{table*}
|
||
|
||
|
||
% 介绍实验方法 分为4个PART
|
||
% 介绍每个part两层的方法普遍有提升，而且对第一层为lg和nb的方法提升较多
|
||
% 对part1提升效果小，因为混乱的内容少，但是对part4提升效果多
|
||
|
||
In this section, an experiment based on the two-stage classifier was conducted to validate our approach.
|
||
The second stage of our approach is designed to extract semantic perplexity information.
|
||
Since the number of confused issues in the training set will influence the experiment results because of conclusion Section \ref{sec:regressionResult},
|
||
we divide the data set according to the proportion of confused issues.
|
||
Relying on first quartile, median and third quartile of confused issues ratio, we divide the 80 projects into 4 parts,
|
||
and each part contains 20 projects (Part I contains the fewest confused issues and Part IV contains the most confused issue).
|
||
For two-stage classifier, we choose different combinations from 4 text-based classifiers and finally we get a total of 16 combinations results.
|
||
In addition, these 4 text-based classifiers are used as comparative experiments in this section.
|
||
|
||
% An experiment based on the two-stage classifier was conducted to validate our approach.
|
||
% In the first stage, SVM was used as text classifier based on the conclusion of RQ1.
|
||
% In the second stage, we selected Logistic Regression as our prediction model, which performed better than other classifier in table~\ref{tag:packages}.
|
||
% As projects that achieve a high $f_{avg}$ (\ie average F-measure) contain few confused issues,
|
||
% our approach has a slight effect on these projects.
|
||
% Thus, to explore the performance of our approach for different projects, the project selection has two cases.
|
||
% In the first case, projects whose $f_{avg}$ is less than the \textit{first quartile} (0.7521) are selected.
|
||
% In the second case, projects whose $f_{avg}$ is less than the \textit{median} (0.7935) are selected.
|
||
% In this paper, \textit{\textbf{SVM}} is selected as the first method,
|
||
% whose $f_{avg}$ is the best among the four different text-based classifiers.
|
||
% Two other approaches are used to explore the effect of \textit{\textbf{developer information}}
|
||
% and \textit{\textbf{perplexity information}} in our two-stage approach.
|
||
% The developer information method only extracts probability of being bug-prone from free text in the first stage, because omitting the structured information is serious in GitHub;
|
||
% thus, the historical activities of the reporter in Stage 2 were selected to build as a classifier similar to work~\cite{zhou2014combining}
|
||
% and are described in Section~\ref{section:secondlevel}.
|
||
% The perplexity information method,
|
||
% which is used to compare the effect of perplexity information and developer information,
|
||
% extracts perplexity in the first stage, and does not use structured developer information in the second stage.
|
||
% The \textit{\textbf{combined method}} is our two-stage approach, which makes use of both perplexity information and developer information.
|
||
|
||
% Table~\ref{table:detailresult} shows the comparison results of precision, recall,
|
||
% and F-measure for two baseline approaches with the results of our approach.
|
||
% The values in Table~\ref{table:detailresult} are the average results of 101 projects.
|
||
% Figure~\ref{figure:2levelresult} shows the comparison results of $f_{avg}$.
|
||
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\caption{\label{tab:compare2stage} Comparisons of different text-based classifiers and 2-stage classifiers}
|
||
\begin{tabular}{ccccccc}
|
||
% \hline
|
||
\toprule
|
||
\multirow{2}{*}{Part} &\multirow{2}{*}{first stage} & \multicolumn{1}{c}{\multirow{2}{*}{1-stage}} & \multicolumn{4}{c}{2-stage} \\ \cline{4-7}
|
||
& & \multicolumn{1}{c}{} & NB & RF & LR & SVM \\
|
||
\midrule
|
||
\multirow{4}{*}{Part I} & NB & 72.73\% & 75.15\% & 82.68\% & 82.71\% & 82.58\% \\
|
||
& RF & 81.26\% & 81.32\% & 81.31\% & 81.26\% & 81.26\% \\
|
||
& LR & 81.73\% & 82.42\% & 84.63\% & 85.28\% & 85.16\% \\
|
||
& SVM & \textbf{85.25\%} & 85.22\% & 84.88\% & \textbf{85.48\%} & 85.35\% \\
|
||
\midrule
|
||
\multirow{4}{*}{Part II} & NB & 70.82\% & 74.20\% & 80.13\% & 79.47\% & 79.10\% \\
|
||
& RF & 77.91\% & 77.93\% & 78.06\% & 77.91\% & 77.91\% \\
|
||
& LR & 76.04\% & 76.64\% & 80.83\% & 81.02\% & 80.77\% \\
|
||
& SVM & \textbf{81.27\%} & 81.26\% & 81.20\% & \textbf{81.56\%} & 81.40\% \\
|
||
\midrule
|
||
\multirow{4}{*}{Part III} & NB & 69.32\% & 71.20\% & 76.80\% & 76.85\% & 76.19\% \\
|
||
& RF & 75.29\% & 75.33\% & 75.30\% & 75.29\% & 75.29\% \\
|
||
& LR & 74.81\% & 75.51\% & 77.81\% & 78.35\% & 78.02\% \\
|
||
& SVM & \textbf{78.23\%} & 78.26\% & 77.83\% & \textbf{78.48\%} & 78.15\% \\
|
||
\midrule
|
||
\multirow{4}{*}{Part IV} & NB & 63.51\% & 66.76\% & 73.59\% & 73.45\% & 72.94\% \\
|
||
& RF & 70.32\% & 70.45\% & 70.46\% & 70.32\% & 70.32\% \\
|
||
& LR & 68.62\% & 69.89\% & 74.66\% & 74.22\% & 74.29\% \\
|
||
& SVM & \textbf{73.81\%} & 73.84\% & 75.25\% & \textbf{75.31\%} & 75.01\% \\
|
||
\bottomrule
|
||
\end{tabular}
|
||
\end{table}
|
||
|
||
Table~\ref{tab:compare2stage} shows the performance of different text-based classifiers and 2-stage classifiers.
|
||
The 1-stage column shows the performance of different text-based classifiers, and 2-stage column shows the performance of 2-stage classifiers which use different classifiers in the second stage.
|
||
From the table, we can see most 2-stage classifiers outperform 1-stage classifiers, and 2-stage classifier \textit{$<SVM,LR>$} performs best among all 1-stage and 2-stage classifiers in 4 parts.
|
||
Compared with text-based classifiers, NB and LR used in first stage can benefit more from 2-stage classifier framework.
|
||
Part IV makes biggest improvements in four parts, which contains most confused issues.
|
||
And 2-stage classifiers have limited effect on the Part I, which contains least confused issues.
|
||
|
||
|
||
% shows that the combined method outperforms all other methods for 80 projects.
|
||
% % Table~\ref{table:detailresult} shows that the average values of precision, recall, and F-measure are all better than
|
||
% % those of the baseline methods.
|
||
% For procedure $\widetilde{\textbf{T}}$ (last rows in Table~\ref{tab:compareML}),
|
||
% all the \textit{p-values} of combined method versus SVM, developer information method and perplexity information method are less than 0.05,
|
||
% and the lower and upper boundaries are greater than zero,
|
||
% which means that combined method is more significant than other approaches.
|
||
|
||
\begin{framed}
|
||
\noindent
|
||
\textbf{Result 3:}
|
||
\textit{
|
||
The two-stage classification approach can achieve a improvement
|
||
compared to traditional text-based classification by integrating our novel perplexity features.
|
||
And the more confused issues exist in the dataset, the better performance the 2-stage classification can achieve.
|
||
%significantly improve the classification performances
|
||
%with the 1.1\% precision and 3.2\% recall on average.
|
||
%The approach that adds perplexity information extracted from free text of issues can significant improve the performance of classification model.
|
||
}
|
||
\end{framed}
|
||
|
||
Although the value of the absolute increase is not impressive,
|
||
the extracted features (semantic perplexity information) are generally effective in improving the performance of the classifier model.
|
||
The result of dividing sentences is not ideal.
|
||
Issue reports in these projects are contrasted sharply,
|
||
combining free text with codes, hyper-link, and stack track, thereby complicating the finetuning for every project.
|
||
The preprocessing is not perfect, which limits the promotion of some projects.
|
||
Even facing this challenge, our approach still achieves a stable improvement.
|
||
We believe that when applied in practice,
|
||
a highly individualized data preprocessing approach can be a great help in extracting features
|
||
that are in agreement with our approach.
|
||
|
||
\subsection{RQ4:Performances with mislabeled dataset}
|
||
|
||
% 折线图
|
||
|
||
% 介绍实验方法
|
||
% 分为4个part
|
||
% 随机转换%比的数据进行训练，然后看实验结果的变化
|
||
|
||
|
||
In this section, we mainly to detect SVM (the best performance of 4 different text-based classifiers) and our 2-stage classifier \textit{$<SVM,LR>$} (the best performance of 2-stage classifiers).
|
||
Since the number of confused issues in the training set will influence the performance of our 2-stage classification approach, we also divide the data set into 4 part as section~\ref{sec:2stage}.
|
||
The proportion of mislabeled issues ranges from 1\% to 30\%, increasing by 1\% each time.
|
||
And the results of performance influenced by mislabeled train set are shown in Figure~\ref{fig:robust}.
|
||
|
||
\begin{figure}
|
||
%\begin{tabular}{cc}
|
||
\begin{minipage}{0.48\linewidth}
|
||
\centerline{\includegraphics[width=6.5cm]{figure/l1}}
|
||
\centerline{(a) Part I}
|
||
\end{minipage}
|
||
\hfill
|
||
\begin{minipage}{0.48\linewidth}
|
||
\centerline{\includegraphics[width=6.5cm]{figure/l2}}
|
||
\centerline{(b) Part II}
|
||
\end{minipage}
|
||
\vfill
|
||
\begin{minipage}{0.48\linewidth}
|
||
\centerline{\includegraphics[width=6.5cm]{figure/l3}}
|
||
\centerline{(c) Part III}
|
||
\end{minipage}
|
||
\hfill
|
||
\begin{minipage}{0.48\linewidth}
|
||
\centerline{\includegraphics[width=6.5cm]{figure/l4}}
|
||
\centerline{(d) Part IV}
|
||
\end{minipage}
|
||
%\end{tabular}
|
||
\caption{Performance with mislabeled dataset}
|
||
\label{fig:robust}
|
||
\end{figure}
|
||
|
||
|
||
The performance of 2-stage classification approach is mainly influenced by 2 factors:
|
||
1) the proportion of confused issues and 2) the accuracy of classification results in first stage.
|
||
We have discussed how the proportion of confused issues influenced the performance of 2-stage classifier in Section~\ref{sec:2stage}————The more confused issues exist in the dataset, the better performance the 2-stage classification approach can achieve.
|
||
For factor 2, most of the features in the second stage is extracted from the first stage.
|
||
The error information introduced by first stage will be passed to the second stage.
|
||
too many mislabeled issues will accumulate the error information, which may decline the performance of 2-stage classifier.
|
||
|
||
The results in Figure~\ref{fig:robust} also confirmed our assumption about the performance of 2-stage classifier.
|
||
For Part I, there are few confused issues in data set, so the performance of 2-stage classifier declines quickly.
|
||
For Part II and Part III, 2-stage classifier is slightly better than SVM between 0 and 10\% noise data.
|
||
But with the increase of noise data, SVM outperform 2-stage classifier.
|
||
Part IV contains most confused issues, which benefits most from 2-stage classifier.
|
||
Therefore, the negative impact of mislabeled data is offset by positive influence earning from confused issues.
|
||
The result of Part IV in Figure~\ref{fig:robust} shows that 2-stage classifier is better than SVM between 0 and 15\% noise data.
|
||
With the increase of noise data, 2-stage classifier decline as SVM.
|
||
|
||
\begin{framed}
|
||
\noindent
|
||
\textbf{Result 4:}
|
||
\textit{
|
||
The performance of 2-stage classifier is influenced by 1)the number of confused issues and 2)the performance of first stage.
|
||
The anti-noise ability of 2-stage classifier is related to the number of confused issues in train set.
|
||
}
|
||
\end{framed} |