qiangge
/
issue_classify
0
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
issue_classify/data_process.tex

439 lines
27 KiB
TeX
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

% \subsection{Category Extraction}
% \label{labeling process}
% Different with traditional ITS, the ITS in GitHub use label system to manage issues.
% %category and other structured information.
% In order to get a pre-labeled training set from GitHub, we need to extract category information
% from the user-defined label system.
% %However, the custom label system makes it difficult.
% In our dataset, there are 7,793 different labels in 1,185 projects
% (the details of our dataset are presented in Section~\ref{sec:dataset}),
% and many projects use different tags (\ie labels) to express same meanings.
% For example, the tags like ``bug'', ``type:bug'', ``error'' and ``defect'' are used to identify bug-related issues.
% In this paper, we present a qualitative study to comprehend how the core team use tags as categories of issues.
% % The ITS of GitHub use label system to
% % labeling process aims to build a pre-labeled training set,
% % which can be used for supervised text-based classification approaches next.
% % The first task in this process is to know what labels are used to distinguish Bug-prone issue reports from Nonbug-prone ones.
% % Compared with traditional ITS, it is more difficult to get pre-labeled training set in GitHub.
% % Because in the ITS of GitHub, it only provides labels for contributors to add extra structured information like categories, priority, etc.
% % And label system in GitHub is user self-defined, which means different project may use different labels to express same meaning.
% % In our dataset, there are 7793 different labels in 101 projects.
% % This flatten and flexible design makes it difficult to understand label usage from so many projects.
% % % On the one hand, different with Bugzilla, the ITS in Github do not have categories information for issue reports.
% % % Users in GitHub distinguish category of issue reports by labels.
% % % On the other hand, GitHub's label system are user self-defined and not all the projects use the same labels to label bug or feature for issue reports.
% % So we need comprehend which labels are used to distinguish issues first.
% In GitHub, there are many projects
% % There are some projects in GitHub migrated from other platform, and at the same time,
% % succeed to the custom of using traditional ITS,
% giving some extra information in tags. For example, in project ``WordPress-Android'', issues are labeled like ``[type] bug'', ``[type] enhancement'', etc.; in project ``angular.js'', issues are labeled like ``type: bug'', ``type: feature'', ``component: form'', etc.
% Tags in these projects not only contain categories of issues, but also contain the categories of tag itself.
% % Despite projects like those are just a small part,
% These information is great helpful for us to know what labels are used most to express categories of issues.
% We design a process to aggregate these tags by making full use of this extra information.
% Firstly, we pick out all tags acting as those forms, and separate the information of them.
% We use a 2-d vector $<C, name>$ to represent these tags,
% where $C$ means category of the tag (like ``type'', ``component'', etc)
% and $name$ means the main information of the tag (like ``bug'', ``feature'', ``enhancement'', etc).
% Secondly, we group tags with same $C$ items as $Group_C$, and the preliminary aggregating process is done.
% Next, we define the similarity of two groups through Equation~\ref{equation:similarity}.
% We iteratively calculate the similarity of different group, and merge the groups whose similarity greater than threshold.
% \begin{equation}
% similarity = \frac{{\left| {Group_{C_i}\bigcap {Group_{C_j}} } \right|}}{{\min \left( {\left| {Group{C_i}} \right|,\left| {Group{C_j}} \right|} \right)}}
% , (i \neq j)
% \label{equation:similarity}
% \end{equation}
% Where $Group_{C_i}$ is a set of tags with the same category $C_i$ and different $name$.
% Finally, we get a structure tag information through above process.
% % \begin{table}[htbp]
% % \centering
% % \caption{Summary of Labels in GitHub}
% % \begin{tabular}{|l|p{0.65\columnwidth}|} \hline
% % \textbf{Type} & \textbf{Labels} \\ \hline
% % bug & \begin{tabular}[c]{@{}l@{}}bug, defect, bug - functional, Bug - Views, \\ Bug: Framework, Issue Type: Bug, kind/bug, etc.\end{tabular} \\ \hline
% % enhancement & \begin{tabular}[c]{@{}l@{}}Enhancement, Improvement, kind: enhancement, \\ t:enhancement, etc.\end{tabular} \\ \hline
% % feature & \begin{tabular}[c]{@{}l@{}}feature,feature request, Feature - High Priority, \\ Feature/Http, feature:bootstrap, etc.\end{tabular} \\ \hline
% % documentation & \begin{tabular}[c]{@{}l@{}}doc, documentation, type/docs, \\ Kind:Documentation, etc.\end{tabular} \\ \hline
% % question & \begin{tabular}[c]{@{}l@{}}kind/question, type/question, etc.\end{tabular} \\ \hline
% % other & \begin{tabular}[c]{@{}l@{}}task, test, support, design, refactor, etc.\end{tabular} \\ \hline
% % \end{tabular}%
% % \label{tag:datacollection}
% % \end{table}%
% \begin{table}[htbp]
% \centering
% \caption{My caption}
% \label{tag:datacollection}
% \begin{tabular}{lccc}
% \toprule
% label & projects & issues & percent (\%) \\
% \midrule
% bug & 644 & 118,155 & 46.9 \\
% enhancement & 412 & 44,947 & 17.8 \\
% feature & 199 & 14,795 & 5.9 \\
% question & 319 & 13,109 & 5.2 \\
% defect & 15 & 7,604 & 3.0 \\
% feature request & 93 & 6,976 & 2.8 \\
% documentation & 239 & 6,422 & 2.5 \\
% type:bug & 13 & 5,684 & 2.3 \\
% improvement & 45 & 5,592 & 2.2 \\
% docs & 122 & 5,510 & 2.2 \\
% \bottomrule
% \end{tabular}
% \end{table}
% Through distinguishing \textit{name} of group ``$tpye$'' manually,
% we divide tags into 6 different categories as shown in Table~\ref{tag:datacollection}.
% Bug, enhancement and feature are most used tags, which are observed in 46.9\%, 17.8\% and 5.9\% of the labeled issues respectively.
% Here we use these structure tag information to judge whether an issue is bug-prone or not.
% The number of issues labeled with both categories is 3,869 in 386 projects, which can be ignored compared with 252,084 labeled issues.
% % To get a overall perspective to label system, we aggregate all other labels according to the group information we aggregated before.
% % In the result, we find the top 3 most used group of labels are type (\textit{i.e.}, bug, feature, enhancement), status (\textit{i.e.}, duplicate, wontfix, invalid) and adverb (\textit{i.e.}, high, critical, major, and this kind of labels are mostly used as priority and severity).
% % To evaluate the coverage of these 3 kinds of labels, we select all labels that are used in more than 5 projects to filter labels with minority usage. We count the usage times of these 3 kinds of labels and calculate the usage rate for all filtered labels. Finally, these 3 groups of labels are used more than half, which achieve 58.7\%.
% % Through aggregating all the labels, we finally get 113 labels used most in the category ``type''. From these labels, we distinguish them and select bug-like labels such as ``bug'', ``defect'', ``type:bug'', ``definition:bug'', etc. and select feature-like labels such as ``feature'', ``enhancement'', ``new feature'', ``feature request'', etc. Then, we label issues as bug or feature using labels we have distinguished, and these labeled issues will be our training data in following process.
% % Because of the need for adequate classified issues for training model, we hand-pick 111 projects which have more than 500 labeled categories information issues. The detail information of these projects are shown in Table \ref{tag:datacollection}.
% % \begin{table}[htbp]
% % \centering
% % \caption{Summary Statistics for Data Collection}
% % \begin{tabular}{|c|c|c|} \hline
% % \textbf{} & \textbf{Count} & \textbf{Mean} \\ \hline
% % Projects & 111 & \\ \hline
% % Issues & 356256 & 3209.5(per project) \\ \hline
% % Labeled issues & 240754 & 2169.0(per project) \\ \hline
% % Labels & 470965 & 1.96(per issue) \\ \hline
% % \end{tabular}%
% % \label{tag:datacollection}
% % \end{table}%
% % We also select 3 different projects (phpadmin, piwik and numpy) as objects of case study, which have enough labeled issues and different proportion of bug to feature. The detail information about them are exhibited in Table \ref{tag:casestudy}.
% % \begin{table}[htbp]
% % \centering
% % \caption{Projects for Case Study}
% % \begin{tabular}{|c|c|c|} \hline
% % \textbf{Projects} & \textbf{Labeled Issues} & \textbf{Bug Proportion (\%)} \\ \hline
% % phpadmin & 6766 & 0.75 \\ \hline
% % piwik & 5389 & 0.66 \\ \hline
% % numpy & 2511 & 0.86 \\ \hline
% % \end{tabular}%
% % \label{tag:casestudy}
% % \end{table}%
% \subsection{Preprocessing of Dataset}
% Through the former process, each issue is characterized by its title and description,
% and part of them can be labeled as ``bug'' or ``non-bug''.
% Here, we select issues labeled by former process to do the following steps.
% First, linguistic features extracted for the text-based classifier undergo the standard processing,
% i.e., lowercase, text filtering, stemming, and indexing~\cite{frakes1992information}.
% We do not remove all stop-words, and leave common English term, such as ``should'', ``might'', ``not''.
% In study~\cite{antoniol2008bug}, they indicate that it may be important for classifying issues,
% and study~\cite{bissyande2013got} also mentions that removing default list of stop-words in common corpora might decrease the classification accuracy.
% For instance, the semantic of a sentence ``This is not a bug'' is completely lost if the Standard English stop-words are removed because the result is ``This is a bug''.
% Then we use vector space model to represent each issue as a weighted vector.
% We segment the issue into different terms (in here a word means a term) and each element in the vector of the issue is the weight of a term, and the value stands for the importance of the term for the issue.
% We utilize term frequency-inverse document frequency (\textit{tf-idf}) to calculate weight, which based on two assumptions: The more a given word appears in the issue, the more important it is for that issue.
% Contrariwise, the more issues a word appears in, the less useful it is to distinguish among these issues.
% % is utilized to indicate the weight of a term.
% The process of calculating \textit{tf-idf} acts as Equation~(\ref{equation:tf})(\ref{equation:idf})(\ref{equation:tfidf}).
% \begin{equation}
% tf(t,i) = \frac{{{n_t}}}{{{N_i}}}
% \label{equation:tf}
% \end{equation}
% \begin{equation}
% idf(t) = \log \left( {\frac{{{N_I}}}{{\left| {i \in I:t \in i} \right|}}} \right)
% \label{equation:idf}
% \end{equation}
% \begin{equation}
% tf-idf(t,i) = tf(t,i) \times idf(t)
% \label{equation:tfidf}
% \end{equation}
% Where \textit{t} is a term, \textit{i} is the corpus of an issue, \textit{I} is the corpus of all issues in the given project, $n_t$ is the count of appearance for term \textit{t} in the issue, $N_i$ is the total number of terms in issue \textit{i} and $N_I$ is the total number of issues in the given project.
\subsection{Text-based Classification}
\label{ML}
Facing the huge changes described in Section~\ref{ITS_T},
determining whether the regular pattern of free text found in research~\cite{antoniol2008bug} still works,
and whether the performance of the text-based classification model in dealing with large-scale projects is efficient, are required.
Many text-based classifications are used to classify issues in different
studies~\cite{antoniol2008bug,herzig2013s,zhou2014combining}.
In this paper, various types of widely used text-based classifications,
such as \textit{Naive Bayes}, \textit{Logistic Regression},
were selected to know which classifier performs best.
Table~\ref{tag:packages} shows the selected text-based classifications and parameter settings,
which are determined by the best performance of numerous tests.
\begin{table}[htbp]
\centering
\caption{Text-based classifications and parameters setting}
\begin{tabular}{|c|l|l|} \hline
\textbf{Classifier} & \textbf{API} & \textbf{Parameters Setting} \\ \hline
SVM & SVC & kernel=`linear' \\ \hline
NB & MultinomialNB & class\_prior=`None' \\ \hline
LR & LogisticRegression & penalty=`l2' \\ \hline
RF & RandomForestClassifier & n\_estimators=100, n\_jobs=-1 \\ \hline
\end{tabular}%
\label{tag:packages}
\end{table}%
A well-labeled dataset, which can be used to train the classification model is constructed by dataset labeling process and preprocessing.
Table~\ref{tag:packages} shows the four classifiers built for each project.
We use APIs of package \textit{sklearn} to implement the methods.
A ten-fold cross-validation was applied to separate dataset samples into training and testing sets to evaluate the classification model.
% The ten-fold cross-validation randomly partitions the dataset into 10 equal-sized subsets.
% One subset is retained as the testing set for evaluating the classifier out of the 10 subsets,
% and the remaining 9 subsets were used as the training set to build the classifier.
The ten-fold cross-validation has a minimal effect on the sample characteristics
and can investigate the stability of item loading on multiple factors.
% ~\cite{van2006five}.
% The advantage of this validation method over repeated random sub-sampling is that all observations are used for both training and validation, and each observation is used for validation exactly once .
\subsection{Regression Analysis of Classification Performance}
\label{sec:regression}
The classification performances of different projects are always various.
This part aims to determine the factors that influenced the performance of text-based classification.
Manual analysis was used to detect factors, and we discovered several misclassified issues containing both bug- and nonbug-prone parts.
For example, contributors may discover some unreasonable designs or problems in the project
when they submit an issue about feature request (\eg EX in Section \ref{section:improvingmodel}).
The issue EX is nonbug prone, but the part of problem description is bug prone,
which may confuse the classification.
Such issues are termed as \textit{\textbf{confused issues}}.
Regression analysis techniques were used to verify our perception.
In this paper, multiple linear mixed effect models were used to investigate the factors that affect classifier performance.
In addition to coefficients, the effect size of each variable obtained from ANOVA analyses was reported.
The model's fit can be evaluated by pseudo R-squared, i.e., the marginal (${R_m}^2$) and conditional (${R_c}^2$) coefficient,
to determine generalized mixed-effect models~\cite{tsay2014let}.
As implemented in the MuMIn package of R~\cite{jiang2004exploration},
(${R_m}^2$) is the proportion of variance explained by the fixed effects alone,
and (${R_c}^2$) is the proportion of variance explained by the fixed and random effects.
All numeric variables were first log transformed (plus 0.5 if necessary) to stabilize variance
and reduce heteroscedasticity~\cite{gharehyazie2014developer}.
The variance inflation factors (VIFs) for each predictor were computed to test for multicollinearity.
If the VIFs of all the remaining factors are below 3, then multicollinearity is absent~\cite{gharehyazie2014developer}.
\textit{1) Outcome:} The outcome measure is \textit{average F-measure} (calculated as Equation~\ref{equation:performance}) of the classification.
Because of ten-fold cross-validation that we use,
we can obtain $10*n_{project}$ records for regression analysis,
where $n_{project}$ is the number of projects.
\textit{2) Predictors:} Project- and issue-level measures were computed in this process.
Project-level measures are features of the status of the project,
whereas issue-level measures are features extracted from textual summary of issues.
\textbf{\underline{\textit{Project-level measures}}}
\textbf{\textit{Star and watch:}} The number of stars and watches of the project.
This can reflect the popularity of the project in GitHub.
\textbf{\textit{Contributors:}} The number of developers active in the project.
The data are acquired in the homepage of the project in GitHub.
\textbf{\textit{Project age:}} The project duration from the creation, in timestamps.
\textbf{\textit{Commits:}} The total number of commits of the project.
\textbf{\textit{Issues:}} The number of issues used in training set.
\textbf{\underline{\textit{Issue-level measures}}}
\textbf{\textit{Confused issues:}} The total number of confused issues.
Each sentence of the issue is predicted using the best model built in Section~\ref{ML}.
if not all sentences of the issue are predicted to the same part, the issue will be considered as a confused issue.
\textbf{\textit{Median of words:}} The median number of words for each instance in the training set.
More words are likely to contain more information, which may help for classifier.
\subsection{Tow-stage Classification}
\label{section:improvingmodel}
Omitting structured information in GitHub, as described in section~\ref{ITS_T},
results in that less information can be used to build a synthesized classification model.
The firsthand features can be extracted, except for free text of issues,
are relating to the historical information about issue contributors,
\eg contributor's identity (core or external developer) and historical developing activities.
Thus, we propose a two-stage classification approach to combine textual summary information
and developer information, which could be expected to improve the performance of classification.
Each stage of our approach is explained in the following paragraphs,
and the overview of our approach is shown in Figure~\ref{figure:framework}.
% which consists of the following steps:}}
% 1) The first stage uses textual summary information, including title and description of issues,
% to build a supervised text-based classification model.
% At this stage, the issue is not directly predicted as bug-prone or not, but the probability of being bug-prone occurs.
% % In this paper, linear SVM was used as our classification model in the first stage.
% Through this process, the semantic perplexity information of each issue was collected.
% All these outputs in the first stage are regarded as features extracted from unstructured free texts,
% which will be used in the second stage.
% 2) The second stage uses the structured information of contributors along with the features extracted from the first stage.
% At this stage, these features were fed into a new machine leaner, which would classify the issue as either bug-prone or non-bug-prone.
% % In the second stage, logistic regression was used as the classification model.
% Each stage of our approach is explained in the following paragraphs,
% and the overview of our approach is shown in Figure~\ref{figure:framework}.
\begin{figure*}[!htbp]
\centering
\setlength{\fboxrule}{0.5pt}
% \setlength{\fboxsep}{1cm}
\fbox{\includegraphics[width = 11.5cm]{figure/framework}}
\caption{Overview of two-stage classification framework}
\label{figure:framework}
\end{figure*}
\subsubsection{Stage 1 - Textual Summary Classification}
In this stage, the main task is to extract the information in the free text.
Similar to the process of Section~\ref{ML}, two main textual information sources,
title and description, were used.
% But in this time, we don't directly build a model to predict whether the issue report is bug-prone.
The classification model was trained from the textual information of training set,
and the probability output of the model was applied to predict the testing set.
After that, the title and description of the issue were divided into sentences,
and the classification model we build before was used to predict each sentence.
The sentence prediction results explain the changes in semantic when contributors report an issue.
% and extract semantic perplexity information contained in the free text.
Free text can be examined further by analyzing the semantic perplexity information of the sentence
and the regular pattern in submitting issues.
The example will explain semantic perplexity.
\textbf{EX:} ``\textit{Currently, auto-archiving cannot be used if Piwik's authentication is configured to use the CAS plugin. I ran into this problem with authentication when running archive.php with CAS plugin enabled on my site... Add a feature to auto-archiving, so that it can succeeds when Piwik uses CAS for authentication instead of the default Login module.}''
Based on the EX, the first sentence describes the problem encountered by the contributor.
For the classification model, this sentence is more likely to be predicted as bug-prone.
However, a new feature is proposed in the last sentence, which is more likely to be predicted as non-bug-prone.
For the classification model, issues similar to EX are difficult to classify because of the perplexity of the text.
This situation can be addressed by extracting the features and dividing the issue into sentences.
In Stage 1, the following features from the textual summary were extracted:
\underline{\textbf{Probability:}} The probability that the issue report is predicted as bug-prone.
The probability output of the classification model is used to obtain this feature.
\underline{\textbf{SentenceCount:}} The total number of sentences in the issue report, including title and description.
\underline{\textbf{MostBugProb:}} The maximum probability of all sentences that are predicted as bug-prone.
\underline{\textbf{MostNonbugProb:}} The maximum probability of all sentences that are predicted as nonbug-prone.
\underline{\textbf{Location:}} The sequence number of the most non-bug sentence.
This feature was used to show where the nonbug sentence appears in the issue report.
\underline{\textbf{BugCount:}} The number of sentences that are predicted as bug-prone.
\underline{\textbf{NonbugCount:}} The number of sentences that are predicted as non-bug-prone.
\underline{\textbf{ChangeCount:}} The number of semantic changes.
For sentences sequence, every time the prone of sentences change from bug-prone to nonbug-prone or vice versa, the semantic changes will add one.
\underline{\textbf{Perplexity:}} The perplexity of the issue.
For sentence sequences of the issue, a series of probabilities that are predicted as bug-prone were collected
and their perplexity were calculated using Equation~\ref{equation:perplexity},
which is borrowed from the perplexity of natural language processing (NLP).
\begin{equation}
Perplexity = \frac {1}{SentenceCount} \sum \log (p_{i+1}-p_i)
\label{equation:perplexity}
\end{equation}
Where $p_i$ is the probability of the i-th sentence.
These features need to be carefully extracted because of using the ten-fold cross-validation.
All these features are assumed to be produced by a training set,
which means that the training set was used to build a prediction model,
and use this model to extract features of each instance in the training set.
This approach allows the extraction of features without using the labeled information of a testing set,
which may introduce some additional information and produce less-scientific results.
\subsubsection{Stage 2 - Combining Free Text and Developer Information Classification}
\label{section:secondlevel}
In Stage 1, the probability of bug-prone and perplexity information of sentences for each issue were obtained from free text.
These features will be part of the input of Stage 2.
The experience of developers may influence the categories of issues.
For example, skilled developers are likely to report a bug-prone issue
and provide issue reports that meet the specification of the core team.
Thus, in Stage 2, some structured features about contributors who submit issue reports were provided.
These features contain identity of contributors in the project,
historical developing activities, and social influence.
The detailed information are as follows:
% 要写考虑这些feature的直觉，可以一个总结下面写几个feature
\underline{\textbf{IsCoreTeam:}} This feature shows whether the contributor who reported the issue is a core team member in the project.
If the contributor is in the core team, this feature is set to 1, otherwise, 0.
\underline{\textbf{IssueCountInProject:}} The number of issues the contributor reports before in the project.
\underline{\textbf{IssueCountInGitHub:}} The number of issues the contributor reports to GitHub.
\underline{\textbf{CommentCountInProject:}} The number of comments that the contributor commits in the project.
\underline{\textbf{CommentCountInGitHub:}} The number of comments the contributor commits in GitHub.
\underline{\textbf{FollowerCount:}} The number of followers that the contributor acquires.
This feature can reveal the social influence of the contributor in GitHub.
\underline{\textbf{RegisterTime:}} This feature shows the duration that the contributor has registered.
The longer a contributor has registered, the more familiar he is with the principles of GitHub.
% To complete the classification process in Stage 2,
% data grafting process is required to combine features from free text and structured contributor information features.
% During Stage 1, the ID of report is not included, only input is the instances with lists of preprocessed terms,
% as well as the corresponding labels.
% A special subprocess called data grafting was used to smooth the linkage of the two stages.
% Data grafting aims to melt datasets from various sources, and combines features into a regular form according to its source.
% Owing to the use of a ten-fold strategy to partition the training set,
% following the change of dataset is required.
% Thus, the same partition process as training set for ID information, which enables tracing every instance, is performed.
% Logistic regression was used as our prediction model in Stage 2.
In Stage 2, it needs to be careful in partitioning datasets into training and testing sets before a prediction model is built.
The output of Stage 1 and input of Stage 2 are associated.
The testing set of Stages 1 and 2 should be similar to ensure that the same training set will be used to build the model,
and to avoid the introduction of extra information from the testing set.
\subsection{Anti-noise text-based classifier}
Mislabeled issue reports are common in open source community~\cite{herzig2013s,antoniol2008bug}.
That is, reports classified as bugs, but actually referring to non-bug issues.
This mislabeled issue reports in train set may bias the performance of text-based classifiers.
So it is very important to analyze the anti-noise ability of the classifiers.
In this section, we will introduce the experimental process of detecting the anti-noise ability of the classifiers.
A ten-fold cross-validation is also applied to separate dataset for each projects.
We randomly mislabel part of issue reports to build train set.
The number of mislabeled issues ranges from 1\% to 30\%, increasing by 1\% each time.
The test set use raw data, not to mislabel.
And for each classification approach to be detected, we train in those train set with mislabeled data and test in test set without mislabeled data.
Finally, we evaluate the anti-noise ability of the classifier by the variation of the performance under different proportions of mislabeled date set.
Reference in New Issue View Git Blame Copy Permalink