171 lines
11 KiB
TeX
171 lines
11 KiB
TeX
\subsection{Data Collection}
|
|
\label{sec:dataset}
|
|
% In our previous work \cite{yu2015wait}, we have composed a comprehensive dataset to study the pull-based model, involving 951,918 issues across
|
|
The dataset from Yu~\cite{yu2015wait}, which is a comprehensive dataset to study the pull-based model,
|
|
involving 951,918 issues across 1,185 main-line projects in GitHub
|
|
(dump dated 10/11/2014 based on GHTorrent~\cite{gousios2013ghtorent,gousios2014lean}),
|
|
was used in this paper.
|
|
Data on title, content, labels, and contributors for each issue were obtained through GitHub API.
|
|
Projects need to contain a sufficient number of labeled issues for training and testing to test which supervised text-based classification performs best.
|
|
Otherwise, an appropriative number of bug and non-bug issues existing in the projects is required to avoid the influence of unbalanced dataset. Thus, the candidate projects from GitHub, which have at least 500 labeled issues and bug rate between 20\% and 80\%
|
|
(the labeling dataset process is presented in section~\ref{labeling process}),
|
|
were finally identified and can be used as the training and testing sets.
|
|
|
|
\subsection{Category Extraction}
|
|
\label{labeling process}
|
|
|
|
Compared to traditional ITS, the ITS in GitHub uses a labeling system to manage issues.
|
|
Category information from the user-defined label system must be extracted to obtain a pre-labeled training set from GitHub.
|
|
Our dataset has 7,793 different labels in 1,185 projects, and many projects use different tags (\ie labels),
|
|
such as “bug,” “type:bug,” “error,” and “defect” to express the same meaning and identify bug-related issues.
|
|
A qualitative study is presented in this paper to comprehend the use of tags as categories of issues in this section.
|
|
|
|
Many projects in GitHub provide additional information in tags.
|
|
For example, in project ``WordPress-Android'', issues are labeled as ``[type] bug'', ``[type] enhancement'', and so on.
|
|
Tags in these projects contain not only categories of issues but also categories of the tag itself.
|
|
The information is useful in knowing what labels are mostly used to express categories of issues.
|
|
|
|
A process is designed to aggregate these tags by fully utilizing the additional information.
|
|
First, all tags that act as those forms were selected, and their information were separated.
|
|
A 2D vector $<C, name>$ was used to represent these tags,
|
|
where $C$ is the category of the tag (such as ``type'', ``component'')
|
|
and $name$ is the main information of the tag (such as ``bug'', ``feature'').
|
|
Second, tags with similar $C$ items were grouped as $Group_C$.
|
|
Then, the similarities of the two groups were defined using Equation~\ref{equation:similarity}.
|
|
The groups whose similarity were greater than the threshold were merged.
|
|
|
|
\begin{equation}
|
|
similarity = \frac{{\left| {Group_{C_i}\bigcap {Group_{C_j}} } \right|}}{{\min \left( {\left| {Group{C_i}} \right|,\left| {Group{C_j}} \right|} \right)}}
|
|
, (i \neq j)
|
|
\label{equation:similarity}
|
|
\end{equation}
|
|
|
|
$Group_{C_i}$ is a set of tags with the same category $C_i$ and different $name$.
|
|
Finally, a structure tag information is obtained through the aforementioned process.
|
|
|
|
% \begin{table}[htbp]
|
|
% \centering
|
|
% \caption{Summary of Labels in GitHub}
|
|
% \begin{tabular}{|l|p{0.65\columnwidth}|} \hline
|
|
% \textbf{Type} & \textbf{Labels} \\ \hline
|
|
% bug & \begin{tabular}[c]{@{}l@{}}bug, defect, bug - functional, Bug - Views, \\ Bug: Framework, Issue Type: Bug, kind/bug, etc.\end{tabular} \\ \hline
|
|
% enhancement & \begin{tabular}[c]{@{}l@{}}Enhancement, Improvement, kind: enhancement, \\ t:enhancement, etc.\end{tabular} \\ \hline
|
|
% feature & \begin{tabular}[c]{@{}l@{}}feature,feature request, Feature - High Priority, \\ Feature/Http, feature:bootstrap, etc.\end{tabular} \\ \hline
|
|
% documentation & \begin{tabular}[c]{@{}l@{}}doc, documentation, type/docs, \\ Kind:Documentation, etc.\end{tabular} \\ \hline
|
|
% question & \begin{tabular}[c]{@{}l@{}}kind/question, type/question, etc.\end{tabular} \\ \hline
|
|
% other & \begin{tabular}[c]{@{}l@{}}task, test, support, design, refactor, etc.\end{tabular} \\ \hline
|
|
|
|
% \end{tabular}%
|
|
% \label{tag:datacollection}
|
|
% \end{table}%
|
|
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{Tags in GitHub}
|
|
\label{tag:datacollection}
|
|
\begin{tabular}{clcccc}
|
|
\toprule
|
|
Category & Label & Projects & Issues & Percent & Total \\
|
|
\midrule
|
|
\multirow{3}{*}{bug} & bug & 644 & 118,155 & 46.9\% & \multirow{3}{*}{52.2\%} \\
|
|
& defect & 15 & 7,604 & 3.0\% & \\
|
|
& type:bug & 13 & 5,684 & 2.3\% & \\
|
|
\midrule
|
|
\multirow{7}{*}{nonbug} & enhancement & 412 & 44,947 & 17.8\% & \multirow{7}{*}{38.6\%} \\
|
|
& feature & 199 & 14,795 & 5.9\% & \\
|
|
& question & 319 & 13,109 & 5.2\% & \\
|
|
& feature request & 93 & 6,976 & 2.8\% & \\
|
|
& documentation & 239 & 6,422 & 2.5\% & \\
|
|
& improvement & 45 & 5,592 & 2.2\% & \\
|
|
& docs & 122 & 5,510 & 2.2\% & \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
|
|
Through the prior process, we extract 149 tags, which can indicate the category of issues, as group ``$type$''.
|
|
Finally, we filter total 252,084 issues with tags in group group ``$type$''.
|
|
Table~\ref{tag:datacollection} shows the most used tags in group ``$type$'', and how many projects and issues they appear.
|
|
These tags were divided into bug-prone or nonbug-prone by manually distinguishing.
|
|
The most used tags are bug, enhancement, and feature, which were observed in 46.9\%, 17.8\%, and 5.9\% of the labeled issues, respectively.
|
|
The issues with other tags
|
|
These structure tags were used in this study to judge whether an issue is bug-prone or not.
|
|
|
|
The number of issues labeled with both categories is 3,869 in 386 projects, which can be ignored compared with 252,084 labeled issues.
|
|
|
|
|
|
|
|
% To get a overall perspective to label system, we aggregate all other labels according to the group information we aggregated before.
|
|
% In the result, we find the top 3 most used group of labels are type (\textit{i.e.}, bug, feature, enhancement), status (\textit{i.e.}, duplicate, wontfix, invalid) and adverb (\textit{i.e.}, high, critical, major, and this kind of labels are mostly used as priority and severity).
|
|
|
|
% To evaluate the coverage of these 3 kinds of labels, we select all labels that are used in more than 5 projects to filter labels with minority usage. We count the usage times of these 3 kinds of labels and calculate the usage rate for all filtered labels. Finally, these 3 groups of labels are used more than half, which achieve 58.7\%.
|
|
|
|
|
|
% Through aggregating all the labels, we finally get 113 labels used most in the category ``type''. From these labels, we distinguish them and select bug-like labels such as ``bug'', ``defect'', ``type:bug'', ``definition:bug'', etc. and select feature-like labels such as ``feature'', ``enhancement'', ``new feature'', ``feature request'', etc. Then, we label issues as bug or feature using labels we have distinguished, and these labeled issues will be our training data in following process.
|
|
|
|
% Because of the need for adequate classified issues for training model, we hand-pick 111 projects which have more than 500 labeled categories information issues. The detail information of these projects are shown in Table \ref{tag:datacollection}.
|
|
|
|
% \begin{table}[htbp]
|
|
% \centering
|
|
% \caption{Summary Statistics for Data Collection}
|
|
% \begin{tabular}{|c|c|c|} \hline
|
|
% \textbf{} & \textbf{Count} & \textbf{Mean} \\ \hline
|
|
% Projects & 111 & \\ \hline
|
|
% Issues & 356256 & 3209.5(per project) \\ \hline
|
|
% Labeled issues & 240754 & 2169.0(per project) \\ \hline
|
|
% Labels & 470965 & 1.96(per issue) \\ \hline
|
|
% \end{tabular}%
|
|
% \label{tag:datacollection}
|
|
% \end{table}%
|
|
|
|
% We also select 3 different projects (phpadmin, piwik and numpy) as objects of case study, which have enough labeled issues and different proportion of bug to feature. The detail information about them are exhibited in Table \ref{tag:casestudy}.
|
|
|
|
% \begin{table}[htbp]
|
|
% \centering
|
|
% \caption{Projects for Case Study}
|
|
% \begin{tabular}{|c|c|c|} \hline
|
|
% \textbf{Projects} & \textbf{Labeled Issues} & \textbf{Bug Proportion (\%)} \\ \hline
|
|
% phpadmin & 6766 & 0.75 \\ \hline
|
|
% piwik & 5389 & 0.66 \\ \hline
|
|
% numpy & 2511 & 0.86 \\ \hline
|
|
% \end{tabular}%
|
|
% \label{tag:casestudy}
|
|
% \end{table}%
|
|
|
|
\subsection{Preprocessing of Dataset}
|
|
|
|
Each issue, which can be labeled as ``bug'' or ``non-bug'',
|
|
is characterized by its title and description.
|
|
In this paper, issues labeled by the former process were selected to perform the following steps.
|
|
First, linguistic features extracted for the text-based classifier undergo standard processing,
|
|
i.e., lowercase, text filtering, stemming, and indexing~\cite{frakes1992information}.
|
|
All stop-words and common English terms, such as ``should'', ``might'', ``not'', were retained.
|
|
The importance of linguistic features for classifying issues was indicated in study~\cite{antoniol2008bug};
|
|
moreover, study~\cite{bissyande2013got} mentions that removing the default list of stop-words in common corpora might decrease the classification accuracy.
|
|
Otherwise, ``\`{}\`{}\`{}'' is used to distinguish the code information in issue reports, because markdown editor is used in GitHub.
|
|
|
|
Then, a vector space model was used to represent each issue as a weighted vector.
|
|
The issue is segmented into different terms (in this paper, a word means term)
|
|
in which each element in the vector of the issue is the weight of a term,
|
|
and the value stands for the importance of the term for the issue.
|
|
Term frequency-inverse document frequency (\textit{tf-idf}) is used to calculate weight.
|
|
Tf-idf is based on two assumptions:
|
|
First, the frequency of the appearance of a given word implies its importance to an issue.
|
|
Second, the frequency of the appearance of a word in several issues causes it to become less useful to distinguish among these issues.
|
|
% tf-idf can be calculated as Equations~(\ref{equation:tf})(\ref{equation:idf})(\ref{equation:tfidf}).
|
|
|
|
% \begin{equation}
|
|
% tf(t,i) = \frac{{{n_t}}}{{{N_i}}}
|
|
% \label{equation:tf}
|
|
% \end{equation}
|
|
% \begin{equation}
|
|
% idf(t) = \log \left( {\frac{{{N_I}}}{{\left| {i \in I:t \in i} \right|}}} \right)
|
|
% \label{equation:idf}
|
|
% \end{equation}
|
|
% \begin{equation}
|
|
% tf-idf(t,i) = tf(t,i) \times idf(t)
|
|
% \label{equation:tfidf}
|
|
% \end{equation}
|
|
|
|
% Where \textit{t} is a term, \textit{i} is the corpus of an issue, \textit{I} is the corpus of all issues in the given project, $n_t$ is the count of appearance for term \textit{t} in the issue, $N_i$ is the total number of terms in issue \textit{i} and $N_I$ is the total number of issues in the given project. |