50 lines
3.6 KiB
TeX
50 lines
3.6 KiB
TeX
\subsection{Research Question}
|
|
A study on a large-scale dataset was performed to build a common and effective text-based classification approach for GitHub projects.
|
|
This paper focuses on classifying bug-prone and non-bug prone issue reports because of the dominance of bugs in ITS.
|
|
We expect to refer the achievement from former research on traditional ITS, so we ask:
|
|
% Thinking of the different between ITS of GitHub and traditional ITS, we propose the first research question:
|
|
|
|
\textbf{RQ1: \textit{Do traditional text-based classification approaches still work on the ITS of GitHub?
|
|
Which classifier performs best?}}
|
|
|
|
In prior research \cite{zhou2014combining,merten2016software},
|
|
combining textual and structured information (\eg priority, assignee, and so on)
|
|
is used routinely to outperform the classifier.
|
|
However, as previously discussed, the omission of structured information is serious in the ITS of GitHub.
|
|
The data that can be used to build a classifier are limited because of the structured information scarcity.
|
|
Thus, only textual summary and the historical data of submitters can be used for the majority of issues.
|
|
Facing this challenge, we expect to study some factors that may influence the performance of classifiers, so we ask:
|
|
|
|
\textbf{RQ2: \textit{What factors influence the performances of text-based classification approaches?}}
|
|
|
|
The factors that influence the performances of text-based classifiers,
|
|
which would guide us to extract more additional features from textual summary,
|
|
can be identified using the regression analysis.
|
|
In this paper, a two-stage classifier framework,
|
|
which can flexibly combine textual information and other types of features, was built.
|
|
To evaluate our approach, we ask:
|
|
|
|
\textbf{RQ3: \textit{How to improve the classification performances
|
|
by integrating different types of features,
|
|
especially for the semantic complexity metrics extracted from textual descriptions?}}
|
|
|
|
Since the noise and bias introduced by misclassification of issue reports is a common situation~\cite{herzig2013s,herzig2013predicting},
|
|
it is important to acknowledge the performances of text-based classification approaches facing the dataset with wrong labeled issue reports. So we ask:
|
|
|
|
\textbf{RQ4: \textit{How much does mislabeled dataset affect the performances of text-based classification approaches?}}
|
|
|
|
\subsection{Dataset}
|
|
|
|
The dataset from Yu~\cite{yu2015wait}, which is a comprehensive dataset to study the pull-based model,
|
|
involving 951,918 issues across 1,185 main-line projects in GitHub
|
|
(dump dated 10/11/2014 based on GHTorrent~\cite{gousios2013ghtorent,gousios2014lean}),
|
|
was used in this paper.
|
|
Data on title, content, labels, and contributors for each issue were obtained through GitHub API.
|
|
Projects need to contain a sufficient number of labeled issues for training and testing to test which supervised text-based classification performs best.
|
|
Otherwise, an appropriative number of bug and non-bug issues existing in the projects is required to avoid the influence of unbalanced dataset. Thus, the candidate projects from GitHub, which have at least 500 labeled issues and bug rate between 20\% and 80\% ,
|
|
were finally identified and can be used as the training and testing sets.
|
|
|
|
Compared to traditional ITS, the ITS in GitHub uses a labeling system to manage issues.
|
|
Category information from the user-defined label system must be extracted to obtain a pre-labeled training set from GitHub.
|
|
In this paper, we use the same dataset and approach used in work~\cite{fan2017road} to extract category of issues.
|
|
Finally, we get 80 projects with total 100814 labeled issues (bug and nonbug). |