issue_classify/introduction.tex

%!TEX root = sci.tex

Nowadays, more and more individual developers, research institutes and even industrial companies prefer to host their software projects
on social coding platforms, \eg GitHub~\footnote{\url{github.com}}, BitBucket~\footnote{\url{https://bitbucket.org}}
and GitLab~\footnote{\url{https://gitlab.com/}}, to attract external contributions~\cite{oreg2008exploring}
and enhance reputations~\cite{ma2013imprs} from developer communities on a global scale.
%, as well as contribute to excellent open source projects.
In order to support the whole life-cycle development and simplify the collaborative process for both experienced and inexperienced contributors, a series of lightweight tools are widely applied, \eg pull request for code-patch submission~\cite{gousios2014exploratory} and Issue Tracking System (ITS) for development task management~\cite{bissyande2013got}.
The lightweight tools lower the barriers to participate in and attract a large amount of developers to provide contributions.
% The prosperity of open source software (OSS) resulted in an increasing number of developers joining in and contributing to OSS communities.
% GitHub is one of the most popular social coding communities that attracts a large number of developers~\cite{gousios2015work} .
% As of April 2016\footnote{\url{https://en.wikipedia.org/wiki/GitHub}},
% over 14 million registered users are collaborating in GitHub,
% showing a great power in driving the OSS project forward.
% At the same time, it brings many problems to project manager.
For many contribution activities, reporting issues using issue tracking system (ITS)
is one of the most important activities in OSS communities ~\cite{jalbert2008automated}.
Section~\ref{ITS_T} describes that the GitHub-provided ITS is more lightweight to use today,
compared to the traditional ITS (e.g., Bugzilla).
A contributor is only required a short textual abstract, containing a title and an optional description,
to report a new issue in GitHub.
Therefore, this simplified process of reporting issues decreases the barrier to entry
and attracts more inexperienced external contributors.
According to our statistics, Ruby on Rails,
%\footnote{\url{https://github.com/rails/rails}},
one of the most active projects in GitHub, receives upwards of 700 new issues each month.
%there are 134 issue reports per month in average,
%and they receive a maximum of 723 issue reports a month.

\begin{figure}[!htbp]
\centering
\setlength{\fboxrule}{0.5pt}
% \setlength{\fboxsep}{1cm}
\fbox{\includegraphics[scale = 0.31]{figure/wrong_usage}}
\caption{Example of undesirable issue in ITS}
\label{figure:example}
\end{figure}

However, the extreme openness of ITS poses a serious challenge for the core team in project maintenance.
In large-scale projects, many undesirable and vague issue reports are submitted by external contributors
(\eg asking questions, as shown in Figure~\ref{figure:example})
because of their reluctance to spend adequate time to read and comprehend the contribution guidelines
(as shown in Figure~\ref{figure:rule}),
which provide details on reporting a high-quality issue and the kind of issue the project prefers.
Thus, issue categorization is a labor-intensive and time-consuming task for project managers.
Furthermore, the core team members have to provide quick responses and resolve the incoming issues in time
to sustain the passion of external contributors~\cite{huang2016effectiveness}.

In addition, some works~\cite{antoniol2008bug,herzig2013s,zhou2014combining}
found that OSS contributors often omit or use default value for some important information.
Most of automatic approach highly depends on these information, so lost information and wrong information is very harmful to build automatic model.
%~\footnote{The guide of Ruby on Rails highlights that please don't ask question in the ITS.}.
%and just submit issue report casually as shown in Figure \ref{figure:example}.
%Most of time, tasks like these are not directly helpful to the development of the project.
%which results in many problems for project managers.
%To better manage these activities, many projects have made some specification about how to use ITS, as shown in Figure \ref{figure:rule}.
%These docs show the steps of how to submit an issue report and specify what categories and formats of issue report they prefer.



%For one thing, the increasing number of contributors accelerate the exploration of bug and bring many issue reports,
%which adds to managers' task.
%Managers have to deal with these issue reports in time to keep passion of these contributors \cite{huang2016effectiveness}.
%And all that makes managers have to spare much time to handle these tasks.
%For another, the increasing number of contributors makes management task more complicated.
% In the process of project developing,
% in order to better manage ITS and improve the efficiency of cooperation with other developers,
% the core developers may make some specification or follow some potential habit about how to use ITS.
% Some projects write docs of rules for how to contribute to the project, as show in Figure \ref{figure:rule}.
%For non-core developers, they may not spend enough time to comprehend the specification of the project, and just submit issue report casually as shown in Figure \ref{figure:example}.
%Most of time, tasks like these are not directly helpful to the development of the project.
%which results in many problems for project managers.



\begin{figure}[!htbp]
\centering
\setlength{\fboxrule}{0.5pt}
% \setlength{\fboxsep}{1cm}
\fbox{\includegraphics[scale = 0.25]{figure/rule}}
\caption{Example of contribution guidelines in GitHub}
\label{figure:rule}
\end{figure}

Most issue management tasks are organized based on the label system in
GitHub\footnote{\url{https://help.github.com/articles/applying-labels-to-issues-and-pull-requests/}}
(as discussed in Section~\ref{ITS_T}).
One of the most popular practices is distinguishing different types~\cite{antoniol2008bug} of issues
(\eg bug, feature request, and refactoring), which is a manual process maintained by core developers.
Thus, the high performance of issue categorizing approach,
especially issues established in limited prior information
(\ie the majority of issues only have textual summary and historical data of submitters),
could significantly reduce the cost of issue management.


This paper focuses on the challenge of distinguishing real bugs from nonbugs among all issues,
similar to prior work~\cite{herzig2013s,zhou2014combining}.
Moreover, we study the impact of mislabeled dataset on building automatic classification model.
Although a fine-grained classification is deferred for future work,
we argue that this work can 1)~greatly improve the efficiency of issue management in GitHub
because well-established projects prefer to maintain limited types of issues,
especially for bug and feature
(\eg Ruby on Rails\footnote{\url{http://edgeguides.rubyonrails.org/contributing_to_ruby_on_rails.html}},
and angualr.js\footnote{\url{https://github.com/angular/angular/blob/master/CONTRIBUTING.md\#issue}});
2)~reduce the noise and bias~\cite{herzig2013s,herzig2013predicting}
introduced by confusing real bugs with other types of issue (\eg feature requests);
3)~how different proportions of mislabeled items in dataset influence efficiency of our approach,
when building bug prediction~\cite{d2010extensive,hata2012bug} or
software quality~\cite{vasilescu2015quality}
models based on mining the big data from GitHub.

%For the academic research based on mining the big data from GitHub,
%identifying real bugs accurately reduce the bias~\cite{herzig2013s,herzig2013predicting}
%for building bug prediction~\cite{d2010extensive,neuhaus2007predicting,hata2012bug} or
%software quality~\cite{kan2002metrics,vasilescu2015quality,yu2016initial} models.

In summary, the key contributions of this paper include:
\begin{itemize}
\item The study of text-based classification approaches on a large-scale dataset.
Four different machine learning classifiers were evaluated on 80 popular projects in GitHub.
The results show that the support vector machines (SVM) achieve the best performance.

% , which has best accuracy on average and performs most stably for different projects.

\item The limitations of text-based classification approaches were analyzed,
and the results showed that semantic perplexity
(\ie an issue's description confuses bug-related sentences with nonbug-related sentences)
is a crucial factor that affects classification performances.
Thus, representative metrics were designed to quantify the semantic perplexity of an issue report.

%In the process of manually analysis, we found that issue reports that are hard to classify always contain perplexing text.
%The result of regression analysis approve our assume that perplexity of free text can significant affect the performance of ML techniques.
% We conclude a scheme of issue reports through manually analyzing 589 issue reports. We find that issue reports whose description discusses both bug and feature are most likely classified incorrectly. The key to distinguish them is to analysis the structure of the description. For features, the feature-like sentences are more likely to appear in the begin or end of the description.

\item A novel two-stage classifying framework was designed to improve the performances of traditional classification models.
Features relating to semantic perplexity were extracted from free text in the first stage,
and then a synthesized classification model was built in the second stage.
The quantitative evaluations show that classification performance can achieve a significant improvement.

% TODO:
\item We compare the efficiency of our tow-stage classifying framework and traditional classification models, under the influence of different proportions of mislabeled items in dataset.
The experiment result shows that the higher semantic perplexity in dataset, the better anti-noise ability of our two-stage approach, compared with tradition classification approach.
%a significant improvement for 101 projects.
% Based on the scheme detected above, we improve the ML classification methods using heuristic method.
% We divide description into sentences and use classification model to classify these sentences,
% then select out issue reports which are most likely to be wrong, according to
% the classification results of individually sentences (see Section 4.3).
% Finally, we evaluate our heuristic method on 111 projects and achieve 7.85\% improvement on average.
\end{itemize}

The structure of this paper is organized as follows.
In Section 2, we introduce the background of our study, and illustrate related work.
In Section 3, we show our research questions and dataset of our work.
In section 4, we present the key process of building effective classification models.
The results and discussion can be found in Section 5.
Finally, we draw our conclusions in Section 6.
This paper is an extension of the paper~\cite{fan2017road} published in the 11th International Symposium on Empirical Software Engineering and Measurement (ESEM 2017).
Compared with the prior work, our new contributions are summarized as follows:

\begin{itemize}
\item We extend the 2-stage classification framework, and compare the performance of different traditional classification combinations in 2-stage framework.
In this process, we find the best performing classification approach combination for 2-stage framework.

\item We compare the anti-noise ability of our 2-stage approach and SVM by manually introducing mislabeled data.
At the same time, we also analysis the impact of semantic perplexity information on anti-noise ability.

\end{itemize}
%Moreover, there are many different categories of issue in ITS~\cite{antoniol2008bug}: information about bugs, requests of new feature or function, refactoring / restructuring activities, etc.
%Developers use labels to distinguish what category the issue report, but it is not mandatory when submit the issue.
%These loose restrictions make users in GitHub only concern what they are interesting in,
%which result in serious lack of categories information that are very useful.
%Repairing the missing categories of issue reports can be a very useful work, but it takes a lot of time and manpower to maintain it.
%So automatic classifier for issue reports seems necessary.


% Issue Tracking System (ITS) plays an important role on guiding the maintenance activities of software developers \cite{jalbert2008automated},
% which is widely used in Open Source Software (OSS) as well as in the industry software.
% With the help of ITS, project managers can clearly understand what happen on their projects
% (i.e., how many issues are reported by users daily; which module is associated with most bugs?)
% In general, there are many different kinds of issue in ITS~\cite{antoniol2008bug}: information about bugs, requests of new feature or function, refactoring / restructuring activities, etc. Users can label issue reports to distinguish categories of them, but most of issues missing the message of that. It takes a lot of time and manpower to maintain it and automatic classifier for issue seems necessary.


%Distinguishing the categories of issue reports also benefits for research works.
%For some bug prediction methods \cite{d2010extensive,neuhaus2007predicting}, they use bug reports information to map vulnerabilities to components, which help them to build prediction model.
%Thus, it is meaningful to accurately tell which issues are bug-prone and which are nonbug-prone.


% The largest social coding platform, such as GitHub, creates a convenient and comfortable environment for many software developers through integrating many developing tools and social media tools. At the same time, GitHub attracts many programming enthusiasts to contribute and stores much valuable historical data of these activity. However, users in GitHub only concern what they are interesting in, which result in serious lack of information that are very useful for a few people, such as the categories if a issue report. Hence, repairing the missing categories of issue reports can be a very useful work.

% Herzig et al~\cite{herzig2013s} illustrate six categories of issue reports from Bugzilla and Jira, i.e., \{BUG, RFE, IMPR, DOC, REFAC, OTHER\} and the list clearly distinguishes the task of different kinds of maintenance work.
% From these categories, issue report about BUG and RFE (feature) are most in GitHub \cite{bissyande2013got}. As for academic research, mining bug \cite{antoniol2008bug,herzig2013s,maalej2015bug,zhou2014combining} and feature \cite{fischer2003analyzing,maalej2015bug} are concerned most.
%In this paper, since a fine-grained classification of issue reports is beyond the scope of this paper, we are more interested in a binary classification, i.e., Bug and Non-Bug, similar to above prior work.
%
%The key contributions of this paper include:
%
%1) We study issue classification models on a large scale.
%We utilize 4 different MLTs on ITS of 101 popular projects in GitHub.
%From evaluation results, we find that SVM is significantly better than other 3 method.
%% , which has best accuracy on average and performs most stably for different projects.
%
%2) In the process of manually analysis, we found that issue reports that are hard to classify always contain perplexing text.
%The result of regression analysis approve our assume that perplexity of free text can significant affect the performance of ML techniques.
% We conclude a scheme of issue reports through manually analyzing 589 issue reports. We find that issue reports whose description discusses both bug and feature are most likely classified incorrectly. The key to distinguish them is to analysis the structure of the description. For features, the feature-like sentences are more likely to appear in the begin or end of the description.

%3) To improve the performance of classification model, We build a 2-level classifier framework, which takes advantage of what we found before.
%The 2-level classifier framework extract features about perplexity of free text in first level,
%and build a better performance classification model in second level.
%The analysis result shows a significant improvement for 101 projects.
%% Based on the scheme detected above, we improve the ML classification methods using heuristic method.
%% We divide description into sentences and use classification model to classify these sentences,
%% then select out issue reports which are most likely to be wrong, according to
%% the classification results of individually sentences (see Section 4.3).
%% Finally, we evaluate our heuristic method on 111 projects and achieve 7.85\% improvement on average.
%
%The structure of this paper is as follows. In Section 2, we introduce background of our study, and illustrates related work and some research questions. In section 3, we present some key process of building effective classification model. Result and discussion can be found in Section 4. Finally, we draw our conclusions in Section 5.

% As for academic research,
% many research have studied about bug prediction . The task of them is to predict whether the issue is talking about a bug. The key problem they meet is reducing wrong information users provide about categories of issue reports. Most of them label issues manually and it limits scope of projects that they can use to test their method.

% The categories of issue is not limited to bug. Besides recording bug report, feature request is another kind of most used issue reports. It is meaningful to distinguish feature from other categories because of a sufficient number of issue reports about feature and useful information they provide. Similar approach  are proposed to mining features from app store. And research \cite{fischer2003analyzing} studied relationship between features through bug report data.