180 lines
13 KiB
TeX
180 lines
13 KiB
TeX
\subsection{Research Questions}
|
|
|
|
Static analysis tools have been widely used by state of the art projects~\cite{beller2016analyzing}.
|
|
However, these are still many problems to utilize these tools in practice.
|
|
Some studies~\cite{hanam2014finding,beller2016analyzing} have worked on improving the static analysis tools by analyzing developers' actions on alerts from tools.
|
|
It is a good approach to evaluate the effect of rules from these tools by detecting whether developers fix the alerts, but in this process, we ignore the purpose of developer.
|
|
In this study, we propose an approach based on SZZ~\cite{sliwerski2005changes} to identify three types of code commits: bug-fixing commits, buggy commits and ``non-buggy'' commits.
|
|
We combine the fixing alerts process with fixing bug process and research the relationship of these alerts with bugs.
|
|
So we ask:
|
|
|
|
\noindent \textbf{RQ 1} What are the differences in scanning results of SonarQube for buggy commits and ``non-buggy'' commits?
|
|
|
|
RQ 1.1 Do the alerts from SonarQube Java rules distribute differently between buggy commits and ``non-buggy'' commits?
|
|
|
|
RQ 1.2 Are all rules of SonarQube for Java significantly related to buggy commits?
|
|
|
|
Developers modify the buggy code and commit patches to fix bugs.
|
|
When fixing bugs, the codes from parches may also fix the alerts when they are associate with the bugs,
|
|
which means developers' actions on alerts when fixing bugs are useful for evaluating the effects of the rules.
|
|
Hence, in our second research question, we ask:
|
|
|
|
\noindent \textbf{RQ 2} What do developers act on the alerts when fixing bugs.
|
|
|
|
% 识别buggy commit and bug-fix commit, normal commit
|
|
\subsection{Classifying Commits}
|
|
\label{sec:classify}
|
|
\begin{figure*}[!htbp]
|
|
\centering
|
|
\includegraphics[scale = 0.45]{figure/szz}
|
|
\caption{Classifying commits process}
|
|
\label{fig:classify}
|
|
\end{figure*}
|
|
|
|
In this process, we selected three types of commits from project repository, which were bug-fixing commits, buggy commits and ``non-buggy'' commits.
|
|
In this paper,
|
|
bug-fixing commits are code commits which fix bugs or defects of projects;
|
|
buggy commits are code commits inducing bugs or defects;
|
|
``non-buggy'' commits are code commits without bugs and defects.
|
|
We adapted original SZZ algorithm~\cite{sliwerski2005changes} to identify the buggy commits and bug-fixing commits.
|
|
We identified bug-fixing commits by utilizing text pattern to recognize issue-link in commit message.
|
|
Then we used git-diff and git-blame to identify buggy commits, which induced bug code to the repository.
|
|
Finally, we identified ``non-buggy'' commits by filtering out commits related to bug and bug-fixing process.
|
|
We illustrate the process with the example in Figure~\ref{fig:classify}.
|
|
The detail of each steps are as described below.
|
|
|
|
\textbf{Bug-fixing commits:}
|
|
First, we filtered out the commits which may be associated with issues in ITS.
|
|
GitHub provided a mechanism to link code commits with issues, which was referring to \textit{issue\_id} in code commit log.
|
|
So we extracted the numerical \textit{issue\_id} by searching for specific textual patterns~\cite{mockus2000identifying,sliwerski2005changes,rahman2011ownership} in code commit logs (e.g., \#issue\_id, fix issue\_id, close issue\_id, etc).
|
|
Then we checked the linked issues to confirm whether the issues existed in ITS and whether they were bug issues.
|
|
% todo: 有疑问
|
|
% Next we compared the committed date of these commits with the closed date of corresponding issues, and removed the records that have a long interval between two date.
|
|
Next, we manually augmented linking rates to mitigate the risk of bias arising from missing and incorrect links~\cite{bachmann2010missing,bird2009fair,rahman2013sample}.
|
|
By reviewing commits and issues in projects with low linking rates, we found more specific textual patterns to automatically link commits and issues (e.g., gh-issue\_id).
|
|
After that, we updated our textual patterns, checked the links via random sampling, and removed spurious linking as much as possible.
|
|
On GitHub, the fix-inducing codes might be gotten by \textit{git push} or a pull-request (\textit{PR})~\cite{gousios2014exploratory}.
|
|
Developers usually need a series of code commits to fix an issue, and commits from \textit{git push} is hard to identify in this process.
|
|
So in this study, we only considered code commits from \textit{PR}.
|
|
For each \textit{PR}, we identified all the commits of \textit{PR} and the time it was merged.
|
|
The two timestamps were used (the merging time of the \textit{PR} and the open time of the corresponding issue) to filter out wrong links.
|
|
In the end of this process, we identified these commits as bug-fixing commits which are linked to at lease one issue.
|
|
|
|
\textbf{Buggy commits:}
|
|
First, we used git diff to identify change details of fixing bug, i.e., files and lines changed in bug-fixing commits.
|
|
Then, we filtered out the bug-fixing commits which may not fix a bug.
|
|
The issue in ITS might be tagged as a bug, feature, doc, etc.
|
|
Nonbug issues could engender bias~\cite{herzig2013s} in our dataset.
|
|
We had to identify the issues which report a bug to mitigate this risk.
|
|
To ensure that, we did flowing steps as work~\cite{yu2016initial}:
|
|
1) the issue has been tagged as a bug;
|
|
2) it was linked to a fixing commit;
|
|
3) developers did indeed change source files to fix it (filtered out commits which only changed doc, test files or configuration file).
|
|
Next, we used git blame command to identify the original commits (buggy commits) where the fix-inducing lines had been last added or modified.
|
|
Finally, we disregarded initial commits and refactoring commits because they do not really change the system behavior~\cite{neto2018impact}.
|
|
|
|
\textbf{``Non-buggy'' commits:}
|
|
Almost all software contain defects.
|
|
Some defects have been found while others have not been found.
|
|
We can accurately distinguish which commits have defects in code by records of development, but it is hard to affirm which commits do not have defects.
|
|
In this study, ``non-buggy'' commits are not commits have none bugs or defects.
|
|
They are commits that are hard to find relations with defects from code commit histories.
|
|
To identify ``non-buggy'' commits, we first filtered out all commits related with defects.
|
|
We mitigate this risk by ensuring that:
|
|
1) the commits were not involved in the process of identifying bug-fixing commits and buggy commits;
|
|
2) the commits message in logs did not contain words like ``fix/fixed'',``bug'',``defect'', etc.
|
|
Moreover, considering the changes of code style with the evolution of the projects, we preferred to select commits around the buggy commits as ``non-buggy'' commits.
|
|
The time similarity between buggy commits and ``non-buggy'' commits is helpful for mitigating the risk, which was risen by the changes of code style.
|
|
|
|
% PS: remove comments, remove refactors commits, meta commits
|
|
\subsection{Scanning Commits}
|
|
|
|
\begin{figure*}[!htbp]
|
|
\centering
|
|
\includegraphics[scale = 0.45]{figure/scan}
|
|
\caption{The process of scanning code commits}
|
|
\label{fig:scan}
|
|
\end{figure*}
|
|
|
|
% To achieve our goals, we first extracted the files of different code commits.
|
|
% Next we used SonarQube to scan these files and stored the reports of analysis.
|
|
% Then we compared the lines number of code changes and alerts of SonarQube reports, and selected the alerts generated for the code changes of the commits.
|
|
% Finally, we store the linked alerts and the response commits.
|
|
% The process of all steps are shown as Figure~\ref{fig:scan}.
|
|
|
|
In this process, we aimed to use SonarQube to scan the commits we selected in Section~\ref{sec:classify}.
|
|
We illustrate the process with the example in Figure~\ref{fig:scan}.
|
|
In the first phase, we propose an implementation of incremental scanning for each commit.
|
|
In our work, we used the version 6.5 of SonarQube.
|
|
This version of SonarQube support multi-thread scanner and it can reduce the scanning time for multiple scanning tasks.
|
|
However, this version of SonarQube does not support incremental scanning, which means we need to scan the whole project for each commit.
|
|
Even though the static analysis of SonarQube is faster compared with dynamic analysis tools, it is still a time-cost process to scan thousands of times for each project.
|
|
To achieve the incremental scanning, we first extracted all files that the commit has changed.
|
|
We used git diff as Section~\ref{sec:classify} to find path and name of the files that the commit has changed.
|
|
Next we used git checkout sha to go back the version of the commit and extracted all these files into one folder with relative path of each file.
|
|
Then we treated these files as one ``project'' and added a configuration file for them.
|
|
The sha of the commit was used as the name of the ``project'', which was useful to search and record information later.
|
|
Finally, we used SonarQube to scan the ``project'' and record all information of the analysis records, including ``project'' name, file name, rule of alert and correspond lines, etc.
|
|
|
|
In the section phase, the lines changed in the commit and the lines where generated the alert were used to link the alert and commit.
|
|
In the first phase we extract all files changed in the commit as a ``project'' and scanned the ``project'' to generate alerts.
|
|
Some of these alerts were related to the commit, but others were not related.
|
|
We linked the commit with the relative alerts by detecting whether the lines where the alert generated were within the lines changed in the commit.
|
|
The alerts whose correspond lines were overlapped with the lines in the commit were recorded.
|
|
|
|
|
|
|
|
\subsection{Data Set}
|
|
|
|
In prior work~\cite{vasilescu2015quality,fan2017road}, we created a large PR and issue data set of 365 popular Java projects in GitHub, for each including the history from their inception until October 13, 2017.
|
|
All of them use the pull-based development model~\cite{yu2015wait}.
|
|
In this study, our extra requirements, derived from the process of identifying post-merged defect, were to identify projects that:
|
|
1) have a sufficient number of issues hosted on GitHub'S issue tracker;
|
|
2) the issues are type-tagged at a rate high enough to identify bug issues, and distinguish them from other issue types (e.g., refactor, feature requests, enhancement, etc.);
|
|
and 3) there is a sufficient number of resolved bug issues that have been fixed.
|
|
|
|
Starting from our existing data set, we chose projects having over 100 issue reports in the issue tracker, with over 40\% of the issues tagged with their types.
|
|
|
|
However, different projects use different (synonymous) labels for the same kind of issue~\cite{fan2017road}. To resolve this, we manually compiled a list of bug-related keywords as per~\cite{vasilescu2015quality} (defect, error, bug, issue, mistake, incorrect, fault, and flaw), proceeded by lowercasing and stemming. We identified tags matching these stems in all projects and tagged an issue as bug if at least one stem matched.
|
|
We found that 27\% of issue tags contained at least one bug-related keyword.
|
|
|
|
Finally, we linked bug issues with bug-fixing commits by checking if the issue-id is mentioned in the commit log~\cite{fischer2003populating} (details above in Section~\ref{sec:classify}), and then filtered out the projects with low linking rate to mitigate the risk of sampling biases (e.g., bug feature and commit feature bias~\cite{bird2009fair}): we required that at least 23.3\% of the bugs (the top quartile of our data set) were linked to at least one bug-fixing commit.
|
|
|
|
This selection process left us with 155 projects that utilized PRs; managed most issues with tags in GitHub; and had a desirable linking rate between bug-fix commits and bug reports.
|
|
Overall, we collected 21,029 linked issues from those projects.
|
|
|
|
First, contributors may submit source code, test code, and documentation . We only considered commits that changed at least one source file, and have been merged into the master branch, because these live on, and are more likely to affect project quality for more users.
|
|
This step left us 31,844 buggy commits.
|
|
|
|
Then, some projects have used one or more static analysis tools and few commits of these projects generated alerts.
|
|
These projects may introduce bias to our data set, which can seriously affects the results.
|
|
To mitigate this risk, we only considered the projects that had a certain proportion of the commits with alerts.
|
|
This step left us 50 projects and 9,565 buggy commits, as shown in Table~\ref{tab:repo}.
|
|
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{\label{tab:repo} Summary statistics for selected projects}
|
|
\rowcolors{2}{gray!25}{white}
|
|
\begin{tabular}{lcccc}
|
|
\toprule
|
|
\toprule
|
|
|
|
Statistic & Mean & Min & Median & Max \\
|
|
\midrule
|
|
issues & 464.9 & 102 & 340 & 1609 \\
|
|
labeled\_issues & 170.3 & 30 & 155 & 703 \\
|
|
PRs & 300.9 & 101 & 194 & 585 \\
|
|
\midrule
|
|
linked\_issues & 120.5 & 12 & 76 & 585 \\
|
|
buggy\_commits & 191.3 & 35 & 142.5 & 715 \\
|
|
|
|
\midrule
|
|
alerted\_buggy\_commits & 40.7 & 11 & 22 & 325 \\
|
|
alert\_ratio & 21.2\% & 5.2\% & 18.2\% & 52.3\% \\
|
|
\bottomrule
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
% 具有相应的代码扫描结果 |