-
Notifications
You must be signed in to change notification settings - Fork 4
/
L34.tex
151 lines (99 loc) · 7.1 KB
/
L34.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
\documentclass[11pt]{article}
\usepackage{listings}
\usepackage{tikz}
\usepackage{url}
%\usepackage{algorithm2e}
\usetikzlibrary{arrows,automata,shapes}
\tikzstyle{block} = [rectangle, draw, fill=blue!20,
text width=5em, text centered, rounded corners, minimum height=2em]
\tikzstyle{bt} = [rectangle, draw, fill=blue!20,
text width=1em, text centered, rounded corners, minimum height=2em]
\newtheorem{defn}{Definition}
\newtheorem{crit}{Criterion}
\newcommand{\true}{\mbox{\sf true}}
\newcommand{\false}{\mbox{\sf false}}
\newcommand{\handout}[5]{
\noindent
\begin{center}
\framebox{
\vbox{
\hbox to 5.78in { {\bf Software Testing, Quality Assurance and Maintenance } \hfill #2 }
\vspace{4mm}
\hbox to 5.78in { {\Large \hfill #5 \hfill} }
\vspace{2mm}
\hbox to 5.78in { {\em #3 \hfill #4} }
}
}
\end{center}
\vspace*{4mm}
}
\newcommand{\lecture}[4]{\handout{#1}{#2}{#3}{#4}{Lecture #1}}
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in
\parindent 0in
\parskip 1.5ex
%\renewcommand{\baselinestretch}{1.25}
\begin{document}
\lecture{34 --- March 30, 2015}{Winter 2015}{Patrick Lam}{version 1}
\section*{Regression Testing}
Regression testing refers to any software testing that uncovers errors by retesting the modified program (Wikipedia). This form of testing often refers to large combined sets of test cases to detect regressions:
\begin{itemize}
\item of a bug fix that a developer has proposed.
\item of related and unrelated other features that have been added.
\end{itemize}
Regression testing usually refers to system level (integration level) testing that runs the entire process.
\subsection*{Attributes of Regression Tests}
Regression tests usually have the following attributes:
\begin{itemize}
\item \textbf{Automated}: no real reason to have manual regression tests.
\item \textbf{Appropriately Sized}: too small and bugs will be missed. Too large and they will take a long time to run. Optimally, we want to run tests continuously.
\item \textbf{Up-to-date}: ensure that tests are valid for the version of program being tested.
\end{itemize}
\subsection*{Automating Regression Tests}
Regression tests often have a low yield in terms of finding bugs. Therefore, automation is very important.
\textbf{Input}
If the input is from a file, regression tests are easy to run. There may still be a problem with validating output. We can also create special mocks that can take input from a file or other sources (e.g. scripting engines). We can also redirect input from a console to a file.
For UIs, the standard approach is to capture and replay events. This approach can be fragile! For example, tests may fail based on window placement or whitespace. Mozilla has a project named MozMill\footnote{\url{https://developer.mozilla.org/en/Mozmill}} that is used to test Firefox and Thunderbird. For web applications, there is capture and replay for HTTP using Selenium.
\textbf{Output}
Verifying output can be hard!! Problems can arise from issues such as resolution, whitespace, window placement etc.
\textbf{Mozilla Case Study\footnote{\url{http://www.mozillazine.org/roc/archives/2005/03/visual\_regressi.html}}}
The case study presents an approach to testing Gecko based applications. Gecko is the layout engine for Mozilla applications like FireFox and ThunderBird.
In the past, a frame tree with coordinates for all UI elements was created and manual testers performed tests. Not optimal! The new approach was to capture screenshots after test cases and compare them with the expected screenshot. There were a few problems with the approach due to nondeterminism:
\begin{itemize}
\item Animated images - no way to ensure tests would function with animated images.
\item Font Hinting - the same character would appear slightly differently after each run of the application.
\item Other bugs - resolution problems, minor changes in layout etc.
\end{itemize}
The problem was ``really hellish'' and the partial solution was to enable logging in the application. The logs would essentially be compared with expected logs. This became very ugly given 1300 or so test cases.
\subsection*{Some Notes}
\begin{itemize}
\item Sometimes it is a good idea to have 1 test case per bug fix. This can get unwieldy over time if redundant test cases are not removed.
\item Test suites can be evaluated using coverage criteria (e.g. how many use cases were covered) or manually in order to get rid of redundant tests.
\item There is some interest in test case prioritization but little practical application since it is unclear how to implement it.
\item Expect low yields in terms of finding bugs so automation is key!
\item Ensure regression tests are up-to-date. This can be a pain but is necessary since when software changes, test suits break or become incomplete. When a new software version comes out, try the old suite.
\begin{itemize}
\item If there are no failures, new tests may need to be added to test new functionality.
\item If there are some failures, ensure if software or test is broken. Tests can depend on inessential features of the output (e.g. order or text etc.).
\end{itemize}
\item Try to get rid of irrelevant tests over time. For example, if you have a bunch of tests that are related to the same bug, keep only 1 test for the bug.
\end{itemize}
\section*{Selenium\footnote{\url{http://seleniumhq.org}}}
Selenium is a tool for testing web applications. With it, web application tests can be automated to test multiple browsers and multiple platforms. Selenium is similar to mock objects since it also uses the idea of expecting certain events and replaying the expected output (record and replay). Mocks are usually low level and deal with code specifics, while Selenium is usually at the system level.
\section*{Industry Processes}
Good software companies have certain testing features that help them deliver good quality software. The following practices are used in industry:
\begin{itemize}
\item \textbf{Unit Tests:} Each class has an associated unit test. If you change a class, you must also modify the unit test.
\item \textbf{Code Reviews:} Each branch in the central version control system has owners. In order to commit code to that branch, you must have your code reviewed and approved by one of the owners. This ensures code quality.
\item \textbf{Continuous Builds:} There is often a machine that continuously checks out and tests the latest code. All unit and regression tests are run and the status is made public to the team. The status contains information about the whether the code was built successfully, whether all unit and regression tests passed, and a list of the last few commits that were made to the branch. This ensures developers try to submit good code since if you break something, everyone knows about it :)
\item \textbf{One-button Deploy:} If all tests have passed, one should be able to deploy to production
with one command.
\item \textbf{Back Button:} Systems should be designed so that it's possible to roll back changes.
\end{itemize}
\end{document}