-
Notifications
You must be signed in to change notification settings - Fork 14
/
plan.txt
116 lines (71 loc) · 4.12 KB
/
plan.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
## Introduction and definitions (defects, faults, failures) (1 week: L1-3)
L1: introduction
L2: faults, errors, failures; dealing with an imperfect world; testing vs debugging (old L2)
L3: in-class exercise on faults/errors/failures; validating a test suite---line intersection (old L3)
L3 is (M) January 9.
## Defining Test Suites (4 weeks: L4-L16)
L4: static vs dynamic/"complete testing"/test cases/observability&controlability/test requirements/quantifying coverage/black-box vs white-box
L5: open-ended exploratory testing (http://katrinatester.blogspot.com.au/2016/12/the-testing-pendulum-finding-balance-in.html)
L6, L7, L8: statement/branch coverage (old L6, L9 on CFGs)
L9: graph-based models/design documentation (round trips; some content from old L07, L22)
L9 is (M) January 23. Target cutoff date for A1 material.
L10, L11: automatically/programmatically generating test suites -- grammars (old L17, L29, L32, L33)
L12 is (M) January 30. A1 due. Was actually the bulk of fuzzing (old L33)
L13: how good can your test suite be? JUnit / mutation
https://avandeursen.com/2012/12/21/line-coverage-lessons-from-junit/
L14/L15/L16/L17: how good? mutation -> PIT for Java (old L18/L19/L20)
L16 is (W) Feb 8.
## Engineering test suites (L17-25)
L18: why write tests?
https://hackernoon.com/treat-yourself-e55a7c522f71
on test automation (http://www.ontestautomation.com/not-so-useful-metrics-in-test-automation/) At its heart, test automation is – or at least should be – meant to increase the effectiveness of testing efforts.
L19: Selenium, web testing (L34), Page object design pattern
http://www.seleniumhq.org/docs/06_test_design_considerations.jsp#chapter06-reference
https://martinfowler.com/bliki/PageObject.html
https://developers.redhat.com/blog/2017/02/13/testing-testing-gcc/
L20: answer questions about midterm;
midterm is after L20
[reading week, A2 due at end of reading week]
L21: must-beliefs vs may-beliefs, coverity
L22: more on must-beliefs vs may-beliefs, given by derek rayside
L23: mike godfrey on clones
L24: regression testing (old L34)
L25: unit tests, test refactoring (old L30), refactoring code to be testable (old L36---RPCs, RESTful APIs, micro-services), badly designed tests (old L35)
L26 is Friday, March 10
L26: mock objects (old L30) continuous integration (Travis), flaky tests, hermetic tests
L27: testing infrastructure/DevOps (see also P4P L35)
http://nerds.airbnb.com/testing-at-airbnb/
https://puppet.com/blog/hitchhikers-guide-to-testing-infrastructure-as-and-code
https://martinfowler.com/articles/serverless.html
https://medium.com/@hichaelmart/lambci-4c3e29d6599b
https://about.gitlab.com/2015/02/03/7-reasons-why-you-should-be-using-ci/
## Tools for V&V (L28-34)
manual "tools":
L28: code review
http://web.mit.edu/6.005/www/fa15/classes/04-code-review/
https://blog.fogcreek.com/increase-defect-detection-with-our-code-review-checklist-example/
L29: reporting bugs (old L28) = march 17
L30: PMD out-of-the-box/XPath
ECE FYDP on March 21
L31: PMD on java / linters
L32: static vs dynamic analysis, static analysis I
static analysis I [fewer tools, more depth] [concurrency, old L10] [aComment, old L11]
L32: static analysis II [type systems, immutability, compiler warnings, software model checking, JML tools. Coverity, FindBugs]
L33: static analysis III [FB Infer / Clang address sanitizer]
L34: dynamic analysis I [assertions, race detectors, valgrind & friends]
L35: dynamic analysis II [bugfinding]
L36: dynamic analysis III
A3 due on March 24 (L32)
SE FYDP on March 28
info:
tools (old L11, L12)- Korat, JPF, FindBugs, Daikon, ESC/Java, valgrind, cppcheck,
clang static analyzer, spling, pex, klee, coverity, grammatech codesonar, FB Infer
https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm
https://github.com/google/sanitizers/wiki/AddressSanitizerComparisonOfMemoryTools
http://maintainablecode.logdown.com/posts/245425-valgrind-is-not-a-leak-checker
http://btorpey.github.io/blog/2014/03/27/using-clangs-address-sanitizer/
http://clang.llvm.org/docs/AddressSanitizer.html
## Debugging (L35-36)
L35:
L36:
reference: Andreas Zeller. \emph{Why Programs Fali: a Guide to Systematic Debugging.}