-
Notifications
You must be signed in to change notification settings - Fork 4
/
cs591psml.html
450 lines (423 loc) · 18.9 KB
/
cs591psml.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>CS591: Programmable Smart Machines</title>
</head>
<body>
<h1>CAS CS 591, Programmable Smart Machines</h1>
<h2>Schedule</h2>
<p>Thu -- 6:00pm-9:00pm</p>
<h2>Course Outline</h2>
<p>
A longstanding goal in computer science has been
to build machines capable of
automatically improving based on their experience and size.
In his 1968 paper,
<a href="http://dx.doi.org/10.1038/218019a0">'Memo' Functions and
Machine Learning</a> (Nature, Apr. '68),
Donald Michie challenged computer scientists with the
following goal:
<blockquote>
"It would be useful if computers could learn from experience and
thus automatically improve the efficiency of their own programs
during execution... When I write a clumsy program for a contemporary
computer a thousand runs on the machine do not re-educate my
handiwork. On every execution, each time-wasting blemish and
crudity, each needless test and redundant evaluation, is
meticulously reproduced."
</blockquote>
Over the past several years the instructors of this course
have been developing the concept of
Programmable Smart Machines (PSMs): hybrid computing systems whose
output is as programmed but which transparently learn and automatically
improve their operation. Both from the theoretical and practical
point of view, our work attempts to side-step the traditional bottlenecks
associated with the von Neumann architecture while preserving its
programming model.
In this course we will look at a cross section of the material
that has informed our work, encompassing
applied and theoretical
topics. Topics will include material from
complexity theory, computer architecture,
operating systems, machine learning and biologically
inspired computation. A fundmental aspect of this course is
to identify and clearly state open questions in this area of work.
</p>
<p>
The course will be predominantly structured as a seminar in which
each week we will be looking at one or two research papers.
You will be required to submit weekly reviews and actively
participate in the discussions. Each student will conduct and a
term long project.
</p>
<h2>Instructors</h2>
<p>Jonathan Appavoo:
<a href="mailto:[email protected]">[email protected]</a>
<p>Steve Homer:
<a href="mailto:[email protected]">[email protected]</a>
<p>Amos Waterland:
<a href="mailto:[email protected]">[email protected]</a>
<h2>Target audience</h2>
<p>This course is targeted towards
graduate and advanced undergraduate students.
We encourage enrollment by both
theory and systems students, especially
those of you willing to push
the boundaries of what you have been thinking about.
You are encouraged to incorporate your personal interests and
background into the class material.
</p>
<h2>Prerequisites</h2>
<p>
Students taking this class must have permision from the
instructors, be a either a senior or graduate student,
and be proficient in the core material of systems and theory.
</p>
<h2>Workload</h2>
<p>
Each week we will be covering on average one to two research papers that
you will be expected to read, review and discuss. These papers
will likely require that you find and read additional material
as necessary to ensure your comprehension. Do not underestimate
the amount of work this can be. You will be required to submit
a written review of the papers prior to class. You will also
be expected to actively participate in the in-class discussion.
Each student is expected to lead one of the class discussions
by summarizing the paper and seeding conversation with questions
and observations from the paper.
</p>
<p>In addition to the weekly paper reviews there will be a final project
in which you will explore in detail an aspect of Programmable
Smart Machines. Projects can range from a theoretical
exploration to an applied experimental evaluation. The topic of your
project must be approved by the instructors.
You will be expected
to present a poster about your project and to submit a brief
written report by the end of the term. As part of the project you
will be expected to establish in conjunction with the
instructors the goals and criteria for its evalution. Each week you will
provide a brief update on the progess of your project.
</p>
<p>
The project is due by the end of the lecture period. The project presentations
will be given in the form of a final poster during the scheduled final
exam slot.
</p>
<p>Students are expected to work individually on the weekly reviews.
With instructor approval the final project may be done in groups of
up to
two. There will be no final exam other than the poster presentations. </p>
<h2>Grading</h2>
Reviews, Discussions, paper presentation: 40%
<p>
Project Poster and Report: 60%
<h2>Tentative schedule</h2>
<table border="1" style="background-color:#FFFFCC;border-collapse:collapse;border:1px solid #FFCC00;color:#000000;width:100%" cellpadding="3" cellspacing="3">
<tr>
<td><b>Date</b></td>
<td><b>Description</b></td>
<td><b>Links</b></td>
<td><b>Deadlines</b></td>
</tr>
<tr><td>01/20</td><td>Class 1: Overview and Introductions</td><td></td></tr>
<tr><td>01/22</td><td>Class 2: ASC Presentation & Discussion</td>
<td><a href="http://people.seas.harvard.edu/~apw/papers/asplos14.pdf">
1: ASC: Automatically Scalable Computation<br>
<a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/review-copy.pdf">2: ieee draft</a>
</td></tr>
<tr><td>01/27</td><td>Class 3: ASC Tutorial</td>
<td>
<a href="http://people.seas.harvard.edu/~apw/papers/asplos14.pdf">1:
ASC: Automatically Scalable Computation<br>
<a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/review-copy.pdf">2: ieee draft</a>
</td></tr>
<tr><td>01/29</td><td>Class 4: Discussion of Projects and Papers</td>
<td>
</td>
<td> Instructors will update lists by this point. You must be ready to
discuss your interests.
</tr>
<tr><td>02/03</td><td>Class 5: History and Foundations</td>
<td>
<a href="http://www.virtualtravelog.net/wp/wp-content/media/2003-08-TheFirstDraft.pdf">
1: First Draft of a Report on the EDVAC</a>
</td></tr>
<tr><td>02/05</td><td>Class 6: History and Foundations</td>
<td>
<a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/McCulloch.and.Pitts.pdf">1: A LOGICAL CALCULUS OF THE IDEAS IMMANENT IN NERVOUS ACTIVITY</a>
</td>
<td>Pick Project and Paper</td>
</tr>
<tr><td>02/10</td><td>Class 7: History and Foundations</td>
<td>
<a href="http://www.cs.utexas.edu/~hunt/research/hash-cons/hash-cons-papers/michie-memo-nature-1968.pdf">
1: "Memo" functions and Machine Learning</a><br>
<a href="http://www.wisdom.weizmann.ac.il/~oded/cc-sum.html">2:
Advice, Space Complexity and Approximate Counting Lecture
Notes on Complexity Theory (focus on Lectures 8 and 10 depends
on 1 and 2) by Oded Goldreich</a>
</td></tr>
<tr><td>02/12</td><td>Class 8: History and
Foundations</td>
<td>
<a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/10.1.1.335.3398.pdf">1: THE PERCEPTRON: A PROBABILISTIC MODEL FOR INFORMATION
STORAGE AND ORGANIZATION IN THE BRAIN</a>
</td></tr>
<tr><td>02/19</td><td>Class 9: History and Foundations</td>
<td>
</td>
<td>One Page Project Topic Proposal Due</td>
</tr>
<tr><td>02/26</td><td>Class 11: TBA </td><td></td></tr>
<tr><td>03/05</td><td>Class 13: Neuromorphic</td><td> <a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/10.1.1.161.9762.pdf">1: Neuromorphic Electronic Systems</a></br>
<a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/p13-monroe.pdf">2: Neuromorphic Computing Gets Ready for the (Really) Big
Time</a> Follow up on references</br>
</td></tr>
<tr><td>03/19</td><td>Class 15: Chris </td><td><a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/neuromorphic/020.IJCNN2013.Corelet.pdf">"Cognitive
Computing Programming Paradigm: A Corelet Language for Composing
Networks of Neurosynaptic Cores", Amir et al.</a> </td><td>Three Page Project
Overview Due</tr>
<tr><td>03/26</td><td>Class 17: Adam and Andrew </td><td><a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/neuromorphic/020.IJCNN2013.Corelet.pdf">1. "Cognitive
Computing Programming Paradigm: A Corelet Language for Composing
Networks of Neurosynaptic Cores", Amir et al.</a> </br>
<a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/neuromorphic/asplos166-hashmi.pdf">2. A
Case for Neuromorphic ISAs</a>
</td></tr>
<tr><td>04/02</td><td>Class 19: Tomislav</td><td><a
href="https://homes.cs.washington.edu/~ruzzo/papers/limits.pdf">Limits
to Parallel Computation: Chapter 2</a></td></tr>
<tr><td>04/09</td><td>Class 21: Allan </td><td><a href="http://theoryofcomputing.org/articles/gs001/gs001.pdf">A Brief Introduction to Fourier Analysis on the Boolean Cube</a></td></td></tr>
<tr><td>04/16</td><td>Class 23: Thomas and Steve </td><td><a
href="https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chilimbi.pdf">1. "Project
Adam: Building an Efficient and Scalable
Deep Learning Training System", Chilimbi et al.</a> </br>
<a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/learning/Li2015SpontaneousRetrievalFrom.pdf">2. Spontaneous Retrieval from Long-Term Memory for a Cognitive Architecture</a>
</td></tr>
<tr><td>04/23</td><td>Class 25: William</td><td><a
href="http://koray.kavukcuoglu.org/publis/lecun-iscas-10.pdf">"Convolutional
Networks and Applications in Vision",LeCun et al.</a></td><td>Poster Outline Due</td</tr>
<tr><td>04/30</td><td>Class 27: Alexandar </td><td><a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/systems/Usenix04.pdf">"Flashback: A Lightweight Extension for
Rollback and Deterministic Replay for Software Debugging",
Srinivasan et al.</a></td><td>Project Report Due</td</tr>
<tr><td>TBA</td><td>EXAM: POSTER SESSION</td><td></td>></tr>
</table>
<h3>Papers & Project Details</h3>
<style type="text/css">
.tg {border-collapse:collapse;border-spacing:0;}
.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
</style>
<table class="tg">
<tr>
<th class="tg-031e">Name</th>
<th class="tg-031e">Paper</th>
<th class="tg-031e">Date</th>
<th class="tg-031e">Project</th>
</tr>
<tr>
<td class="tg-031e">William</td>
<td class="tg-031e"><a
href="http://koray.kavukcuoglu.org/publis/lecun-iscas-10.pdf">"Convolutional
Networks and Applications in Vision",LeCun et al.</a></td>
<td class="tg-031e">04/23</td>
<td class="tg-031e"><a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/projects/williamang.pdf">Adding Convolutional Neural Network Support to ASC</a></td>
</tr>
<tr>
<td class="tg-031e">Alexandar</td>
<td class="tg-031e"><a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/systems/Usenix04.pdf">"Flashback: A Lightweight Extension for
Rollback and Deterministic Replay for Software Debugging",
Srinivasan et al.</a></td>
<td class="tg-031e">04/30</td>
<td class="tg-031e"><a href="http://breen.io/asc/proposal.html">Extending ASC to support peer-to-peer exchange of data</a></td>
</tr>
<tr>
adamperson.pdf
<td class="tg-031e">Andrew</td>
<td class="tg-031e">
<a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/neuromorphic/asplos166-hashmi.pdf">A Case for Neuromorphic ISAs</a>
</td>
<td class="tg-031e">03/26</td>
<td class="tg-031e">Training and Evaluating a game playing network
on ASC</td>
</tr>
<tr>
<td class="tg-031e">Christopher</td>
<td class="tg-031e"><a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/approximatecomputing/analognpu-isca14.pdf">General-Purpose Code Acceleration with Limited-Precision Analog Computation</a></td>
<td class="tg-031e">03/19</td>
<td class="tg-031e"><a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/projects/chriskuech.pdf">ASC for Amateur Programmers</a></td>
</tr>
<tr>
<td class="tg-031e">Steve</td>
<td class="tg-031e"><a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/learning/Li2015SpontaneousRetrievalFrom.pdf">Spontaneous Retrieval from Long-Term Memory for a Cognitive Architecture</a></td>
<td class="tg-031e">04/16</td>
<td class="tg-031e"><a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/projects/steveman.pdf">The
Brain and AI</a></td>
</tr>
<tr>
<td class="tg-031e">Thomas</td>
<td class="tg-031e"><a href="https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chilimbi.pdf">"Project Adam: Building an Efficient and Scalable
Deep Learning Training System", Chilimbi et al.</a></td>
<td class="tg-031e">04/16</td>
<td class="tg-031e"></td>
</tr>
<tr>
<td class="tg-031e">Adam</td>
<td class="tg-031e"><a
href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/neuromorphic/020.IJCNN2013.Corelet.pdf">"Cognitive
Computing Programming Paradigm: A Corelet Language for Composing Networks of Neurosynaptic Cores", Amir et al.</a></td>
<td class="tg-031e">03/26</td>
<td class="tg-031e"><a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/projects/adamperson.pdf">Specialization vs Generalization in the context of learning state space structure.</a></td>
</tr>
<tr>
<td class="tg-031e">Tomislav</td>
<td class="tg-031e"><a
href="https://homes.cs.washington.edu/~ruzzo/papers/limits.pdf">Limits
to Parallel Computation: Chapter 2</a></td>
<td class="tg-031e">04/02</td>
<td class="tg-031e"></td>
</tr>
<tr>
<td class="tg-031e">Allan</td>
<td class="tg-031e"> <a href="http://theoryofcomputing.org/articles/gs001/gs001.pdf">A Brief Introduction to Fourier Analysis on the Boolean Cube</a></td>
<td class="tg-031e">04/09</td>
<td class="tg-031e">Fourier Analysis of known compromised Crypto Software</td>
</tr>
</table>
<h3> List of some possible papers: </h3>
This list will be extended in the first few weeks of class
based on students back grounds and interests
<ol>
<b><i>Theory</i></b>
<li>
Kushilevitz Mansour learning algorithm via Fourier coefficient analysis.
<a href="http://www.eecs.berkeley.edu/~luca/pacc/lecture9.pdf">link</a>
KUSHILEVITZ, E., and MANSOUR, Y., "Learning decision trees using the
Fourier spectrum", SIAM Journal on Computing, Vol. 22, No. 6,
pp. 1331-1348, 1993. If you want to dig deeper you can look up the
by paper Goldreich and Levin which is referenced in the KUSHILEVITZ,
E., and MANSOUR, Y paper and forms the basis for their method.
</li>
<li>
Fourier analysis on the cube."A Brief Introduction to Fourier
Analysis on the Boolean Cube" By Ron de Wolf.
<a href="http://theoryofcomputing.org/articles/gs001/gs001.pdf">link</a>
This is a good intro to some of the ideas underlying the learning
theory above.
</li>
<li>
Paul Vitanyi et al on compression, relative compressibility,
normalized compression distance and various thing related to
Kolmogorov complexity: "Clustering by compression", by Vitanyi and Cilibrasi
(<a href="http://arxiv.org/abs/cs/0312044">link</a>) and follow the
backward and forward references to many related papers.
Other papers by Vitanyi and others include: Nonapproximablity of the
Normalized Information Distance, New applications of the
incompressibility method: Part II Harry Buhrman, Tao Jiang, Ming Li, Paul Vitanyi,
Philosophical Issues in Kolmogorov Complexity, Normalized
Information Distance is Not Semicomputable, On Prediction by Data
Compression
</li>
<li>
<a href="http://www.cs.utexas.edu/~hunt/research/hash-cons/hash-cons-papers/michie-memo-nature-1968.pdf">
"Memo" functions and Machine Learning</a><br>
D. Michie<br>
Nature, 1968
</li>
<li>
<a href="http://web.mit.edu/6.976/www/handout/valiant2.pdf">
A bridging model for parallel computation</a><br>
L. Valiant<br>
CACM, 1990
</li>
<li>
<a href="https://homes.cs.washington.edu/~ruzzo/papers/limits.pdf">
Limits to Parallel Computation</a> (Chapter 2)<br>
R. Greenlaw<br>
1991
</li>
<li>
<a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/systems">Systems</a>
</li>
<li>
<a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/learning">Learning</a>
</li>
<li>
<a href="http://www.cs.bu.edu/~jappavoo/Resources/591psml/papers/neuromorphic">Neuromorphic</a>
</li>
</ol>
<h3> Project Ideas </h3>
This list will be extended in the first few weeks of class
based on students' backgrounds and interests.
<ol>
<b><i>Fourier Analysis</i></b>
<li>
Study a set of functions using Fourier Analysis
</li>
<b><i>ASC</i></b>
<li> Add Input output support </li>
<li> Break on PC values of read write open extend for IO </li>
<li> Build OS calls into ASC </li>
<li> Visualization </li>
<li> Speculative execution with IO </li>
<li> Study / explore a class of programs in asc </li>
</li>
<br>
<b><i>Neueral Network Hardware</i></b>
<li>
Add a UART TX/RX wrapper interface to a version of DANA that can be
used to communication an NN Transaction to DANA
</li>
<li>
Currently, the modified FANN configuration only supports an
all-to-all connectivity subset of FANN configurations. This needs to
be relaxed to enable more limited connection strategies. This is a
precursor to enabling Convolutional Neural Network (CNN)
support. Note, that this specific issue does not allow for
connections that skip layers.
</li>
<li>
There exists the potential for some fault-tolerance within neural
networks as evidenced by previous work with modular redundancy and
no voting, see BARC 2015. A more advanced approach involves
dynamically splitting "important" nodes or ignoring "un-important"
nodes while an NN is executing (on an NN accelerator like DANA or
simply in software). Much of this work is TBD, but this should be at
least one publishable unit of work, here.
</li>
<li>
The FANN should be generic enough to support Convolutional Neural
Networks (CNNs). However, this support doesn't already exist. At the
software level, we need to add support for this. What is primarily needed:
1) Add max-pooling activation function 2) Possibly some support for
describing a CNN and then mapping this to a FANN configuration 3)
Evaluation and testing
</li>
<li>
The current version of DANA (at least at the level of gem5
integration) uses an x86 ISA interface. This needs to be moved to
the RISC-V ISA. Note, that the RISC-V ISA should have support for
generic "accelerator" extensions. However, we may need more support
than this currently offers.
</li>
</ol>
<h2>Collaboration and Academic Honesty</h2>
<p>
All course participants must adhere to the
CAS Academic Conduct Code. Instances of academic
dishonesty will be reported to the
academic conduct committee.
</p>
</body>
</html>