-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
139 lines (126 loc) · 6.42 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>SPIN</title>
<!-- favicon -->
<link rel="apple-touch-icon" sizes="180x180" href="assets/favicon/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="assets/favicon/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="assets/favicon/favicon-16x16.png">
<link rel="manifest" href="assets/favicon/site.webmanifest">
<!-- google font -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Roboto+Condensed:ital,wght@0,100..900;1,100..900&display=swap"
rel="stylesheet">
<!-- material symbols -->
<link href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined" rel="stylesheet" />
<link rel="stylesheet" href="styles/styles.css">
<script type="importmap">
{
"imports": {
"three": "https://unpkg.com/[email protected]/build/three.module.js",
"OrbitControls": "https://unpkg.com/[email protected]/examples/jsm/controls/OrbitControls.js",
"EffectComposer": "https://unpkg.com/[email protected]/examples/jsm/postprocessing/EffectComposer.js",
"RenderPass": "https://unpkg.com/[email protected]/examples/jsm/postprocessing/RenderPass.js",
"UnrealBloomPass": "https://unpkg.com/[email protected]/examples/jsm/postprocessing/UnrealBloomPass.js",
"BokehPass": "https://unpkg.com/[email protected]/examples/jsm/postprocessing/BokehPass.js"
}
}
</script>
</head>
<body>
<script type="module" src="src/main.js"></script>
<div class="orientation_msg panel-intro roboto-condensed">
<h2>Please rotate your device to landscape to enjoy better experience.</h2>
</div>
<div class="gradient-overlay"></div>
<div id="intro" class="content panel-intro roboto-condensed">
<button id="startPhase0" class="transit-button material-symbols-outlined">play_arrow</button>
<h1>
Exploring SPIN<span><img src="assets/favicon/android-chrome-192x192.png"></span>
</h1>
<h2>
<u>Sp</u>arsifying and <u>In</u>tegrating Internal Neurons in Large Language Models for Text Classification
</h2>
<h3>
<i>Difan Jiao, Yilun Liu, Zhenwei Tang, Daniel Matter, Jürgen Pfeffer, Ashton Anderson</i>
</h3>
<div class="button-list">
<a href="https://aclanthology.org/2024.findings-acl.277/" target="_blank">
<object type="image/svg+xml"
data="https://img.shields.io/badge/ACL'24-Findings-8c1b13.svg?logo=data:image/svg%2bxml;base64,PHN2ZyBoZWlnaHQ9IjI2MC4wOTA0ODIiIHZpZXdCb3g9IjAgMCA2OCA0NiIgd2lkdGg9IjM4NC40ODE1ODIiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+PHBhdGggZD0ibTQxLjk3NzU1MyAwdjMuMDE1OGgtMzQuNDkwNjQ3Ni03LjQ4NjkwNTR2Ny40ODQ5OSAyNy45Nzc4OCA3LjUyMTMzaDcuNDg2OTA1NCA0Mi4wMTM4OTY2IDcuNDg2OTA2IDExLjAxMjI5MnYtMTUuMDA2MzJoLTExLjAxMjI5MnYtMjAuNDkyODktNy40ODQ5OWMwLTEuNTczNjkgMC0xLjI1NDAyIDAtMy4wMTU4em0tMjYuOTY3Mzk4IDE3Ljk4NTc4aDI2Ljk2NzM5OHYxMy4wMDc5aC0yNi45NjczOTh6IiBmaWxsPSIjZmZmZmZmIiBmaWxsLXJ1bGU9ImV2ZW5vZGQiLz48L3N2Zz4=">
ACL'24 Findings
</object>
</a>
<a href="https://arxiv.org/abs/2311.15983" target="_blank">
<object type="image/svg+xml" data="https://img.shields.io/badge/arXiv-2311.15983-b31b1b.svg?logo=arxiv">
arXiv 2311.15983
</object>
</a>
<a href="https://github.com/difanj0713/SPIN/" target="_blank">
<object type="image/svg+xml" data="https://img.shields.io/badge/GitHub-Code_Repo-blue.svg?logo=github">
GitHub Code Repo
</object>
</a>
<a href="https://github.com/liuyilun2000/spin-visualization/" target="_blank">
<object type="image/svg+xml"
data="https://img.shields.io/badge/GitHub-Web_Demo_Repo-seagreen.svg?logo=github">
GitHub Web Demo Repo
</object>
</a>
</div>
</div>
<div class="content panel-left roboto-condensed" style="z-index: 10;">
<h3><a class="phase phase0 hidden">At the beginning...</a></h3>
<p><a class="phase phase0 hidden">
For each given sentence, we retrieve its neuron activation patterns of FFN neurons at each layer of
the transformer model.
</a></p>
<button id="startPhase1" class="transit-button material-symbols-outlined hidden">skip_next</button>
</div>
<div class="content panel-right roboto-condensed" style="z-index: 9;">
<h3><a class="phase phase1 hidden">Pooling Operation</a></h3>
<p><a class="phase phase1 hidden">
A max pooling is then performed across all tokens, yielding each neuron's maximum activation record
for the sentence.
</a></p>
<button id="startPhase2" class="transit-button material-symbols-outlined hidden">skip_next</button>
</div>
<div class="content panel-left roboto-condensed" style="z-index: 8;">
<h3><a class="phase phase2 hidden">Layer-Wise Linear Probes</a></h3>
<p><a class="phase phase2 hidden">
Linear probing is applied layer-wise, identifying salient neurons for the targeted task among
each layer of the model.
<br><br>
The magnitudes of learned <i>L</i>1-regularized logistic regression weights indicate the
relative contribution to the prediction, i.e., saliency, of each neuron.
</a></p>
<button id="startPhase3" class="transit-button material-symbols-outlined hidden">skip_next</button>
</div>
<div class="content panel-right roboto-condensed" style="z-index: 7;">
<h3><a class="phase phase3 hidden">Salient Neuron Selection</a></h3>
<p><a class="phase phase3 hidden">
The internal neurons are then sparsified based on their saliency.
Guided by a sparsification threshold <i>η</i>, we calculate the cumulative
contribution to have the most significant neurons.
</a></p>
<button id="startPhase4" class="transit-button material-symbols-outlined hidden">skip_next</button>
</div>
<div class="content panel-left roboto-condensed" style="z-index: 6;">
<h3><a class="phase phase4 hidden">Salient Neuron Integration</a></h3>
<p><a class="phase phase4 hidden">
For the sparsified internal neurons from each respective layer, we proceed to integrate
them as cross-layer multi-grained representations into the classification head.
</a></p>
<button id="startPhase5" class="transit-button material-symbols-outlined hidden">skip_next</button>
</div>
<div class="content panel-right roboto-condensed" style="z-index: 5;">
<h3><a class="phase phase5 hidden">Classification Head</a></h3>
<p><a class="phase phase5 hidden">
For each layer, the integrated neurons from that layer and below are concatenated and feed into the
classification head.
</a></p>
</div>
</body>
</html>