-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathNotes
174 lines (115 loc) · 3.17 KB
/
Notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
Mnist Dense training with y labels and z=5, ==============
Reconstruction loss 17,2 with adam 0.0002
Reconstruction loss 15.77 with adam 0.00004
Reconstruction loss 15.54 with adam 0.000008
-------------------------------------------
0.0002 -> 0.025644
0.00004 -> 0.023410
After RMSProp -> 0.022315
Reconstruction Lost 0.024178
Discrimination Lost 0.693033
Encoder Lost 0.693788
Mnist Dense training without y labels and z=5,=============
Reconstruction loss 25.64 with adam 0.0002
Reconstruction loss 21.64 with adam 0.00004
-------------------------------------------------
0.0002 -> 0.0278
0.00004 -> 0.026297
After RMSProp -> 0.025360
Mnist Conv training with y labels and z=5,
Reconstruction loss 23.01 with adam 0.0002
Reconstruction loss 21.48 with adam 0.00004
0.0002 -> 0.027869
0.00004 -> 0.026446
After RMSProp -> 0.027287
------------
FixedBN:
0.0002 -> 0.028565
Subpix noy epoch ============================================
Parameters: 1042810
Reconstruction loss 82.41 with adam 0.0002
Reconstruction loss 78.76 with adam 0.00004
--------------------------------------------
Number of parameters in model 2171994
LR : 0.0002, 0.0002, 0.0002
Starting epoch 40
Epoch took 1235 seconds.
Reconstruction Lost 94.468611
Discrimination Lost 0.693490
Encoder Lost 0.695844
LR : 0.00005, 0.0002, 0.0002
--------------------------------------------
Epoch 15:
0.033366
Celeb Conv noy ==============================================
Parameters: 827690
93.35
--------------------------------------------
Number of parameters in model 826762
LR : 0.0002, 0.0002, 0.0002
Starting epoch 33
Epoch took 1443 seconds.
Reconstruction Lost 102.588236
Discrimination Lost 0.683555
Encoder Lost 0.720770
LR : 0.00005, 0.0002, 0.0002
---------------------------------------------
0.051
Mnist Conv training without y labels and z=5,
Reconstruction loss 27.10 with adam 0.0002
Reconstruction loss 23.05 with adam 0.00004
Celeb Res y
105.7
Celeb Res noy
85.34
Celeb Conv y ================================================
110.73
-------------------------------------------
LR : 0.0002, 0.0002, 0.0002
Starting epoch 33
Epoch took 1449 seconds.
Reconstruction Lost 139.458588
Discrimination Lost 0.647007
Encoder Lost 0.850248
Celeb Gan Res noy
102.23
Celeb 128
1836.60
We train subpix without gan and we get:
Starting epoch 14
Epoch took 3523 seconds.
Reconstruction Lost 0.033569
Discrimination Lost 0.676666
Encoder Lost 0.736910
Model saved as models/model_Celeb_Subpix_noy_bn.ckpt
Starting epoch 15
Epoch took 3560 seconds.
Reconstruction Lost 0.033366
Discrimination Lost 0.673484
Encoder Lost 0.745762
Model saved as models/model_Celeb_Subpix_noy_bn.ckpt
Starting epoch 16
Epoch took 3409 seconds.
Reconstruction Lost 0.033367
Discrimination Lost 0.669159
Encoder Lost 0.757511
Next we try to see what changes when we put batch norm before relu is applied
# SCENARIO 5
LR 0.0001
LR 0.0002
LR 0.0002
GAN:
DISC 0.0001
GEN 0.0001
Features just X_Image
Recover from CONV
# SCENARIO 6
ADD BN ON GAN INPUT
Features as output from first conv (No nonlinearities)
k=8
All lr 0.0002
Scenario 7
k=8
lr 3x - 0.0002
lr - 0.0001
Not Restored, Train until gan has loss 0.69, ximage + features from 3rd layer