-
Notifications
You must be signed in to change notification settings - Fork 2
/
task.html
422 lines (293 loc) · 10 KB
/
task.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
<html>
<head>
<title>KDD-CUP-99 Task Description </title>
</head>
<body>
<p>This document is adapted
from the paper <I>Cost-based Modeling and Evaluation for Data Mining</I>
<I>With Application to Fraud and Intrusion Detection: Results from the
JAM Project</I> by Salvatore J. Stolfo, Wei Fan, Wenke Lee, Andreas Prodromidis,
and Philip K. Chan.
<BR>
<BR>
<H4>
INTRUSION DETECTOR LEARNING</H4>
Software to detect network intrusions protects a computer network from
unauthorized users, including perhaps insiders. The intrusion detector
learning task is to build a predictive model (i.e. a classifier) capable
of distinguishing between ``bad'' connections, called intrusions or attacks,
and ``good'' normal connections.
<P>The 1998 DARPA Intrusion Detection Evaluation Program was prepared and
managed by MIT Lincoln Labs. The objective was to survey and evaluate research
in intrusion detection. A standard set of data to be audited, which
includes a wide variety of intrusions simulated in a military network environment,
was provided. The 1999 KDD intrusion detection contest uses a version
of this dataset.
<P>Lincoln Labs set up an environment to acquire nine weeks of raw TCP
dump data for a local-area network (LAN) simulating a typical U.S. Air
Force LAN. They operated the LAN as if it were a true Air Force environment,
but peppered it with multiple attacks.
<P>The raw training data was about four gigabytes of compressed binary
TCP dump data from seven weeks of network traffic. This was processed
into about five million connection records. Similarly, the two weeks
of test data yielded around two million connection records.
<P>A connection is a sequence of TCP packets starting and ending at some
well defined times, between which data flows to and from a source IP address
to a target IP address under some well defined protocol. Each connection
is labeled as either normal, or as an attack, with exactly one specific
attack type. Each connection record consists of about 100 bytes.
<P>Attacks fall into four main categories:
<UL>
<LI>
DOS: denial-of-service, e.g. syn flood;</LI>
<LI>
R2L: unauthorized access from a remote machine, e.g. guessing password;</LI>
<LI>
U2R: unauthorized access to local superuser (root) privileges, e.g.,
various ``buffer overflow'' attacks;</LI>
<LI>
probing: surveillance and other probing, e.g., port scanning.</LI>
</UL>
It is important to note that the test data is not from the same probability
distribution as the training data, and it includes specific attack types
not in the training data. This makes the task more realistic.
Some intrusion experts believe that most novel attacks are variants of
known attacks and the "signature" of known attacks can be sufficient to
catch novel variants. The datasets contain a total of 24 <A HREF="training_attack_types">training
attack types</A>, with an additional 14 types in the test data only.
<BR>
<BR>
<H4>
DERIVED FEATURES</H4>
Stolfo et al. defined higher-level features that help in distinguishing
normal connections from attacks. There are several categories of
derived features.
<P>The ``same host'' features examine only the connections in the past
two seconds that have the same destination host as the current connection,
and calculate statistics related to protocol behavior, service, etc.
<P>The similar ``same service'' features examine only the connections in
the past two seconds that have the same service as the current connection.
<P>"Same host" and "same service" features are together called time-based
traffic features of the connection records.
<P>Some probing attacks scan the hosts (or ports) using a much larger time
interval than two seconds, for example once per minute. Therefore,
connection records were also sorted by destination host, and features were
constructed using a window of 100 connections to the same host instead
of a time window. This yields a set of so-called host-based traffic
features.
<P>Unlike most of the DOS and probing attacks, there appear to be no sequential
patterns that are frequent in records of R2L and U2R attacks. This is because
the DOS and probing attacks involve many connections to some host(s) in
a very short period of time, but the R2L and U2R attacks are embedded in
the data portions
<BR>of packets, and normally involve only a single connection.
<P>Useful algorithms for mining the unstructured data portions of packets
automatically are an open research question. Stolfo et al. used domain
knowledge to add features that look for suspicious behavior in the data
portions, such as the number of failed login attempts. These features
are called ``content'' features.
<P>A complete listing of the set of features defined for the connection
records is given in the three tables below. The data schema of the
contest dataset is available in <A HREF="kddcup.names">machine-readable
form</A> .
<BR>
<BR>
<CENTER><TABLE BORDER WIDTH="80%" NOSAVE >
<TR NOSAVE>
<TD><I>feature name</I></TD>
<TD NOSAVE><I>description </I></TD>
<TD><I>type</I></TD>
</TR>
<TR>
<TD>duration </TD>
<TD>length (number of seconds) of the connection </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>protocol_type </TD>
<TD>type of the protocol, e.g. tcp, udp, etc. </TD>
<TD>discrete</TD>
</TR>
<TR>
<TD>service </TD>
<TD>network service on the destination, e.g., http, telnet, etc. </TD>
<TD>discrete</TD>
</TR>
<TR>
<TD>src_bytes </TD>
<TD>number of data bytes from source to destination </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>dst_bytes </TD>
<TD>number of data bytes from destination to source </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>flag </TD>
<TD>normal or error status of the connection </TD>
<TD>discrete </TD>
</TR>
<TR>
<TD>land </TD>
<TD>1 if connection is from/to the same host/port; 0 otherwise </TD>
<TD>discrete</TD>
</TR>
<TR>
<TD>wrong_fragment </TD>
<TD>number of ``wrong'' fragments </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>urgent </TD>
<TD>number of urgent packets </TD>
<TD>continuous</TD>
</TR>
<CAPTION ALIGN=BOTTOM>
<BR>Table 1: Basic features of individual TCP connections.</CAPTION>
</TABLE></CENTER>
<CENTER><TABLE BORDER WIDTH="80%" NOSAVE >
<TR>
<TD><I>feature name</I></TD>
<TD><I>description </I></TD>
<TD><I>type</I></TD>
</TR>
<TR>
<TD>hot </TD>
<TD>number of ``hot'' indicators</TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>num_failed_logins </TD>
<TD>number of failed login attempts </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>logged_in </TD>
<TD>1 if successfully logged in; 0 otherwise </TD>
<TD>discrete</TD>
</TR>
<TR>
<TD>num_compromised </TD>
<TD>number of ``compromised'' conditions </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>root_shell </TD>
<TD>1 if root shell is obtained; 0 otherwise </TD>
<TD>discrete</TD>
</TR>
<TR>
<TD>su_attempted </TD>
<TD>1 if ``su root'' command attempted; 0 otherwise </TD>
<TD>discrete</TD>
</TR>
<TR>
<TD>num_root </TD>
<TD>number of ``root'' accesses </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>num_file_creations </TD>
<TD>number of file creation operations </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>num_shells </TD>
<TD>number of shell prompts </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>num_access_files </TD>
<TD>number of operations on access control files </TD>
<TD>continuous</TD>
</TR>
<TR NOSAVE>
<TD>num_outbound_cmds</TD>
<TD NOSAVE>number of outbound commands in an ftp session </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>is_hot_login </TD>
<TD>1 if the login belongs to the ``hot'' list; 0 otherwise </TD>
<TD>discrete</TD>
</TR>
<TR>
<TD>is_guest_login </TD>
<TD>1 if the login is a ``guest''login; 0 otherwise </TD>
<TD>discrete</TD>
</TR>
<CAPTION ALIGN=BOTTOM>
<BR>Table 2: Content features within a connection suggested by domain knowledge.</CAPTION>
</TABLE></CENTER>
<CENTER><TABLE BORDER WIDTH="80%" NOSAVE >
<TR>
<TD><I>feature name</I></TD>
<TD><I>description </I></TD>
<TD><I>type</I></TD>
</TR>
<TR>
<TD>count </TD>
<TD>number of connections to the same host as the current connection in
the past two seconds </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD></TD>
<TD><I>Note: The following features refer to these same-host connections.</I></TD>
<TD></TD>
</TR>
<TR>
<TD>serror_rate </TD>
<TD>% of connections that have ``SYN'' errors </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>rerror_rate </TD>
<TD>% of connections that have ``REJ'' errors </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>same_srv_rate </TD>
<TD>% of connections to the same service </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>diff_srv_rate </TD>
<TD>% of connections to different services </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>srv_count </TD>
<TD>number of connections to the same service as the current connection
in the past two seconds </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD></TD>
<TD><I>Note: The following features refer to these same-service connections.</I></TD>
<TD></TD>
</TR>
<TR>
<TD>srv_serror_rate </TD>
<TD>% of connections that have ``SYN'' errors </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>srv_rerror_rate </TD>
<TD>% of connections that have ``REJ'' errors </TD>
<TD>continuous</TD>
</TR>
<TR>
<TD>srv_diff_host_rate </TD>
<TD>% of connections to different hosts </TD>
<TD>continuous </TD>
</TR>
<CAPTION ALIGN=BOTTOM>
<BR>Table 3: Traffic features computed using a two-second time window.</CAPTION>
</TABLE></CENTER>
</body>
</html>