-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathdocs.html
170 lines (151 loc) · 8.82 KB
/
docs.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Getting Started with TEMPO</title>
<style>
body {
font-family: Arial, sans-serif;
line-height: 1.6;
color: #e0e0e0;
background-color: #121212;
max-width: 800px;
margin: 0 auto;
padding: 20px;
}
h1, h2, h3 {
color: #8b0000;
}
code {
background-color: #1e1e1e;
color: #d4d4d4;
padding: 2px 4px;
border-radius: 4px;
}
pre {
background-color: #1e1e1e;
color: #d4d4d4;
padding: 10px;
border-radius: 4px;
overflow-x: auto;
}
.section {
margin-bottom: 30px;
}
</style>
</head>
<body>
<h1>Getting Started with TEMPO (Using Sample Data)</h1>
<p>This guide will walk you through the process of setting up and using the TEMPO Native App in your Snowflake environment with provided sample data. The demo dataset is netflow from approximately 500 boxes on a corporate network, running for one week.</p>
<div class="section">
<h2>Cost of Running Data</h2>
<p>[Insert cost information here]</p>
</div>
<div class="section">
<h2>Architecture Diagram</h2>
<p><img width="1000" src="https://github.com/user-attachments/assets/a7c0ed15-521b-40d5-903e-c083a0a68e62"></p>
</div>
<div class="section">
<h2>Step 1: Install the TEMPO Native App</h2>
<ol>
<li>Obtain the TEMPO native app from the Snowflake Marketplace.</li>
<li>Once installed, the app will be available for use in your Snowflake environment.</li>
<li>Grant the app privileges to create the required compute resources:</li>
</ol>
<pre><code>GRANT CREATE COMPUTE POOL ON ACCOUNT TO APPLICATION TEMPO;
GRANT CREATE WAREHOUSE ON ACCOUNT TO APPLICATION TEMPO;
GRANT EXECUTE TASK ON ACCOUNT TO APPLICATION TEMPO;
GRANT EXECUTE MANAGED TASK ON ACCOUNT TO APPLICATION TEMPO;</code></pre>
<p>It's recommended to create a dedicated warehouse for the application with the following properties:</p>
<pre><code>CREATE WAREHOUSE <YOUR_WAREHOUSE_NAME>
WAREHOUSE_TYPE = 'SNOWPARK-OPTIMIZED'
WAREHOUSE_SIZE = 'LARGEMEDIUM' ;
GRANT USAGE ON WAREHOUSE <YOUR_WAREHOUSE_NAME> TO APPLICATION TEMPO;</code></pre>
<p>The application comes with its own in-house warehouse (TEMPO_WH), which will be used for container services and live inference only.</p>
</div>
<div class="section">
<h2>Step 2: Initialize the Application</h2>
<p>Call the startup procedure to initialize the app:</p>
<pre><code>CALL TEMPO.MANAGER.STARTUP();</code></pre>
</div>
<div class="section">
<h2>Step 3: Grant Permissions for Sample Data Access</h2>
<p>Grant the TEMPO app the necessary permissions to access the sample data:</p>
<pre><code>GRANT USAGE ON DATABASE tempo_sample TO APPLICATION TEMPO;
GRANT USAGE ON SCHEMA tempo_sample.inference_samples TO APPLICATION TEMPO;
GRANT USAGE ON SCHEMA tempo_sample.training_samples TO APPLICATION TEMPO;
GRANT SELECT ON ALL TABLES IN SCHEMA tempo_sample.inference_samples TO APPLICATION TEMPO;
GRANT SELECT ON ALL TABLES IN SCHEMA tempo_sample.training_samples TO APPLICATION TEMPO;</code></pre>
</div>
<div class="section">
<h2>Step 4: Perform Inference (TempoInference)</h2>
<p>Use the <code>TEMPOINFERENCE</code> stored procedure to perform inference on sample static log data. It takes the table's FQN (fully qualified name), a Boolean to control whether full logs or only anomalies are returned, and the log type, in this order.</p>
<pre><code>CALL TEMPO.DETECTION.TEMPOINFERENCE('<TABLE_FQN>',<RETURN BOOLEAN>, '<LOG_TYPE>');</code></pre>
<p>Example:</p>
<pre><code>CALL TEMPO.DETECTION.TEMPOINFERENCE('tempo_sample.inference_samples.workstations_inference', True, 'workstation');
CALL TEMPO.DETECTION.TEMPOINFERENCE('tempo_sample.inference_samples.webservers_inference',False,'webserver');</code></pre>
</div>
<div class="section">
<h2>Optional Step: Fine-Tuning Models</h2>
<p>If you need to adjust the models for better performance with your specific data patterns, you can use the fine-tuning feature. This step is optional and should be performed only if the default models don't meet your specific needs.</p>
<h3>Step 5a: Grant Permissions for Fine-Tuning</h3>
<p>If you decide to fine-tune, you will be making use of the in-house compute pool with GPU access and the following configuration:</p>
<p>[Configuration details here]</p>
<h3>Step 5b: Fine-Tune Models</h3>
<p>The ideal fine-tuning sequence based on data availability is Device before Anomaly, so that the Anomaly model can use the Device outputs. To fine-tune the models, you would execute the following queries:</p>
<pre><code>CALL TEMPO.FINE_TUNE.DEVICE_MODEL('<SERVICE_NAME>');
CALL TEMPO.FINE_TUNE.WORKSTATION_MODEL('<SERVICE_NAME>');</code></pre>
<p>Example:</p>
<pre><code>CALL TEMPO.FINE_TUNE.DEVICE_MODEL('first device model fine tuning');</code></pre>
<h3>Step 5c: Monitor Fine-Tuning Services</h3>
<p>Check the status of fine-tuning services:</p>
<pre><code>CALL SYSTEM$GET_SERVICE_STATUS('<FINE_TUNE_SERVICE_NAME>');</code></pre>
<p>Example:</p>
<pre><code>CALL SYSTEM$GET_SERVICE_STATUS('FINE_TUNE.sample_workstation');</code></pre>
<p>Note: Fine-tuning can be a time-consuming process and may incur additional computational costs. Only proceed with fine-tuning if you have specific requirements that aren't met by the default models.</p>
</div>
<div class="section">
<h2>Step 6: Live Inference</h2>
<h3>Step 6a: Setting Up Live Inference</h3>
<p>To start live inference, you need to initialize the necessary components such as streams, tasks, and live update tables. Before that, you need to allow <code>change tracking</code> on the specific table you would like live inference on:</p>
<pre><code>ALTER TABLE '<TABLE_NAME>' SET CHANGE_TRACKING = TRUE;</code></pre>
<h3>Step 6b: Start Live Inference</h3>
<p>To start the live inference process, use the <code>START_LIVE_INFERENCE</code> procedure. This procedure sets up a data stream, creates a live updates table, and schedules a task that continuously monitors incoming data and applies the inference model.</p>
<pre><code>CALL TEMPO.LIVE_INFERENCE.START_LIVE_INFERENCE('<TABLE_PATH>', '<MODEL_TYPE>', '<REFRESH_TIME>');</code></pre>
<p>Example:</p>
<pre><code>CALL TEMPO.LIVE_INFERENCE.START_LIVE_INFERENCE('mydb.myschema.mytable', 'workstation', '1 M');</code></pre>
<h3>Step 6c: Monitor, Suspend, and Resume Task</h3>
<p>After creating a task, the task name should be visible in the Snowflake interface.</p>
<ul>
<li>Monitoring tasks:</li>
</ul>
<pre><code>SELECT *
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
TASK_NAME => '<Task_name>'
));</code></pre>
<ul>
<li>Suspending Task:</li>
</ul>
<pre><code>ALTER TASK <Task_name> SUSPEND;</code></pre>
<ul>
<li>Resuming tasks:</li>
</ul>
<pre><code>ALTER TASK <Task_name> RESUME;</code></pre>
<h3>Step 6d: Shutting Down Live Inference</h3>
<p>When you need to stop live inference, use the <code>SHUTDOWN_LIVE_INFERENCE</code> procedure to safely terminate the process.</p>
<pre><code>CALL TEMPO.LIVE_INFERENCE.SHUTDOWN_LIVE_INFERENCE('<TABLE_NAME>');</code></pre>
<p>Example:</p>
<pre><code>CALL TEMPO.LIVE_INFERENCE.SHUTDOWN_LIVE_INFERENCE('mytable');</code></pre>
<p>After live inference is set up, review the results stored in the live updates table. This table will contain the incoming data along with the model's classification.</p>
<pre><code>SELECT * FROM TEMPO.LIVE_INFERENCE.LIVE_UPDATES_ON_<TABLE_NAME>;</code></pre>
<p>Example:</p>
<pre><code>SELECT * FROM TEMPO.LIVE_INFERENCE.LIVE_UPDATES_ON_mytable;</code></pre>
</div>
<div class="section">
<h2>Additional Notes</h2>
<p><strong>Performance Considerations:</strong> Live inference tasks can consume significant resources. Ensure that your Snowflake environment is properly scaled to handle the workload.</p>
<p>This guide uses sample data provided with TEMPO. For using your own data, please refer to the custom data guide.</p>
</div>
</body>
</html>