Project

General

Profile

ThreatScript Definitions » History » Version 23

Luke Murphey, 05/28/2011 07:44 PM

1 22 Luke Murphey
{{>toc}}
2 22 Luke Murphey
3 1 Luke Murphey
h1. ThreatScript Definitions
4 1 Luke Murphey
5 1 Luke Murphey
ThreatScript Definitions are written in ECMAScript (basically the same as JavaScript). The ThreatScript definitions return a Result object which indicates whether a match was observed.
6 1 Luke Murphey
7 1 Luke Murphey
h2. ThreatScript Example
8 1 Luke Murphey
9 1 Luke Murphey
Below is an example of a ThreatScript that triggers if the web-page has a form element.
10 1 Luke Murphey
11 1 Luke Murphey
<pre><code class="javascript">
12 1 Luke Murphey
/*
13 1 Luke Murphey
 * Name: Example.General.Has_Form_Tag
14 1 Luke Murphey
 * Version: 1
15 1 Luke Murphey
 * ID: 1000000
16 1 Luke Murphey
 * Message: Indicates if the page has as a form tag
17 1 Luke Murphey
 * Severity: Low
18 11 Luke Murphey
 * Reference: url,threatfactor.com
19 1 Luke Murphey
 */
20 1 Luke Murphey
21 1 Luke Murphey
importPackage(Packages.ThreatScript);
22 1 Luke Murphey
importPackage(Packages.HTTP);
23 1 Luke Murphey
24 8 Luke Murphey
function analyze( httpResponse, variables, environment ){
25 1 Luke Murphey
26 1 Luke Murphey
	var parser = httpResponse.getDocumentParser();
27 1 Luke Murphey
	var location = new URL( httpResponse.getLocation() );
28 1 Luke Murphey
29 1 Luke Murphey
	//Get a list of all script tags
30 1 Luke Murphey
	var tagNameFilter = new TagNameFilter("form");
31 1 Luke Murphey
	var nodesList = parser.extractAllNodesThatMatch(tagNameFilter); 
32 1 Luke Murphey
        if( nodesList.size() > 0 ){
33 1 Luke Murphey
	     return new Result( true, "A form was detected" );
34 1 Luke Murphey
	}
35 1 Luke Murphey
        
36 1 Luke Murphey
	return new Result( false, "No forms detected" );
37 1 Luke Murphey
}
38 1 Luke Murphey
</code>
39 1 Luke Murphey
</pre>
40 2 Luke Murphey
41 3 Luke Murphey
h2. Analysis Function
42 3 Luke Murphey
43 12 Luke Murphey
ThreatScripts must provide an analyze function that takes 3 arguments:
44 3 Luke Murphey
45 7 Luke Murphey
| *Name*       | *Type*            | *Note*                                                                           |
46 7 Luke Murphey
| httpResponse | HttpResponseData  | See source:trunk/src/net/lukemurphey/nsia/scan/HttpResponseData.java             |
47 7 Luke Murphey
| variables    | Variables         | See source:trunk/src/net/lukemurphey/nsia/scan/scriptenvironment/Variables.java  |
48 7 Luke Murphey
| environment  | Environment       | See source:trunk/src/net/lukemurphey/nsia/scan/ScriptDefinition.java#L605        |
49 1 Luke Murphey
50 16 Luke Murphey
h2. Saving / Loading Saved Data (using the environment object)
51 16 Luke Murphey
52 16 Luke Murphey
ThreatScripts can save and load data using the environment object. The environment allows the script to recall information between runs of the definition for a given URL and rule.
53 16 Luke Murphey
54 16 Luke Murphey
h3. Retrieving Values
55 16 Luke Murphey
56 16 Luke Murphey
Values can be retrieved using the _get_ function of the environment object. Call _getValue_ on the returned object to get the value. Below is an example:
57 16 Luke Murphey
58 16 Luke Murphey
<pre><code class="javascript">
59 16 Luke Murphey
var saved = environment.get("ValueName");
60 16 Luke Murphey
var value = saved.getValue();
61 16 Luke Murphey
</code></pre>
62 16 Luke Murphey
63 16 Luke Murphey
h3. Saving Values
64 16 Luke Murphey
65 16 Luke Murphey
Values can be saved using the _set_ function of the environment object. Optionally, the set argument accepts a boolean indicating if the provided value should only be returned when the definition is running against the same URL. Below is an example:
66 16 Luke Murphey
67 16 Luke Murphey
<pre><code class="javascript">
68 16 Luke Murphey
var test = "1234ABCD";
69 17 Luke Murphey
70 17 Luke Murphey
//Will only be returned when the given definition is executed against the current URL:
71 17 Luke Murphey
environment.set("ValueName", test);
72 17 Luke Murphey
73 23 Luke Murphey
//Will be returned when the given definition is executed against any URL (within the given rule):
74 17 Luke Murphey
environment.set("ValueName", test, false);
75 16 Luke Murphey
</code></pre>
76 16 Luke Murphey
77 20 Luke Murphey
h3. Saving / Loading Collections
78 20 Luke Murphey
79 20 Luke Murphey
If your script needs to load collections then you ought to use the Vector class since it will perform the necessary conversions that are required to make sure that the contents are the same when loaded.
80 20 Luke Murphey
81 20 Luke Murphey
<pre><code class="javascript">
82 20 Luke Murphey
v = new Vector();
83 20 Luke Murphey
v.push("one");
84 20 Luke Murphey
v.get(0);
85 20 Luke Murphey
v.pop();
86 20 Luke Murphey
v.remove();
87 20 Luke Murphey
environment.set("Vector", test);
88 20 Luke Murphey
</code></pre>
89 20 Luke Murphey
90 1 Luke Murphey
h2. Baseline Function
91 1 Luke Murphey
92 4 Luke Murphey
ThreatScripts may declare a baseline function that will allow the definition to be configured to baseline itself against the previous set of scan results. The baseline function is called by NSIA when a user presses the baseline method for a rule. The objective of the baseline function is to view the provided scan results and ignore the particular finding for the given resource in the future. For example, a definition that triggers when the hash of the web-page changes may define a baseline function that causes it to not trigger unless the web-page hashes changes to yet another value.
93 4 Luke Murphey
94 4 Luke Murphey
Below is an example:
95 4 Luke Murphey
96 4 Luke Murphey
<pre>
97 4 Luke Murphey
<code class="javascript">
98 4 Luke Murphey
function baseline( environment ){
99 4 Luke Murphey
	var previousValue = environment.get("LastObservedHash");
100 4 Luke Murphey
101 4 Luke Murphey
	if( previousValue != null && previousValue.getValue() != null ){
102 4 Luke Murphey
		environment.set("Hash", previousValue.getValue() );
103 4 Luke Murphey
	}
104 4 Luke Murphey
105 4 Luke Murphey
	return true;
106 4 Luke Murphey
}
107 4 Luke Murphey
</code>
108 4 Luke Murphey
</pre>
109 4 Luke Murphey
110 21 Luke Murphey
h2. Link Extraction
111 21 Luke Murphey
112 21 Luke Murphey
ThreatScript definitions can add URLs to the list that will be scanned. This allows ThreatScripts to parse content and enumerate other content that is linked. URLs can be added by calling addURL on the Result object or by specifying the URLs in the Result constructor. See below for an example:
113 21 Luke Murphey
114 21 Luke Murphey
<pre>
115 21 Luke Murphey
<code class="javascript">
116 21 Luke Murphey
importPackage(Packages.ThreatScript);
117 21 Luke Murphey
importPackage(Packages.HTTP);
118 21 Luke Murphey
119 21 Luke Murphey
function analyze( httpResponse, operation, environment ){
120 21 Luke Murphey
    a = new Array();
121 21 Luke Murphey
    a[0] = new URL("http://mydomain.com/");
122 21 Luke Murphey
    a[1] = new URL("http://someotherdomain.com/");
123 21 Luke Murphey
124 21 Luke Murphey
    // You can add URLs via the results constructor (or add them manually via the "addURL" methods) 
125 21 Luke Murphey
    result = new Result( false, "Two URLs extracted", a);
126 21 Luke Murphey
127 21 Luke Murphey
    // This URL will not be scanned unless it matches the domain restriction in the rule
128 21 Luke Murphey
    result.addURL( new URL("http://mydomain.com/") );
129 21 Luke Murphey
130 21 Luke Murphey
    // This URL will be scanned regardless of whether it matches the domain restriction in the rule
131 21 Luke Murphey
    result.addURL( new URL("http://someotherdomain.com"), true);
132 21 Luke Murphey
    return result;
133 21 Luke Murphey
}
134 21 Luke Murphey
</code>
135 21 Luke Murphey
</pre>
136 21 Luke Murphey
137 21 Luke Murphey
138 14 Luke Murphey
h2. Terminate Function
139 3 Luke Murphey
140 14 Luke Murphey
ThreatScripts can also declare a terminate function that allows the scan engine to terminate the definition it execution exceeds the maximum execution time.
141 14 Luke Murphey
142 14 Luke Murphey
Below is an example of a (useless) rule that uses the terminate function to stop execution:
143 14 Luke Murphey
144 14 Luke Murphey
<pre>
145 14 Luke Murphey
<code class="javascript">
146 14 Luke Murphey
importPackage(Packages.ThreatScript);
147 14 Luke Murphey
148 14 Luke Murphey
var keep_going = true;
149 14 Luke Murphey
150 14 Luke Murphey
function analyze( httpResponse, operation, environment ){
151 14 Luke Murphey
152 14 Luke Murphey
    while(keep_going ){
153 14 Luke Murphey
        //Infinite loop
154 14 Luke Murphey
    }
155 14 Luke Murphey
156 14 Luke Murphey
    return new Result( false, "Definition did not match the input");
157 14 Luke Murphey
}
158 14 Luke Murphey
159 14 Luke Murphey
function terminate(){
160 14 Luke Murphey
    keep_going = false; //Will cause the analysis to stop
161 14 Luke Murphey
}
162 14 Luke Murphey
</code>
163 14 Luke Murphey
</pre>
164 3 Luke Murphey
165 1 Luke Murphey
h2. Meta-Data
166 1 Luke Murphey
167 1 Luke Murphey
ThreatScripts must provide a meta-data that indicates the following information:
168 1 Luke Murphey
169 3 Luke Murphey
| *Name*  | *Valid Input*                                      | *Notes* |
170 3 Luke Murphey
| Name    | <category>.<sub_category>.<definition_name>        |         |
171 3 Luke Murphey
| Version | integer                                            | Should be incremented each time the definition is updated |
172 3 Luke Murphey
| ID      | integer                                            | Must be 1000000 or greater (only official definitions can be less than 1000000)        |
173 3 Luke Murphey
| Message | message to be displayed when definition matches    |         |
174 3 Luke Murphey
| Severity| Either: Low, Medium or High                        |         |
175 3 Luke Murphey
| Invasive| Either: True or False (this argument is optional)  |         |
176 1 Luke Murphey
177 1 Luke Murphey
This meta-data is provided in a comment as name-value pairs (see above ThreatScript example).
178 1 Luke Murphey
179 9 Luke Murphey
h3. Definition Name
180 8 Luke Murphey
181 1 Luke Murphey
{{include(Definition_Naming_Convention)}}
182 1 Luke Murphey
183 9 Luke Murphey
h3. Definition Severity
184 9 Luke Murphey
185 9 Luke Murphey
{{include(Definition_Severity)}}
186 11 Luke Murphey
187 11 Luke Murphey
h3. Definition References 
188 11 Luke Murphey
189 11 Luke Murphey
{{include(Definition_References)}}
190 11 Luke Murphey
191 11 Luke Murphey
Note that definition references are defined a comment block with the "Reference:" as a prefix (Example: "// Reference: url,threatfactor.com").
192 9 Luke Murphey
193 3 Luke Murphey
h2. Available Packages
194 1 Luke Murphey
195 3 Luke Murphey
A series of packages are available to ThreatScripts in order to perform analysis.
196 3 Luke Murphey
197 1 Luke Murphey
| *Package*            | *Class*            | *Description*                                     |
198 13 Luke Murphey
|/9.HTTP               | URL                | Same as java.net.URL                              |
199 1 Luke Murphey
                       | TagNameFilter      | See http://htmlparser.sourceforge.net/javadoc/org/htmlparser/filters/TagNameFilter.html |
200 15 Luke Murphey
                       | GetRequest         | [[ThreatScript Web client|Web-client]] for performing HTTP Get requests  |
201 15 Luke Murphey
                       | PostRequest        | [[ThreatScript Web client|Web-client]] for performing HTTP Post requests |
202 15 Luke Murphey
                       | DeleteRequest      | [[ThreatScript Web client|Web-client]] for performing HTTP Delete requests |
203 15 Luke Murphey
                       | PutRequest         | [[ThreatScript Web client|Web-client]] for performing HTTP Put requests |
204 15 Luke Murphey
                       | TraceRequest       | [[ThreatScript Web client|Web-client]] for performing HTTP Trace requests |
205 15 Luke Murphey
                       | HeadRequest        | [[ThreatScript Web client|Web-client]] for performing HTTP Head requests |
206 15 Luke Murphey
                       | OptionsRequest     | [[ThreatScript Web client|Web-client]] for performing HTTP Option requests |
207 13 Luke Murphey
|/2.<default>          | StringUtils        | Provides a trim function for Strings, see source:/trunk/src/net/lukemurphey/nsia/scan/scriptenvironment/StringUtils.java              |
208 13 Luke Murphey
| Debug                | Provides method that allows scripts to create log messages, see source:/trunk/src/net/lukemurphey/nsia/scan/scriptenvironment/Debug.java              |
209 19 Luke Murphey
|/3.ThreatScript       | Result             | Indicates the results of analysis, see source:/trunk/src/net/lukemurphey/nsia/scan/scriptenvironment/Result.java                 |
210 19 Luke Murphey
                       | DataAnalysis       | Provides functions useful for analysis, see source:/trunk/src/net/lukemurphey/nsia/scan/ScriptSignatureUtils.java            |
211 18 Luke Murphey
                       | Vector             | Provides a storage class for scripts, see source:/trunk/src/net/lukemurphey/nsia/scan/scriptenvironment/Vector.java            |
212 1 Luke Murphey
213 1 Luke Murphey
214 1 Luke Murphey
h2. Debugging ThreatScripts
215 8 Luke Murphey
216 8 Luke Murphey
ThreatScripts can create event log messages by using the sendMessage() function in the Debug class. Simply call _Debug.sendMessage_ with a string as an argument to create an event log message. The event log messages can be viewed in the event log for NSIA.
217 8 Luke Murphey
218 8 Luke Murphey
Generally, script created log messages are used only for debugging and should be disabled on rules you want to use in production.
219 8 Luke Murphey
220 8 Luke Murphey
h2. General Notes When Writing Definitions
221 8 Luke Murphey
222 8 Luke Murphey
h3. ThreatScript Maximum Runtime
223 8 Luke Murphey
224 8 Luke Murphey
ThreatScript  definitions are forceably terminated by the scan engine if the script runs for longer than 10 seconds. Thus, it is important to write definitions that can complete within the timeframe alloted; otherwise, the definition will be flagged as having an error.
225 8 Luke Murphey
226 8 Luke Murphey
h3. Maximum Data Size
227 8 Luke Murphey
228 8 Luke Murphey
The scan engine only provides the first 1 MB of the data observed to the scan engine. Therefore, do not design ThreatScripts that won't work if only the first 1 MB of a larger file is provided.