Project

General

Profile

Email Alerting » History » Version 2

Luke Murphey, 06/18/2015 06:05 PM

1 1 Luke Murphey
h1. Email Alerting
2 1 Luke Murphey
3 1 Luke Murphey
You can setup email alerting in Splunk such that you get notified when sites respond slowly or post errors.
4 1 Luke Murphey
5 1 Luke Murphey
h2. Splunk Configuration
6 1 Luke Murphey
7 1 Luke Murphey
You will need to configure Splunk to work with an email server. To set this up, go the Splunk Manager at Server settings » Email settings.
8 1 Luke Murphey
9 1 Luke Murphey
h2. Configuring the Search
10 1 Luke Murphey
11 1 Luke Murphey
The app includes a search named "website_performance_problems" that works well for email alerting. To configure it for email alerting, open the "website_performance_problems" search in the Manager (Manager » "Searches, reports, and alerts"). Click the checkbox next to "Send email" alert action to enable it and complete the information necessary to use the alert action.
12 1 Luke Murphey
13 1 Luke Murphey
h2. Customizing Response Time Threshold
14 1 Luke Murphey
15 1 Luke Murphey
You may want to change the threshold that is used to determine if a site has been down too long. To do so, edit the "response_time_threshold" macro. You can change the macro in the Manager by going to "Advanced search" » "Search macros" and editing the "response_time_threshold" macro.
16 1 Luke Murphey
17 1 Luke Murphey
h2. Filtering Out Sites
18 1 Luke Murphey
19 1 Luke Murphey
You may want not want to receive email notifications for some sites. To filter these sites out, add a where clause to the "website_performance_problems" search just after the first search part. Below is an example of a search using where clauses to filter out some sites:
20 1 Luke Murphey
21 1 Luke Murphey
<pre>
22 2 Luke Murphey
sourcetype="web_ping" (response_code>=400 OR timed_out=True) OR (total_time>`response_time_threshold`) | where NOT like(url,"%splunk.com%") | fillnull response_code value="Connection failed" | eval response_code=if(timed_out == "True", "Connection timed out", response_code) | stats count as count max(total_time) as max_total_time by title url response_code | eval max_total_time=round(max_total_time, 2)." ms"
23 2 Luke Murphey
</pre>
24 2 Luke Murphey
25 2 Luke Murphey
h2. Ignoring Connection Failures or Timeouts
26 2 Luke Murphey
27 2 Luke Murphey
You may want not want to receive email notifications for connection failures if your Splunk box is on a connection that tends to be unstable. To filter these sites out, add a where clause to the "website_performance_problems" search that will filter out a response_code of "Connection timed out" or "Connection failed". Below is an example of a search using a where clause to filter out connection failures:
28 2 Luke Murphey
29 2 Luke Murphey
<pre>
30 2 Luke Murphey
sourcetype="web_ping" (response_code>=400 OR timed_out=True) OR (total_time>`response_time_threshold`) | fillnull response_code value="Connection failed" | eval response_code=if(timed_out == "True", "Connection timed out", response_code) | where NOT (response_code="Connection timed out" OR response_code="Connection failed") | stats count as count max(total_time) as max_total_time by title url response_code | eval max_total_time=round(max_total_time, 2)." ms"
31 1 Luke Murphey
</pre>