Email Alerting¶

You can setup email alerting in Splunk such that you get notified when sites respond slowly or post errors.

Splunk Configuration¶

You will need to configure Splunk to work with an email server. To set this up, go the Splunk Manager at Server settings » Email settings.

Configuring the Search¶

The app includes a search named "website_performance_problems" that works well for email alerting. To configure it for email alerting, open the "website_performance_problems" search in the Manager (Manager » "Searches, reports, and alerts"). Click the checkbox next to "Send email" alert action to enable it and complete the information necessary to use the alert action.

Customizing Response Time Threshold¶

You may want to change the threshold that is used to determine if a site has been down too long. To do so, edit the "response_time_threshold" macro. You can change the macro in the Manager by going to "Advanced search" » "Search macros" and editing the "response_time_threshold" macro.

Filtering Out Sites¶

You may want not want to receive email notifications for some sites. To filter these sites out, add a where clause to the "website_performance_problems" search just after the first search part. Below is an example of a search using where clauses to filter out some sites:

sourcetype="web_ping" (response_code>=400 OR timed_out=True) OR (total_time>`response_time_threshold`) | where NOT like(url,"%splunk.com%") | fillnull response_code value="Connection failed" | eval response_code=if(timed_out == "True", "Connection timed out", response_code) | stats count as count max(total_time) as max_total_time by title url response_code | eval max_total_time=round(max_total_time, 2)." ms" 

Ignoring Connection Failures or Timeouts¶

You may want not want to receive email notifications for connection failures if your Splunk box is on a connection that tends to be unstable. To filter these sites out, add a where clause to the "website_performance_problems" search that will filter out a response_code of "Connection timed out" or "Connection failed". Below is an example of a search using a where clause to filter out connection failures:

sourcetype="web_ping" (response_code>=400 OR timed_out=True) OR (total_time>`response_time_threshold`) | fillnull response_code value="Connection failed" | eval response_code=if(timed_out == "True", "Connection timed out", response_code) | where NOT (response_code="Connection timed out" OR response_code="Connection failed") | stats count as count max(total_time) as max_total_time by title url response_code | eval max_total_time=round(max_total_time, 2)." ms"