Project

General

Profile

Troubleshooting » History » Version 7

Luke Murphey, 05/01/2017 05:17 PM

1 1 Luke Murphey
h1. Troubleshooting
2 1 Luke Murphey
3 1 Luke Murphey
Below are some pointers to help in debugging the application:
4 1 Luke Murphey
5 1 Luke Murphey
h2. Search for modular input errors
6 1 Luke Murphey
7 1 Luke Murphey
Run a search for the following and see if the logs indicate a problem with the modular input:
8 1 Luke Murphey
9 1 Luke Murphey
<pre>
10 1 Luke Murphey
index=_internal sourcetype=web_availability_modular_input
11 1 Luke Murphey
</pre>
12 1 Luke Murphey
13 4 Luke Murphey
14 4 Luke Murphey
h2. Search for tracebacks
15 4 Luke Murphey
16 4 Luke Murphey
Run a search for the following and see if the logs indicate a crash with the modular input:
17 4 Luke Murphey
18 4 Luke Murphey
<pre>
19 4 Luke Murphey
index=_internal traceback
20 4 Luke Murphey
</pre>
21 4 Luke Murphey
22 4 Luke Murphey
23 1 Luke Murphey
h2. Search for REST handler errors
24 1 Luke Murphey
25 1 Luke Murphey
Run a search for the following and see if the logs indicate a problem with the REST handler:
26 1 Luke Murphey
27 1 Luke Murphey
<pre>
28 1 Luke Murphey
index=_internal source=*website_monitoring_rest_handler.log
29 1 Luke Murphey
</pre>
30 2 Luke Murphey
31 2 Luke Murphey
h2. Input fails and generates a "ConnectionError"
32 2 Luke Murphey
33 2 Luke Murphey
This can happen when the following is
34 2 Luke Murphey
35 2 Luke Murphey
# The IP address is invalid (if referring to the host by IP)
36 2 Luke Murphey
# The domain name cannot be resolved
37 2 Luke Murphey
# The port number provided is invalid (if a port is specified)
38 2 Luke Murphey
# Network connectivity is lost or a firewall blocks the connection
39 3 Luke Murphey
40 3 Luke Murphey
h2. Troubleshooting modular inputs that don't appear in the Splunk Manager
41 3 Luke Murphey
42 3 Luke Murphey
Look in the messages link in the top right of the page to see if you have a message like this:
43 3 Luke Murphey
44 3 Luke Murphey
<pre>
45 3 Luke Murphey
Unable to initialize modular input "web_ping" defined inside the app "website_monitoring": Introspecting scheme=web_input: script running failed (exited with code 1).
46 3 Luke Murphey
</pre>
47 3 Luke Murphey
48 3 Luke Murphey
You can search the internal log for related errors that indicate the modular input could not start:
49 3 Luke Murphey
50 3 Luke Murphey
<pre>
51 3 Luke Murphey
index=_internal "ModularInputs - Unable to initialize modular input"
52 3 Luke Murphey
</pre>
53 3 Luke Murphey
54 3 Luke Murphey
The best way to identify the source of the error if the modular input cannot be loaded by Splunk is to try to load it via Splunk's python interpreter directly.
55 3 Luke Murphey
56 3 Luke Murphey
Here is an example. Consider the modular input is in the following path:  /opt/splunk/etc/apps/website_monitoring/bin/web_ping.py
57 3 Luke Murphey
58 3 Luke Murphey
To test this, go to the path /opt/splunk/etc/apps/website_monitoring/bin/ using the CLI:
59 3 Luke Murphey
60 3 Luke Murphey
<pre>
61 3 Luke Murphey
cd /opt/splunk/etc/apps/website_monitoring/bin/
62 3 Luke Murphey
</pre>
63 3 Luke Murphey
64 3 Luke Murphey
Next, run Splunk's python against the file:
65 3 Luke Murphey
66 3 Luke Murphey
<pre>
67 3 Luke Murphey
/opt/splunk/bin/splunk cmd python web_ping.py
68 3 Luke Murphey
</pre>
69 3 Luke Murphey
70 3 Luke Murphey
This should show an error, such as:
71 3 Luke Murphey
72 3 Luke Murphey
<pre>
73 3 Luke Murphey
Traceback (most recent call last):
74 3 Luke Murphey
  File "web_input.py", line 1, in <module>
75 3 Luke Murphey
    somethingnotdefined
76 3 Luke Murphey
NameError: name 'somethingnotdefined' is not defined
77 3 Luke Murphey
</pre>
78 5 Luke Murphey
79 5 Luke Murphey
h2. Determining the cause of errors relating to "splunkd connection error"
80 5 Luke Murphey
81 5 Luke Murphey
This happens when the input is running but Splunkd fails to respond when the input attempts to determine if a proxy server was configured. This appears to happen to Splunk installs that are running fine. In most cases that I can verify, it happens when Splunk has just been started. These can likely be safely ignored unless other issues have been observed that are related or if you see a large number of them.
82 6 Luke Murphey
83 6 Luke Murphey
h2. Authentication is failing against sites that use Windows Authentication (a.k.a. NTLM authentication)
84 6 Luke Murphey
85 6 Luke Murphey
Search the Website Monitoring logs for information that indicates which authentication type is being used. The app will attempt to discover the authentication type automatically. If this fails, then authentication will not succeed.
86 6 Luke Murphey
87 6 Luke Murphey
<pre>
88 6 Luke Murphey
| search index=_internal sourcetype="web_availability_modular_input" auth_type
89 6 Luke Murphey
</pre>
90 7 Luke Murphey
91 7 Luke Murphey
h2. Why are newly added URLs triggering continuously irrespective of Interval time? Or, how do I fix errors "Failed to save checkpoint directory"?
92 7 Luke Murphey
93 7 Luke Murphey
This can happen when the app doesn't have access to the modular input checkpoint directory. Check the permissions of the modular input checkpoint directory (e.g. "C:\Program Files\Splunk\var\lib\splunk\modinputs\web_ping") to see if the account that Splunk runs under has read/write access to it.
94 7 Luke Murphey
95 7 Luke Murphey
If it doesn't, edit the permissions. The input should begin functioning correctly as soon as the permissions are set correctly.