Troubleshooting » History » Version 6
Luke Murphey, 02/22/2017 06:04 AM
1 | 1 | Luke Murphey | h1. Troubleshooting |
---|---|---|---|
2 | 1 | Luke Murphey | |
3 | 1 | Luke Murphey | Below are some pointers to help in debugging the application: |
4 | 1 | Luke Murphey | |
5 | 1 | Luke Murphey | h2. Search for modular input errors |
6 | 1 | Luke Murphey | |
7 | 1 | Luke Murphey | Run a search for the following and see if the logs indicate a problem with the modular input: |
8 | 1 | Luke Murphey | |
9 | 1 | Luke Murphey | <pre> |
10 | 1 | Luke Murphey | index=_internal sourcetype=web_availability_modular_input |
11 | 1 | Luke Murphey | </pre> |
12 | 1 | Luke Murphey | |
13 | 4 | Luke Murphey | |
14 | 4 | Luke Murphey | h2. Search for tracebacks |
15 | 4 | Luke Murphey | |
16 | 4 | Luke Murphey | Run a search for the following and see if the logs indicate a crash with the modular input: |
17 | 4 | Luke Murphey | |
18 | 4 | Luke Murphey | <pre> |
19 | 4 | Luke Murphey | index=_internal traceback |
20 | 4 | Luke Murphey | </pre> |
21 | 4 | Luke Murphey | |
22 | 4 | Luke Murphey | |
23 | 1 | Luke Murphey | h2. Search for REST handler errors |
24 | 1 | Luke Murphey | |
25 | 1 | Luke Murphey | Run a search for the following and see if the logs indicate a problem with the REST handler: |
26 | 1 | Luke Murphey | |
27 | 1 | Luke Murphey | <pre> |
28 | 1 | Luke Murphey | index=_internal source=*website_monitoring_rest_handler.log |
29 | 1 | Luke Murphey | </pre> |
30 | 2 | Luke Murphey | |
31 | 2 | Luke Murphey | h2. Input fails and generates a "ConnectionError" |
32 | 2 | Luke Murphey | |
33 | 2 | Luke Murphey | This can happen when the following is |
34 | 2 | Luke Murphey | |
35 | 2 | Luke Murphey | # The IP address is invalid (if referring to the host by IP) |
36 | 2 | Luke Murphey | # The domain name cannot be resolved |
37 | 2 | Luke Murphey | # The port number provided is invalid (if a port is specified) |
38 | 2 | Luke Murphey | # Network connectivity is lost or a firewall blocks the connection |
39 | 3 | Luke Murphey | |
40 | 3 | Luke Murphey | h2. Troubleshooting modular inputs that don't appear in the Splunk Manager |
41 | 3 | Luke Murphey | |
42 | 3 | Luke Murphey | Look in the messages link in the top right of the page to see if you have a message like this: |
43 | 3 | Luke Murphey | |
44 | 3 | Luke Murphey | <pre> |
45 | 3 | Luke Murphey | Unable to initialize modular input "web_ping" defined inside the app "website_monitoring": Introspecting scheme=web_input: script running failed (exited with code 1). |
46 | 3 | Luke Murphey | </pre> |
47 | 3 | Luke Murphey | |
48 | 3 | Luke Murphey | You can search the internal log for related errors that indicate the modular input could not start: |
49 | 3 | Luke Murphey | |
50 | 3 | Luke Murphey | <pre> |
51 | 3 | Luke Murphey | index=_internal "ModularInputs - Unable to initialize modular input" |
52 | 3 | Luke Murphey | </pre> |
53 | 3 | Luke Murphey | |
54 | 3 | Luke Murphey | The best way to identify the source of the error if the modular input cannot be loaded by Splunk is to try to load it via Splunk's python interpreter directly. |
55 | 3 | Luke Murphey | |
56 | 3 | Luke Murphey | Here is an example. Consider the modular input is in the following path: /opt/splunk/etc/apps/website_monitoring/bin/web_ping.py |
57 | 3 | Luke Murphey | |
58 | 3 | Luke Murphey | To test this, go to the path /opt/splunk/etc/apps/website_monitoring/bin/ using the CLI: |
59 | 3 | Luke Murphey | |
60 | 3 | Luke Murphey | <pre> |
61 | 3 | Luke Murphey | cd /opt/splunk/etc/apps/website_monitoring/bin/ |
62 | 3 | Luke Murphey | </pre> |
63 | 3 | Luke Murphey | |
64 | 3 | Luke Murphey | Next, run Splunk's python against the file: |
65 | 3 | Luke Murphey | |
66 | 3 | Luke Murphey | <pre> |
67 | 3 | Luke Murphey | /opt/splunk/bin/splunk cmd python web_ping.py |
68 | 3 | Luke Murphey | </pre> |
69 | 3 | Luke Murphey | |
70 | 3 | Luke Murphey | This should show an error, such as: |
71 | 3 | Luke Murphey | |
72 | 3 | Luke Murphey | <pre> |
73 | 3 | Luke Murphey | Traceback (most recent call last): |
74 | 3 | Luke Murphey | File "web_input.py", line 1, in <module> |
75 | 3 | Luke Murphey | somethingnotdefined |
76 | 3 | Luke Murphey | NameError: name 'somethingnotdefined' is not defined |
77 | 3 | Luke Murphey | </pre> |
78 | 5 | Luke Murphey | |
79 | 5 | Luke Murphey | h2. Determining the cause of errors relating to "splunkd connection error" |
80 | 5 | Luke Murphey | |
81 | 5 | Luke Murphey | This happens when the input is running but Splunkd fails to respond when the input attempts to determine if a proxy server was configured. This appears to happen to Splunk installs that are running fine. In most cases that I can verify, it happens when Splunk has just been started. These can likely be safely ignored unless other issues have been observed that are related or if you see a large number of them. |
82 | 6 | Luke Murphey | |
83 | 6 | Luke Murphey | h2. Authentication is failing against sites that use Windows Authentication (a.k.a. NTLM authentication) |
84 | 6 | Luke Murphey | |
85 | 6 | Luke Murphey | Search the Website Monitoring logs for information that indicates which authentication type is being used. The app will attempt to discover the authentication type automatically. If this fails, then authentication will not succeed. |
86 | 6 | Luke Murphey | |
87 | 6 | Luke Murphey | <pre> |
88 | 6 | Luke Murphey | | search index=_internal sourcetype="web_availability_modular_input" auth_type |
89 | 6 | Luke Murphey | </pre> |