Project

General

Profile

Feature #1726

Add ability to include empty matches

Added by Luke Murphey over 7 years ago. Updated about 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Dashboard: Input Editor
Target version:
Start date:
02/04/2017
Due date:
% Done:

100%


Description

Adding the ability to include empty matches would help in cases where I am using two selectors and I need to include empty matches so that I can match up the results accordingly (like in cases where the the results come in pairs).

For example, use the selector "td:nth-child(1), td:nth-child(5)" on http://downdetector.com/top10/. Notice how empty rows are not included making it difficult to know what trend the site is for.

Associated revisions

Revision 361 (diff)
Added by lukemurphey about 7 years ago

Adding ability to include empty matches

Reference #1726

Revision 368 (diff)
Added by lukemurphey about 7 years ago

Adding ability to include empty matches

Reference #1726

Revision 370 (diff)
Added by lukemurphey about 7 years ago

Updating unit tests to test empty matching

Reference #1726

History

#1 Updated by Luke Murphey about 7 years ago

Make sure to check:

  1. The manager page
  2. The main page: that it gets loaded and saved
  3. Preview works
  4. Search command works

#2 Updated by Luke Murphey about 7 years ago

  • % Done changed from 0 to 50

#3 Updated by Luke Murphey about 7 years ago

empty_matches only prevents entire fields that are blank from being included; it doesn't include blank matches between the separator, like with this example:

| webscrape selector="tr" url="http://downdetector.com/top10/" depth_limit=25 text_separator="::" empty_matches=1

I'm also not seeing the empty fields in the results and in the search command output.

#4 Updated by Luke Murphey about 7 years ago

I can get empty strings to be included with text separators in get_text() by not using append_if_not_empty().

#5 Updated by Luke Murphey about 7 years ago

I'm seeing that the raw_match_count is the expected value. It appears the values are getting stripped before being outputted somehow.

#6 Updated by Luke Murphey about 7 years ago

Need to fix btool errors too.

#7 Updated by Luke Murphey about 7 years ago

It looks like it still might not be including the empty matches:

| webscrape selector="tr.dot > td" url="http://downdetector.com/top10/" depth_limit=25 empty_matches=1
| table raw_match_count match

Note that the raw match count is less than the actual.

#8 Updated by Luke Murphey about 7 years ago

get_result_single() seems to be including all of the entries. It appears that Splunk is filtering them in splunk.Intersplunk.outputResults().

#9 Updated by Luke Murphey about 7 years ago

I might want to switch to have a default value when no value is found (like fullnull). It could be called: null_value, value_for_empty, empty_value.

#10 Updated by Luke Murphey about 7 years ago

Need to still:

  1. Add the option to the modular input page
  2. Add the option to the wizard page
  3. Make the search command use the option
  4. Make sure the preview window uses the option too
  5. Add the option to the search BNF
  6. Add the option to inputs.conf.spec
  7. Add test cases

#11 Updated by Luke Murphey about 7 years ago

  • Status changed from New to Closed
  • % Done changed from 50 to 100

Also available in: Atom PDF