Splunk Apps » Website Input

1. Make a separate scraper class
2. Pass arguments as a config object
3. Pass arguments as *args or **kwargs

Here are the functions invoked by scrape_page():

• resolve_proxy_type
• get_result_single
  • get_result_built_in_client
    • detect_encoding
  • get_result_browser
    • get_display
    • get_firefox_profile
    • add_auth_to_url
  • unescape
  • escape_field_name
  • extract_links
    • cleanup_link

Here are the functions that are modular input specific:

• get_file_path
• run

I think the best approach is to

1. break out the modular input and leave everything else in the scraper
2. make the arguments to scrape_page class variables
3. updated the controller, search command, tests

What should be in the constructor call versus the command call?

• basics: url, username, password, etc. (in scrape_page call)
• output customization: output_matches_as_mv, output_matches_as_separate_fields, etc. (in constructor)
• charset detection: charset_detect_meta_enabled, etc.
• proxy info: proxy_type, etc.
• spider info: page_limit, etc. (in constructor)

1. Break out WebScraper class
2. Add constructor options for proxy info and charset
   1. Modify get_result_browser, get_result_single, get_result_built_in_client, get_http_client
3. Update search command
4. Update modular input
5. Update controller