Splunk Apps » Website Input

This may be a good search command. With a search command, users could set up some processing to do things like get certain page numbers. For example, if I wanted to pull notes from Bible.com (https://www.bible.com/users/LukeMurphey/notes), then I would need to increment the page number variable to get the next page.

See http://answers.splunk.com/answers/239251/what-is-the-best-way-to-monitor-a-web-page-contain.html

http://www.xavierdupre.fr/blog/2013-10-27_nojs.html

To simplify, I think I could just use the main URL as the seed URL and then have a separate section for spidering.

The spider section would include:

• URL filter
• Depth limit
• URL limit

A good test case is Ryobi Tools. In that case, I want it to extract the content from https://www.ryobitools.com/outdoor/products/list/family/one-plus and then go to the pages. I need to have the app distinguish between pages that are identical but just have a different page number.

Could use Etags but these are not necessarily provided by all servers.

Could you use Scrapy: http://scrapy.org/download/

I will likely need to break up the scrape_page function up. It currently:

1. Preps and validates the arguments
2. Resolves the proxy configuration
3. Resolves the user-agent string
4. Performs the HTTP request, records the stats regarding load time, size
5. Resolves the encoding and decodes the content
6. Parses the HTML
7. Runs the selector against the HTML
8. Outputs the matches

Question about using scrapy:

• Can apply a limit?
  • http://stackoverflow.com/questions/19160594/scrapy-limit-the-number-of-request-or-request-bytes
• How are links extracted?
• How to handle proxies?
  • http://stackoverflow.com/questions/4710483/scrapy-and-proxies
• How to set user-agent string?
  • http://stackoverflow.com/questions/18920930/scrapy-python-set-up-user-agent
  • http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Spider.custom_settings
• How is encoding detected?
  • http://stackoverflow.com/questions/10735836/scrapy-spider-dealing-with-pages-that-have-incorrectly-defined-character-encodi
• What types of proxy servers does scrapy support?
  • How can authentication be handled?

• Write extractors
• Loop on the results