Splunk Apps » Website Input

Here is the script I used:

from selenium import webdriver
browser = webdriver.Firefox()
browser.get('http://textcritical.net/works')

import time
time.sleep(3)

tags = browser.find_elements_by_css_selector("table")
for t in tags:
   print t.text

#browser.page_source

browser.close()

See https://automatetheboringstuff.com/chapter11/

http://stackoverflow.com/questions/7861775/python-selenium-accessing-html-source

Looks like I need to use:

webdriver.execute_script("return document.documentElement.outerHTML")

http://stackoverflow.com/questions/26584215/selenium-page-source-does-not-return-modified-dom-tree

Need to have some method of knowing when content is ready. Options include:

1. When document.ready is called
2. When the selector matches
3. On a timeout

See http://www.obeythetestinggoat.com/how-to-get-selenium-to-wait-for-page-load-after-a-click.html
http://stackoverflow.com/questions/11001030/how-i-can-check-whether-a-page-is-loaded-completely-or-not-in-web-driver/11002061#11002061

Apparently, IE and Chrome need another driver:

• https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver
• https://sites.google.com/a/chromium.org/chromedriver/downloads

I might be able to get the Chrome driver to work by just including the binaries in the app (http://stackoverflow.com/questions/13724778/how-to-run-selenium-webdriver-test-cases-in-chrome).

• response code
• encoding

(index=* sourcetype=web_input) OR (index=_internal sourcetype=web_input*)

Need to make sure the browser parameter down to the args. Crawling might not work yet.