Splunk Apps » Website Input

"Sometimes you might need to create an account and login to access the information you need. If you have a good HTTP library that handles logins and automatically sending session cookies (did I mention how awesome Requests is?), then you just need your scraper login before it gets to work."

http://blog.hartleybrody.com/web-scraping/

http://wwwsearch.sourceforge.net/mechanize/
http://stackoverflow.com/questions/11685235/login-using-python-in-basic-html-form

Might be able to support with this: https://github.com/lorien/grab

http://answers.splunk.com/answers/301772/website-input-how-do-i-monitor-a-forum-that-requir.html#answer-305006

https://answers.splunk.com/answers/339200/website-input-how-far-off-is-support-for-forms-bas.html

Might be able to make this work using mechanize (http://wwwsearch.sourceforge.net/mechanize/)

See http://stackoverflow.com/questions/4847226/form-based-authentication-with-python

http://wwwsearch.sourceforge.net/mechanize/

httplib2 doesn't seem to persist sessions very well, see:

• https://github.com/jcgregorio/httplib2/wiki/Examples
• https://stackoverflow.com/questions/923296/keeping-a-session-in-python-while-making-http-requests

Request is likely a better option: https://stackoverflow.com/questions/19566645/python-httplib2-https-login-fails

The user is going to need to define several things:

1. Authentication URL
2. Username
3. Password
4. Username field name
5. Password field name

1. How do I get the built-in client and the browser to both support this?
   1. Can the cookies be transferred?
2. What about CSRF protection?

A conf file might look like this:

authentication_url=https://domain.com/auth
username_field=username
password_field=password
authentication_action=POST

Tried mechanize. It actually works. Here is a Redmine login:

import mechanize

url = "http://Lukemurphey.net/login" 
br = mechanize.Browser()
br.set_handle_robots(False) # ignore robots
br.open(url)

br.select_form(nr=2)

br.form['username'] = 'Luke'
br.form['password'] = 'OPENSESAME'

res = br.submit()
content = res.read()

• mechanize: https://github.com/python-mechanize/mechanize
• html5lib 0.999999999: https://pypi.python.org/pypi/html5lib
• six 1.10: https://pypi.python.org/pypi/six
• webencodings 0.5.1: https://pypi.python.org/pypi/webencodings

To convert this over, I would need:

1. The ability to obtain the raw HTML in get_result_built_in_client()
   1. https://stackoverflow.com/questions/9552773/raw-html-vs-dom-scraping-in-python-using-mechanize-and-beautiful-soup
2. The ability to obtain the response code in get_result_built_in_client()
   1. https://stackoverflow.com/questions/11809696/python-mechanize-browser-openurl-status-code
3. The ability to assign a proxy server in get_http_client()
   1. Note that get_http_client() is not currently used by scrape page
   2. https://stackoverflow.com/questions/1997894/pythons-mechanize-proxy-support
4. The ability to assign HTTP credentials
   1. https://stackoverflow.com/questions/5291576/basic-and-form-authentication-with-mechanize-ruby
   2. https://stackoverflow.com/questions/40919543/python-mechanize-implementation-of-http-basic-auth
   3. https://stackoverflow.com/questions/1097380/can-python-mechanize-handle-http-auth
5. The ability to set the user-agent
   1. http://stockrt.github.io/p/emulating-a-browser-in-python-with-mechanize/

• Break out authentication from scrape_page() into a setAuthentication() call
• Put the HTTP client under an abstraction layer so that I can switch it out as I see fit

http://mechanize.readthedocs.io/en/latest/browser_api.html

Need to have some logging if the fields cannot be determined.