Project

General

Profile

Feature #2140

Allow high volume of pings in the modular input

Added by Luke Murphey about 4 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
Start date:
11/09/2017
Due date:
% Done:

100%

Associated revisions

Revision 208 (diff)
Added by lukemurphey about 4 years ago

Updating the modular input to handle multiple threads

Reference #2140

History

#1 Updated by Luke Murphey about 4 years ago

I'm thinking I should migrate the threading functions into the base class.

What I could move:
  1. Thread limiting check
  2. Threads list
  3. Thread limit var
  4. Thread cleanup function
  5. App config getter system

#2 Updated by Luke Murphey about 4 years ago

input_template = '''
[ping://192.168.0.%s]
hosts = 192.168.0.%s
interval = 30s
runs = 1
'''

inputs = ''

for i in range(1, 201):
    inputs += input_template % (i, i)

f = file('/Users/lmurphey/Desktop/inputs.conf', 'w')
f.write(inputs)
f.close()

#3 Updated by Luke Murphey about 4 years ago

Observations:
  1. The current design multi-threads the inputs but not the hosts within a single input
  2. I see lots of logs noting that the thread limit was reached. This happens because the input goes over each input and tries to check the thread limit again.

#4 Updated by Luke Murphey about 4 years ago

According to #2135, I need to support about 8 inputs per second.

#5 Updated by Luke Murphey about 4 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

#6 Updated by Luke Murphey about 4 years ago

This input was able to handle a thread limit of 200 and was able to keep up with 200 inputs that were set to run every 30 seconds. The memory use is around 37 MB and I am seeing around 900 results per minute (15 per second).

Also available in: Atom PDF