Project

General

Profile

Feature #2048

Replication of the backup lookups

Added by Luke Murphey over 6 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
Start date:
09/15/2017
Due date:
% Done:

100%


lookup_editor.tgz (336 KB) Luke Murphey, 09/06/2017 05:04 PM


Subtasks

Task #2066: Get basic handler workingClosedLuke Murphey

Task #2067: Assign handler a capabilityRejectedLuke Murphey

Task #2068: Migrate in functions from controllerClosedLuke Murphey

Task #2069: Change front-end calls to handlerClosedLuke Murphey

Task #2070: Remove controllerClosed

Task #2071: Sourcetype the REST handler logsClosed

Task #2072: Create config settings to support SHCClosed

Associated revisions

Revision 346 (diff)
Added by lukemurphey over 6 years ago

Initial version of REST handler

Reference #2048

Revision 349 (diff)
Added by lukemurphey over 6 years ago

Adding support for retrieving lookup backups from the REST handler

Reference #2048

Revision 353 (diff)
Added by lukemurphey over 6 years ago

Migrating save to the REST handler

Reference #2048

Revision 354 (diff)
Added by lukemurphey over 6 years ago

Fixing issue where backups were not being persisted

Reference #2048

Revision 355 (diff)
Added by lukemurphey over 6 years ago

Consolidated the save functionality into the shared library

Reference #2048

History

#1 Updated by Luke Murphey over 6 years ago

#3 Updated by Luke Murphey over 6 years ago

The files are being stored in /etc/apps/lookup_editor/lookup_file_backups

#4 Updated by Luke Murphey over 6 years ago

I could handle this by using allowRestReplay from https://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Restmapconf:

allowRestReplay=[true|false]
* POST/PUT/DELETE requests can be replayed on other nodes in the deployment.
* This enables centralized management.
* Turn on or off this feature. You can also control replay at each endpoint
  level. This feature is currently INTERNAL and should not be turned on witout
  consulting splunk support.
* Defaults to false

#7 Updated by Luke Murphey over 6 years ago

APIs that need to be converted:

  1. get_lookup_info: Get information about a lookup file (owner, size, etc.)
  2. get_lookup_backups_list: Get a list of the lookup file backups rendered as JSON.
  3. save: save the JSON contents to the lookup file (this is the only one that must be converted to support replication)
  4. get_original_lookup_file: provides the lookup file in a way to be downloaded by the browser.
  5. get_lookup_contents: Provides the contents of a lookup file as JSON.

#8 Updated by Luke Murphey over 6 years ago

This code could use some breaking up (not including the public functions above):

  • Backups
    • get_backup_files (getBackupFiles)
    • get_backup_directory (getBackupDirectory)
    • backup_lookup_file (backupLookupFile)
  • Shortcuts
    • append_if_not_none
    • flatten_dict
    • make_lookup_filename (makeLookupFilename)
    • is_lookup_in_users_path (isLookupInUsersPath)
    • convert_array_to_csv
    • escape_filename (escapeFilename)
    • is_file_name_valid
  • Lookup
    • force_lookup_replication
    • getCapabilities4User
    • resolve_lookup_filename
    • get_kv_lookup
    • get_lookup (signature changed)
  • Controller specific
    • render_error_json

#9 Updated by Luke Murphey over 6 years ago

  • backup_lookup_file: function signature has changed (moved resolved_file_path)

#10 Updated by Luke Murphey over 6 years ago

I think I will need to break out the backups handler from the main handler so that allowRestReplay can be enabled only for backups.

#11 Updated by Luke Murphey over 6 years ago

Some things left to do:
  • Get backups to work
  • Clean up post_lookup_contents() which currently does too much and has to call lookupfiles directly

#12 Updated by Luke Murphey over 6 years ago

  • Status changed from New to Closed

Also available in: Atom PDF