Project

General

Profile

Task #1309

Feature #762: Web spider support

Task #1305: Update input to recursively spider websites

Implement extracted link processing

Added by Luke Murphey over 8 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Luke Murphey
Category:
-
Target version:
-
Start date:
05/01/2016
Due date:
% Done:

100%

Description

Need to ensure that links already processed are not processed twice

History

#1 Updated by Luke Murphey over 8 years ago

Need to figure out how to determine if more links exist that need processing:

  1. Use a dictionary and track if the link was processed already
  2. Do a diff between the processed list and the discovered list each time

#2 Updated by Luke Murphey over 8 years ago

Only issue is that I am not getting the same number of links every time.

#3 Updated by Luke Murphey over 8 years ago

  • % Done changed from 0 to 50

#4 Updated by Luke Murphey over 8 years ago

  • % Done changed from 50 to 100

#5 Updated by Luke Murphey over 8 years ago

  • Status changed from New to Closed

Also available in: Atom PDF