Exclude files with matching sigs #6
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…plicates and truncations
fabioroger
requested changes
Feb 3, 2017
sender
Outdated
| f.close(); | ||
|
|
||
| finally: | ||
| f.close(); |
There was a problem hiding this comment.
file might be closed twice.. possibly resulting in an exception
sender
Outdated
| log("Exception: %s" % e) | ||
| log("METRIC ns=forwarder.error.preprocess file=\"%s\" exception=\"%s\"" % (fn, str(e).replace("\n", ""))) | ||
| log("Exception=\"%s\"" % e) | ||
| f.close(); |
There was a problem hiding this comment.
file might not be open at this point. to guarantee it, put the f = open before try:
sender
Outdated
| try: | ||
| if not os.path.isfile(fn): | ||
| log("File no longer exists: %s" % fn) | ||
| continue | ||
| if os.path.getsize(fn) < self.signatureLength and not self.isCompressed(fn): | ||
| log("Skipping as sig too short or compressed: %s" % fn) |
There was a problem hiding this comment.
isn't the message here 'skipping as file to short and not compressed'?
sender
Outdated
| @@ -319,6 +331,7 @@ class Tail(object): | |||
| if should_stop(): | |||
| return | |||
| for line in f: | |||
| debug("line %s" % line.replace("\n","")) | |||
There was a problem hiding this comment.
calls like this in production will first resolve the string template, only to find out the debug is OFF... so the ideal, for performance in production, is to:
if debug_on: debug(...)
aggregate-access-log
Outdated
| @@ -0,0 +1,217 @@ | |||
| #!/usr/bin/env awk -f | |||
There was a problem hiding this comment.
is this file part of this PR?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is help cope with major issue where duplicates get processed multiple times. It should ignore any files when there are more than 1 with the same signature