Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IP_ADDRESS_CACHE cache improvements #501

Merged
merged 1 commit into from
Apr 1, 2024
+8 −8
Merged

IP_ADDRESS_CACHE cache improvements #501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion parsedmarc/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
MAGIC_XML = b"\x3c\x3f\x78\x6d\x6c\x20"
MAGIC_JSON = b"\7b"

IP_ADDRESS_CACHE = ExpiringDict(max_len=10000, max_age_seconds=1800)
IP_ADDRESS_CACHE = ExpiringDict(max_len=10000, max_age_seconds=14400)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may worth making this a config file setting, defaulting to the initial 1800, as different use-cases may warrant different values.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, making it a config file setting may be a good enhancement, but it may take me a while to figure out how to do it. I can take a look at it after the holiday.

I don't think that there are currently many people locked into the 1,800 setting since it was not working at all until last week.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If someone would like to contribute a PR to add that option, be aware that although IP_ADDRESS_CACHE is only referenced twice:

https://github.com/domainaware/parsedmarc/blob/master/parsedmarc/__init__.py#L103

https://github.com/domainaware/parsedmarc/blob/master/parsedmarc/__init__.py#L897

configuration items related to report processing must go through several layers of function calls, such as when I just added the new function calls to address the concerns raised in #500.

041296b...acef7bd

REVERSE_DNS_MAP = dict()


Expand Down
14 changes: 7 additions & 7 deletions parsedmarc/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,8 @@ def get_reverse_dns(ip_address, cache=None, nameservers=None, timeout=2.0):
nameservers=nameservers,
timeout=timeout)[0]

except dns.exception.DNSException:
except dns.exception.DNSException as e:
logger.warning(f"get_reverse_dns({ip_address}) exception: {e}")
pass

return hostname
Expand Down Expand Up @@ -379,10 +380,6 @@ def get_ip_address_info(ip_address, ip_db_path=None,
if info:
logger.debug(f"IP address {ip_address} was found in cache")
return info
else:
logger.debug(f"IP address {ip_address} not found in cache")
else:
logger.debug("IP address cache was not specified")
info = OrderedDict()
info["ip_address"] = ip_address
if offline:
Expand All @@ -407,8 +404,11 @@ def get_ip_address_info(ip_address, ip_db_path=None,
info["type"] = service["type"]
info["name"] = service["name"]

if cache is not None:
cache[ip_address] = info
if cache is not None:
cache[ip_address] = info
logger.debug(f"IP address {ip_address} added to cache")
else:
logger.debug(f"IP address {ip_address} reverse_dns not found")

return info

Expand Down
Loading