Skip to content

Commit

Permalink
Merge pull request #555 from /issues/554
Browse files Browse the repository at this point in the history
fixes #554 - 12.0.0 release
  • Loading branch information
jantman authored Aug 4, 2021
2 parents 8921393 + f7cf057 commit 411ad9e
Show file tree
Hide file tree
Showing 27 changed files with 3,201 additions and 184 deletions.
32 changes: 32 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,38 @@
Changelog
=========

.. _changelog.12_0_0:

12.0.0 (2021-08-04)
-------------------

IMPORTANT - Breaking Changes
++++++++++++++++++++++++++++

* This release **removes** the ``EC2 / Max spot instance requests per region`` limit, which has been removed by AWS, in favor of six new vCPU-based limits: ``All F Spot Instance Requests``, ``All G Spot Instance Requests``, ``All Inf Spot Instance Requests``, ``All P Spot Instance Requests``, ``All X Spot Instance Requests``, and ``All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests``.
* This release **adds two new services**: ``CertificateManager`` (ACM) and ``CloudFront``.
* This release **requires additional IAM permissions**: ``acm:ListCertificates``, ``cloudfront:ListCloudFrontOriginAccessIdentities``, ``cloudfront:ListKeyGroups``, ``cloudfront:ListDistributions``, ``cloudfront:ListCachePolicies``, and ``cloudfront:ListOriginRequestPolicies``.

IMPORTANT - Seeking New Maintainer
++++++++++++++++++++++++++++++++++

As I commented in `Issue #500 <https://github.com/jantman/awslimitchecker/issues/500>`__, I'm looking for someone to share (and perhaps take over) maintenance of this project. awslimitchecker is, and has always been, a personal-time-only project for me; the only time I've done work on it during my day job is when my employer was experiencing an issue or requested a specific feature. Because of a variety of issues, including changing personal interests and my employer relying on this project much less (following an AWS account restructuring that largely avoids service limits), I've been spending much less time on this project than it deserves. As a result, I'm looking for someone to help with maintenance... at the very least, helping review PRs and get them to a merge-able state. If you're interested, please comment on `Issue #500 <https://github.com/jantman/awslimitchecker/issues/500>`__ or contact me directly. While I am *incredibly* flattered by the offers I've received for sponsorship, paid support, or other financial incentive, I'd ask that anyone who's willing to make that commitment instead dedicate a few hours to working on issues or PRs. I, for my part, will make a concerted effort to quickly merge and release any PRs that meet all of the :ref:`development.pull_request_guidelines`.

All Changes
+++++++++++

* `PR #532 <https://github.com/jantman/awslimitchecker/pull/532>`__ - Add Quotas Service support for ECS Fargate quotas. Thanks to `robpickerill <https://github.com/robpickerill>`__ for this contribution.
* `PR #533 <https://github.com/jantman/awslimitchecker/pull/533>`__ / Fixes `Issue #527 <https://github.com/jantman/awslimitchecker/issues/527>`__ - Fix Quotas Service quota names for EIPs. Thanks to `robpickerill <https://github.com/robpickerill>`__ for this contribution.
* `PR #534 <https://github.com/jantman/awslimitchecker/pull/534>`__ / Fixes `Issue #521 <https://github.com/jantman/awslimitchecker/issues/521>`__ - Update Quotas Service quota names for EBS. Thanks to `robpickerill <https://github.com/robpickerill>`__ for this contribution.
* `PR #535 <https://github.com/jantman/awslimitchecker/pull/535>`__ / Fixes `Issue #518 <https://github.com/jantman/awslimitchecker/issues/518>`__ - Fix EC2 Security Group counts to only include groups owned by the current account. Thanks to `robpickerill <https://github.com/robpickerill>`__ for this contribution.
* `PR #536 <https://github.com/jantman/awslimitchecker/pull/536>`__ / Fixes `Issue #512 <https://github.com/jantman/awslimitchecker/issues/512>`__ - Fix CloudWatch metrics queries to get data from one minute ago, to fix bug where GetMetricData is not yet populated. Thanks to `robpickerill <https://github.com/robpickerill>`__ for this contribution.
* `PR #543 <https://github.com/jantman/awslimitchecker/pull/543>`__ / Fixes `Issue #538 <https://github.com/jantman/awslimitchecker/issues/538>`__ - Fix issue with calculation of usage for EC2 Rules Per Network ACL. Thanks to `jwu2 <https://github.com/jwu2>`__ for this contribution.
* `PR #537 <https://github.com/jantman/awslimitchecker/pull/537>`__ - Use boto3 adaptive retry mode. Thanks to `robpickerill <https://github.com/robpickerill>`__ for this contribution.
* `PR #547 <https://github.com/jantman/awslimitchecker/pull/547>`__ / Fixes `Issue #502 <https://github.com/jantman/awslimitchecker/issues/502>`__ - Replace ``EC2 / Max spot instance requests per region`` limit, which has been removed by AWS, with new vCPU-based spot instance requests limits. This also switches to using CloudWatch metric data to retrieve current usage. Thanks to `TagadaPoe <https://github.com/TagadaPoe>`__ for this contribution.
* `PR #546 <https://github.com/jantman/awslimitchecker/pull/546>`__ / Fixes `Issue #540 <https://github.com/jantman/awslimitchecker/issues/540>`__ - Add support for ACM (Certificate Manager) limits. Thanks to `TagadaPoe <https://github.com/TagadaPoe>`__ for this contribution.
* `PR #545 <https://github.com/jantman/awslimitchecker/pull/545>`__ / Fixes `Issue #539 <https://github.com/jantman/awslimitchecker/issues/539>`__ - Add support for CloudFront limits. Thanks to `TagadaPoe <https://github.com/TagadaPoe>`__ for this contribution.
* `Issue #551 <https://github.com/jantman/awslimitchecker/issues/551>`__ - Allow custom host for Datadog metric provider.

.. _changelog.11_0_0:

11.0.0 (2021-04-20)
Expand Down
13 changes: 11 additions & 2 deletions awslimitchecker/connectable.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,9 +112,13 @@ def connect(self):
"""
if self.conn is not None:
return

default_config = Config(retries={'mode': 'adaptive'})
kwargs = dict(self._boto3_connection_kwargs)
kwargs['config'] = default_config

if self._max_retries_config is not None:
kwargs['config'] = self._max_retries_config
kwargs['config'] = default_config.merge(self._max_retries_config)
self.conn = boto3.client(self.api_name, **kwargs)
logger.info("Connected to %s in region %s",
self.api_name, self.conn._client_config.region_name)
Expand All @@ -132,9 +136,14 @@ def connect_resource(self):
"""
if self.resource_conn is not None:
return

default_config = Config(retries={'mode': 'adaptive'})
kwargs = dict(self._boto3_connection_kwargs)
kwargs['config'] = default_config

if self._max_retries_config is not None:
kwargs['config'] = self._max_retries_config
kwargs['config'] = default_config.merge(self._max_retries_config)

self.resource_conn = boto3.resource(self.api_name, **kwargs)
logger.info("Connected to %s (resource) in region %s", self.api_name,
self.resource_conn.meta.client._client_config.region_name)
13 changes: 9 additions & 4 deletions awslimitchecker/metrics/datadog.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ class Datadog(MetricsProvider):

def __init__(
self, region_name, prefix='awslimitchecker.', api_key=None,
extra_tags=None
extra_tags=None, host='https://api.datadoghq.com'
):
"""
Initialize the Datadog metrics provider. This class does not have any
Expand All @@ -68,6 +68,11 @@ def __init__(
:param api_key: Datadog API key. May alternatively be specified by the
``DATADOG_API_KEY`` environment variable.
:type api_key: str
:param host: The datadog host URL to use; defaults to
``https://api.datadoghq.com``. This parameter is overridden by the
``DATADOG_HOST`` environment variable, if set. This must NOT end with
a trailing slash.
:type host: str
:param extra_tags: CSV list of additional tags to send with metrics.
All metrics will automatically be tagged with ``region:<region name>``
:type extra_tags: str
Expand All @@ -78,6 +83,7 @@ def __init__(
if extra_tags is not None:
self._tags.extend(extra_tags.split(','))
self._api_key = os.environ.get('DATADOG_API_KEY')
self._host = os.environ.get('DATADOG_HOST', host)
if api_key is not None:
self._api_key = api_key
if self._api_key is None:
Expand All @@ -88,7 +94,7 @@ def __init__(
self._validate_auth(self._api_key)

def _validate_auth(self, api_key):
url = 'https://api.datadoghq.com/api/v1/validate?api_key=%s'
url = self._host + '/api/v1/validate?api_key=%s'
logger.debug('Validating Datadog API key: GET %s', url)
url = url % api_key
r = self._http.request('GET', url)
Expand Down Expand Up @@ -149,8 +155,7 @@ def flush(self):
logger.info('POSTing %d metrics to datadog', len(series))
data = {'series': series}
encoded = json.dumps(data).encode('utf-8')
url = 'https://api.datadoghq.com/api/v1/series' \
'?api_key=%s' % self._api_key
url = self._host + '/api/v1/series?api_key=%s' % self._api_key
resp = self._http.request(
'POST', url,
headers={'Content-type': 'application/json'},
Expand Down
3 changes: 3 additions & 0 deletions awslimitchecker/services/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,10 @@
from awslimitchecker.services.base import _AwsService
from awslimitchecker.services.apigateway import _ApigatewayService
from awslimitchecker.services.autoscaling import _AutoscalingService
from awslimitchecker.services.certificatemanager import \
_CertificatemanagerService
from awslimitchecker.services.cloudformation import _CloudformationService
from awslimitchecker.services.cloudfront import _CloudfrontService
from awslimitchecker.services.cloudtrail import _CloudTrailService
from awslimitchecker.services.directoryservice import _DirectoryserviceService
from awslimitchecker.services.dynamodb import _DynamodbService
Expand Down
4 changes: 2 additions & 2 deletions awslimitchecker/services/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -353,8 +353,8 @@ def _get_cloudwatch_usage_latest(
}
}
],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
StartTime=datetime.utcnow() - timedelta(hours=1, minutes=1),
EndTime=datetime.utcnow() - timedelta(minutes=1),
ScanBy='TimestampDescending',
MaxDatapoints=1
)
Expand Down
117 changes: 117 additions & 0 deletions awslimitchecker/services/certificatemanager.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
"""
awslimitchecker/services/certificatemanager.py
The latest version of this package is available at:
<https://github.com/jantman/awslimitchecker>
################################################################################
Copyright 2015-2018 Jason Antman <[email protected]>
This file is part of awslimitchecker, also known as awslimitchecker.
awslimitchecker is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
awslimitchecker is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with awslimitchecker. If not, see <http://www.gnu.org/licenses/>.
The Copyright and Authors attributions contained herein may not be removed or
otherwise altered, except to add the Author attribution of a contributor to
this work. (Additional Terms pursuant to Section 7b of the AGPL v3)
################################################################################
While not legally required, I sincerely request that anyone who finds
bugs please submit them at <https://github.com/jantman/awslimitchecker> or
to me via email, and that you send any contributions or improvements
either as a pull request on GitHub, or to me via email.
################################################################################
AUTHORS:
Jason Antman <[email protected]> <http://www.jasonantman.com>
################################################################################
"""

import abc # noqa
import logging

from .base import _AwsService
from ..limit import AwsLimit
from ..utils import paginate_dict

logger = logging.getLogger(__name__)


class _CertificatemanagerService(_AwsService):

service_name = 'CertificateManager'
api_name = 'acm' # AWS API name to connect to (boto3.client)
quotas_service_code = 'acm'

def find_usage(self):
"""
List CloudFront distributions by calling AWS list_certificates, and
update usage in self.limits for the limit 'ACM certificates'
"""
logger.debug("Checking usage for service %s", self.service_name)
self.connect()
for lim in self.limits.values():
lim._reset_usage()

self._find_usage_certificates()

self._have_usage = True
logger.debug("Done checking usage.")

def _find_usage_certificates(self):
"""find usage for ACM certificates"""
res = paginate_dict(
self.conn.list_certificates,
alc_marker_path=['NextToken'],
alc_data_path=['CertificateSummaryList'],
alc_marker_param='NextToken'
)
if 'CertificateSummaryList' not in res:
nb_certificates = 0
else:
nb_certificates = len(res['CertificateSummaryList'])
self.limits['ACM certificates']._add_current_usage(nb_certificates)

def get_limits(self):
"""
Return all known limits for this service, as a dict of their names
to :py:class:`~.AwsLimit` objects.
:returns: dict of limit names to :py:class:`~.AwsLimit` objects
:rtype: dict
"""
if self.limits != {}:
return self.limits
limits = {}
limits['ACM certificates'] = AwsLimit(
'ACM certificates',
self,
1000,
self.warning_threshold,
self.critical_threshold
)
self.limits = limits
return limits

def required_iam_permissions(self):
"""
Return a list of IAM Actions required for this Service to function
properly. All Actions will be shown with an Effect of "Allow"
and a Resource of "*".
:returns: list of IAM Action strings
:rtype: list
"""
return [
"acm:ListCertificates",
]
Loading

0 comments on commit 411ad9e

Please sign in to comment.