Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This adds generating VTIMEZONE components from tzinfo objects #741

Merged
merged 42 commits into from
Nov 22, 2024
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
ac3ccd6
Add timezone generation
niccokunzmann Nov 2, 2024
574c1da
Generate one STANDARD and one DAYLIGHT timezone in Europe/Berlin
niccokunzmann Nov 4, 2024
af51200
todo test
niccokunzmann Nov 4, 2024
5d52304
speed up computation
niccokunzmann Nov 9, 2024
3c035a6
Use timezones by zoneinfo/pytz in case they exist
niccokunzmann Nov 10, 2024
40263d0
isulate pytz error
niccokunzmann Nov 10, 2024
f372e83
Use provided timezone in Timezone.to_tz() if possible and wished for
niccokunzmann Nov 10, 2024
ef238c6
Remove timezone test failure
niccokunzmann Nov 10, 2024
7f25e3a
skip pytz conversion tests
niccokunzmann Nov 10, 2024
4ff0df4
Check actual datetime values and their offset
niccokunzmann Nov 11, 2024
cc62b34
Check that the dates are in range
niccokunzmann Nov 11, 2024
c944541
Test fixed offset UTC
niccokunzmann Nov 11, 2024
3cd9980
log changes
niccokunzmann Nov 11, 2024
57f8a57
Document functionality
niccokunzmann Nov 11, 2024
3c4ce10
improve examples
niccokunzmann Nov 11, 2024
215bda0
fix tests for timezone ambiguity
niccokunzmann Nov 11, 2024
8e07f72
Correct tests
niccokunzmann Nov 11, 2024
acb6d40
Improve documentation
niccokunzmann Nov 11, 2024
825abe1
Find a set of missing tzids
niccokunzmann Nov 11, 2024
9b7c3fb
Make repr(vUTCOffset) nicer
niccokunzmann Nov 11, 2024
ee7e4fe
Add missing timezones
niccokunzmann Nov 11, 2024
b7bba44
Improve docs and add tests
niccokunzmann Nov 11, 2024
ff49220
Skip test for newly introduced file.
niccokunzmann Nov 12, 2024
5a3a54e
Start identifying dateutil and other timezones based on tzname()
niccokunzmann Nov 12, 2024
436c384
Generate lookup table for timezone ids
niccokunzmann Nov 13, 2024
1b7e23f
Create tzid module to identify timezones
niccokunzmann Nov 13, 2024
86360ed
Add tests for pytz, zoneinfo and dateutil timezone indentification
niccokunzmann Nov 13, 2024
10a6a40
generate with zoneinfo and dateutil and pytz
niccokunzmann Nov 13, 2024
27636c5
Print remaining computation time
niccokunzmann Nov 13, 2024
45fa50c
add link to stackoverflow
niccokunzmann Nov 13, 2024
d514f7d
use multiprocessing to speed up computation
niccokunzmann Nov 13, 2024
200cb41
Run with all timezones
niccokunzmann Nov 13, 2024
30ea2fe
Update src/icalendar/tests/test_issue_722_generate_vtimezone.py
niccokunzmann Nov 14, 2024
c723950
Update src/icalendar/tests/test_issue_722_generate_vtimezone.py
niccokunzmann Nov 14, 2024
04bf007
Update src/icalendar/cal.py
niccokunzmann Nov 14, 2024
28308bc
failed to identify 420 dateutil timezones
niccokunzmann Nov 15, 2024
2a639dd
Properly map dateutil timezones to tzid
niccokunzmann Nov 15, 2024
368ae94
correct ci errors
niccokunzmann Nov 15, 2024
56c6d74
Handle special cases for timezone identification
niccokunzmann Nov 21, 2024
95c023d
Merge main
niccokunzmann Nov 21, 2024
a5bc3bd
log changes
niccokunzmann Nov 21, 2024
e028e7d
Correct py38 import error
niccokunzmann Nov 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ Changelog

Minor changes:

- Added ``end``, ``start``, ``duration``, ``DTSTART``, ``DUE``, and ``DURATION`` attributes to ``Todo`` components. See `Issue 662`_.
- Add ``end``, ``start``, ``duration``, ``DTSTART``, ``DUE``, and ``DURATION`` attributes to ``Todo`` components. See `Issue 662`_.
- Add ``DTSTART``, ``TZOFFSETTO`` and ``TZOFFSETFROM`` properties to ``TimezoneStandard`` and ``TimezoneDaylight``. See `Issue 662`_.
- Format test code with Ruff. See `Issue 672 <https://github.com/collective/icalendar/issues/672>`_.
- Document the Debian package. See `Issue 701 <https://github.com/collective/icalendar/issues/701>`_.
- Document component classes with description from :rfc:`5545`.
Expand All @@ -19,11 +20,15 @@ New features:

- Add ``VALARM`` properties for :rfc:`9074`. See `Issue 657 <https://github.com/collective/icalendar/issues/657>`_
- Test compatibility with Python 3.13
- Add ``Timezone.from_tzinfo()`` and ``Timezone.from_tzid()`` to create a ``Timezone`` component from a ``datetime.tzinfo`` timezone. See `Issue 722`_.
- Add ``icalendar.prop.tzid_from_tzinfo``.

Bug fixes:

- Add ``icalendar.timezone`` to the documentation.

.. _`Issue 722`: https://github.com/collective/icalendar/issues/722

6.0.1 (2024-10-13)
------------------

Expand Down
254 changes: 239 additions & 15 deletions src/icalendar/cal.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,26 @@
from __future__ import annotations

import os
from datetime import date, datetime, timedelta
from collections import defaultdict
from datetime import date, datetime, timedelta, tzinfo
from typing import List, Optional, Tuple

import dateutil.rrule
import dateutil.tz

from icalendar.caselessdict import CaselessDict
from icalendar.parser import Contentline, Contentlines, Parameters, q_join, q_split
from icalendar.parser_tools import DEFAULT_ENCODING
from icalendar.prop import TypesFactory, vDDDLists, vDDDTypes, vText, vDuration
from icalendar.timezone import tzp
from icalendar.prop import (
TypesFactory,
tzid_from_tzinfo,
vDDDLists,
vDDDTypes,
vDuration,
vText,
vUTCOffset,
)
from icalendar.timezone import TZP, tzp


def get_example(component_directory: str, example_name: str) -> bytes:
Expand Down Expand Up @@ -282,7 +292,7 @@ def set_inline(self, name, values, encode=1):
#########################
# Handling of components

def add_component(self, component):
def add_component(self, component: Component):
"""Add a subcomponent to this component.
"""
self.subcomponents.append(component)
Expand All @@ -297,7 +307,7 @@ def _walk(self, name, select):
result += subcomponent._walk(name, select)
return result

def walk(self, name=None, select=lambda c: True):
def walk(self, name=None, select=lambda c: True) -> list[Component]:
"""Recursively traverses component and subcomponents. Returns sequence
of same. If name is passed, only components with name will be returned.

Expand Down Expand Up @@ -497,8 +507,23 @@ def __eq__(self, other):
#######################################
# components defined in RFC 5545

def create_single_property(prop:str, value_attr:str, value_type:tuple[type], type_def:type, doc:str):
"""Create a single property getter and setter."""
def create_single_property(
prop:str,
value_attr:Optional[str],
value_type:tuple[type],
type_def:type,
doc:str,
vProp:type=vDDDTypes # noqa: N803
):
"""Create a single property getter and setter.

:param prop: The name of the property.
:param value_attr: The name of the attribute to get the value from.
:param value_type: The type of the value.
:param type_def: The type of the property.
:param doc: The docstring of the property.
:param vProp: The type of the property from :mod:`icalendar.prop`.
"""

def p_get(self : Component):
default = object()
Expand All @@ -507,7 +532,7 @@ def p_get(self : Component):
return None
if isinstance(result, list):
raise InvalidCalendar(f"Multiple {prop} defined.")
value = getattr(result, value_attr, result)
value = result if value_attr is None else getattr(result, value_attr, result)
if not isinstance(value, value_type):
raise InvalidCalendar(f"{prop} must be either a {' or '.join(t.__name__ for t in value_type)}, not {value}.")
return value
Expand All @@ -518,7 +543,7 @@ def p_set(self:Component, value) -> None:
return
if not isinstance(value, value_type):
raise TypeError(f"Use {' or '.join(t.__name__ for t in value_type)}, not {type(value).__name__}.")
self[prop] = vDDDTypes(value)
self[prop] = vProp(value)
if prop in self.exclusive:
for other_prop in self.exclusive:
if other_prop != prop:
Expand Down Expand Up @@ -892,7 +917,6 @@ class FreeBusy(Component):
)
multiple = ('ATTENDEE', 'COMMENT', 'FREEBUSY', 'RSTATUS',)


class Timezone(Component):
"""
A "VTIMEZONE" calendar component is a grouping of component
Expand All @@ -904,20 +928,23 @@ class Timezone(Component):
required = ('TZID',) # it also requires one of components DAYLIGHT and STANDARD
singletons = ('TZID', 'LAST-MODIFIED', 'TZURL',)

_DEFAULT_FIRST_DATE = date(1970, 1, 1)
_DEFAULT_LAST_DATE = date(2038, 1, 1)

@classmethod
def example(cls, name: str) -> Calendar:
"""Return the calendar example with the given name."""
return cls.from_ical(get_example("timezones", name))

@staticmethod
def _extract_offsets(component, tzname):
def _extract_offsets(component: TimezoneDaylight|TimezoneStandard, tzname:str):
"""extract offsets and transition times from a VTIMEZONE component
:param component: a STANDARD or DAYLIGHT component
:param tzname: the name of the zone
"""
offsetfrom = component['TZOFFSETFROM'].td
offsetto = component['TZOFFSETTO'].td
dtstart = component['DTSTART'].dt
offsetfrom = component.TZOFFSETFROM
offsetto = component.TZOFFSETTO
dtstart = component.DTSTART

# offsets need to be rounded to the next minute, we might loose up
# to 30 seconds accuracy, but it can't be helped (datetime
Expand All @@ -942,6 +969,7 @@ def _extract_offsets(component, tzname):
# here we construct local times without tzinfo, the offset to UTC
# gets subtracted in to_tz().
transtimes = [dt.replace (tzinfo=None) for dt in rrule]
print("transtimes", transtimes)

# or rdates
elif 'RDATE' in component:
Expand Down Expand Up @@ -975,9 +1003,20 @@ def _make_unique_tzname(tzname, tznames):
tznames.add(tzname)
return tzname

def to_tz(self, tzp=tzp):
def to_tz(self, tzp:TZP=tzp, lookup_tzid:bool=True):
"""convert this VTIMEZONE component to a timezone object

:param tzp: timezone provider to use
:param lookup_tzid: whether to use the TZID property to look up existing
timezone definitions with tzp.
If it is False, a new timezone will be created.
If it is True, the existing timezone will be used
if it exists, otherwise a new timezone will be created.
"""
if lookup_tzid:
tz = tzp.timezone(self.tz_name)
if tz is not None:
return tz
return tzp.create_timezone(self)

@property
Expand Down Expand Up @@ -1057,6 +1096,149 @@ def get_transitions(self) -> Tuple[List[datetime], List[Tuple[timedelta, timedel
transition_info.append((osto, dst_offset, name))
return transition_times, transition_info

# binary search
_from_tzinfo_skip_search = [
timedelta(days=days) for days in (64, 32, 16, 8, 4, 2, 1)
] + [
# we know it happens in the night usually around 1am
timedelta(hours=4),
timedelta(hours=1),
# adding some minutes and seconds for faster search
timedelta(minutes=20),
timedelta(minutes=5),
timedelta(minutes=1),
timedelta(seconds=20),
timedelta(seconds=5),
timedelta(seconds=1),
]
@classmethod
def from_tzinfo(
cls,
timezone: tzinfo,
tzid:Optional[str]=None,
first_date:date=_DEFAULT_FIRST_DATE,
last_date:date=_DEFAULT_LAST_DATE
) -> Timezone:
"""Return a VTIMEZONE component from a timezone object.

This works with pytz and zoneinfo and any other timezone.
The offsets are calculated from the tzinfo object.

Parameters:

:param tzinfo: the timezone object
:param tzid: the tzid for this timezone in case we cannot determine it
None for pytz and zoneinfo is fine.
niccokunzmann marked this conversation as resolved.
Show resolved Hide resolved
:param first_date: a datetime that is earlier than anything that happens in the calendar
:param last_date: a datetime that is later than anything that happens in the calendar

.. note::
This can take some time. Please cache the results.
"""
if tzid is None:
tzid = tzid_from_tzinfo(timezone)
if tzid is None:
raise ValueError(f"Cannot get TZID from {timezone}. Please set the tzid parameter.")
normalize = getattr(timezone, "normalize", lambda dt: dt) # pytz compatibility
first_datetime = datetime(first_date.year, first_date.month, first_date.day) # noqa: DTZ001
last_datetime = datetime(last_date.year, last_date.month, last_date.day) # noqa: DTZ001
if hasattr(timezone, "localize"): #pytz compatibility
first_datetime = timezone.localize(first_datetime)
last_datetime = timezone.localize(last_datetime)
else:
first_datetime = first_datetime.replace(tzinfo=timezone)
last_datetime = last_datetime.replace(tzinfo=timezone)
# from, to, tzname, is_standard -> start
offsets :dict[tuple[Optional[timedelta], timedelta, str, bool], list[datetime]] = defaultdict(list)
start = first_datetime
offset_to = None
while start < last_datetime:
offset_from = offset_to
end = start
offset_to = end.utcoffset()
for add_offset in cls._from_tzinfo_skip_search:
last_end = end # we need to save this as we might be left and right of the time change
end = normalize(end + add_offset)
try:
while end.utcoffset() == offset_to:
last_end = end
end = normalize(end + add_offset)
except OverflowError:
# zoninfo does not go all the way
break
# retract if we overshoot
end = last_end
# Now, start (inclusive) -> end (exclusive) are one timezone
is_standard = start.dst() == timedelta()
name = start.tzname()
if name is None:
name = str(offset_to)
key = (offset_from, offset_to, name, is_standard)
# first_key = (None,) + key[1:]
# if first_key in offsets:
# # remove the first one and claim it changes at that day
# offsets[first_key] = offsets.pop(first_key)
offsets[key].append(start.replace(tzinfo=None))
start = normalize(end + cls._from_tzinfo_skip_search[-1])
tz = cls()
tz.add("TZID", tzid)
tz.add("COMMENT", f"This timezone only works from {first_date} to {last_date}.")
for (offset_from, offset_to, tzname, is_standard), starts in offsets.items():
first_start = min(starts)
starts.remove(first_start)
if first_start.date() == last_date:
first_start = datetime(last_date.year, last_date.month, last_date.day) # noqa: DTZ001
subcomponent = TimezoneStandard() if is_standard else TimezoneDaylight()
if offset_from is None:
offset_from = offset_to # noqa: PLW2901
subcomponent.TZOFFSETFROM = offset_from
subcomponent.TZOFFSETTO = offset_to
subcomponent.add("TZNAME", tzname)
subcomponent.DTSTART = first_start
if starts:
subcomponent.add("RDATE", starts)
tz.add_component(subcomponent)
return tz

@classmethod
def from_tzid(
cls,
tzid:str,
tzp:TZP=tzp,
first_date:date=_DEFAULT_FIRST_DATE,
last_date:date=_DEFAULT_LAST_DATE
) -> Timezone:
"""Create a VTIMEZONE from a tzid like ``"Europe/Berlin"``.

:param tzid: the id of the timezone
:param tzp: the timezone provider
:param first_date: a datetime that is earlier than anything that happens in the calendar
:param last_date: a datetime that is later than anything that happens in the calendar

>>> from icalendar import Timezone
>>> tz = Timezone.from_tzid("Europe/Berlin")
>>> print(tz.to_ical()[:36])
BEGIN:VTIMEZONE
TZID:Europe/Berlin

.. note::
This can take some time. Please cache the results.
"""
return cls.from_tzinfo(tzp.timezone(tzid), tzid, first_date, last_date)

@property
def standard(self) -> list[TimezoneStandard]:
"""The STANDARD subcomponents as a list."""
return self.walk("STANDARD")

@property
def daylight(self) -> list[TimezoneDaylight]:
"""The DAYLIGHT subcomponents as a list.

These are for the daylight saving time.
"""
return self.walk("DAYLIGHT")


class TimezoneStandard(Component):
"""
Expand All @@ -1070,6 +1252,45 @@ class TimezoneStandard(Component):
singletons = ('DTSTART', 'TZOFFSETTO', 'TZOFFSETFROM',)
multiple = ('COMMENT', 'RDATE', 'TZNAME', 'RRULE', 'EXDATE')

DTSTART = create_single_property(
"DTSTART",
"dt",
(datetime,),
datetime,
"""The mandatory "DTSTART" property gives the effective onset date
and local time for the time zone sub-component definition.
"DTSTART" in this usage MUST be specified as a date with a local
time value."""
)
TZOFFSETTO = create_single_property(
"TZOFFSETTO",
"td",
(timedelta,),
timedelta,
"""The mandatory "TZOFFSETTO" property gives the UTC offset for the
time zone sub-component (Standard Time or Daylight Saving Time)
when this observance is in use.
""",
vUTCOffset
)
TZOFFSETFROM = create_single_property(
"TZOFFSETFROM",
"td",
(timedelta,),
timedelta,
"""The mandatory "TZOFFSETFROM" property gives the UTC offset that is
in use when the onset of this time zone observance begins.
"TZOFFSETFROM" is combined with "DTSTART" to define the effective
onset for the time zone sub-component definition. For example,
the following represents the time at which the observance of
Standard Time took effect in Fall 1967 for New York City:

DTSTART:19671029T020000
TZOFFSETFROM:-0400
""",
vUTCOffset
)


class TimezoneDaylight(Component):
"""
Expand All @@ -1083,6 +1304,9 @@ class TimezoneDaylight(Component):
singletons = TimezoneStandard.singletons
multiple = TimezoneStandard.multiple

DTSTART = TimezoneStandard.DTSTART
TZOFFSETTO = TimezoneStandard.TZOFFSETTO
TZOFFSETFROM = TimezoneStandard.TZOFFSETFROM

class Alarm(Component):
"""
Expand Down
Loading
Loading