-
-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix handling of tag list in Creator.add_metadata (fix #125) #126
Conversation
Thank you ; it looks good (except maybe the comments). @benoit74 will review but merge might be delayed a bit as he is in the process of changing the setup of this repo so it may be easier for him to merge his stuff first. |
openzim#290) This commit upgrades python-scraperlib to version 3.x to implement issue openzim#290. It depends on openzim/python-scraperlib#126 and will likely need at least one revision. Honestly, I am not happy with the splitted Illustration handling.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you ! Minor change to remove comments.
@rgaudin any views on this? |
Sure bothers me as well. I'd prefer we implement the date/datetime conversion here as well and eventually (needs a major release) remove the conversion there as pylibzim is just a wrapper around the C libzim. |
Issue is opened in python-libzim: openzim/python-libzim#188 |
I've removed the comments as requested. I've also added the datetime conversion here so all metadata value conversion happens in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very good.
Good idea to already add the conversion of Date here.
To everybody:
I have another PR with lots of code changes to migrate to our new Python conventions, I will merge this other PR first to limit the risk of conflicts / simplify resolution. Do not mind about it, I will take care of everything (just do not delete your branch / repo until then ;-)).
This fixes issue openzim#125. Previously, the type annotations of Creator.config_metadata() allowed "Tags" to be a list of strings, but didn't handle the conversion to a string, which resulted in python-libzim raising an error. This commit modifies Creator.add_metadata() to accept a list of strings and, if the key of the metadata is "Tags", join them into a single string. A regression test is included.
I didn't think I'd ever encounter an open source project opposed to comments, but here we are. Some comments will soon be outdated anyway.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #126 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 32 32
Lines 1332 1337 +5
Branches 225 227 +2
=========================================
+ Hits 1332 1337 +5 ☔ View full report in Codecov by Sentry. |
I rebased your changes and it made me realize there was a typing issue + we can mutualise your new test with the existing one, this is the intent of my last commit. As mentioned in this last commit, I had to ignore the typing issue, it was hard to solve without TypeGuards (3.10) or would necessitate a significant code change. We can live with it for now. |
@rgaudin would you mind to review the last commit please, I'm not sure you will agree with my suggestion to ignore the typing issue for now. |
Why not be safe with if name == "Tags" and not isinstance(content, str) and isinstance(content, collections.abc.Iterable):
content = ";".join(str(tag) for tag in content) |
pyright ignore is mandatory because the type checker has no idea about the impact of validate_metadata/validate_tag. This might be fixed by a shared logic and a TypeGuard but is not available until Python 3.10 ; other solution would be to transfer metadata to a way more typed container after validation.
Thank you very much! I chose to be even safer with
Just in case some day we begin to accept bytes ... don't see why but who knows. |
#290) This commit upgrades python-scraperlib to version 3.x to implement issue #290. It depends on openzim/python-scraperlib#126 and will likely need at least one revision. Honestly, I am not happy with the splitted Illustration handling.
Fixes #125
Previously, the type annotations of
Creator.config_metadata()
allowed "Tags" to be a list of strings, but didn't handle the conversion to a string, which resulted in python-libzim raising an error.This commit modifies
Creator.add_metadata()
to accept a list of strings and, if the key of the metadata is "Tags", join them into a single string. A regression test is included. The metadata validation seems have already been written with a tag list in mind (though there is no check that a single tag in such a list should not contain";"
, not sure if this is intended).Note: the join has been implemented in
Creator.add_metadata
, but the conversion ofdatetime.datetime
for the"Date"
metadata key happens inpython-libzim
. I think splitting the conversion logic between those two libraries is suboptimal, we should consider implementing the tag list conversion inpython-libzim
instead.