Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Solution to frequently missing taxonomy specifications in UK submissions #112

Open
manusimidt opened this issue May 20, 2023 · 22 comments
Open

Solution to frequently missing taxonomy specifications in UK submissions #112

manusimidt opened this issue May 20, 2023 · 22 comments
Assignees
@manusimidt
Labels
enhancement New feature or request

Comments

@manusimidt
Copy link
Owner

Many UK submissions are missing a proper taxonomy declaration.
They just use the prefix without specifying the corresponding namespace and/or do not reference the taxonomy schema file.
Currently, the parser is not able to handle those issues, since It does not know where this taxonomy is located.

This issue occurs frequently with the following prefixes:

  • dpl-countries
  • core
@manusimidt manusimidt added the enhancement New feature or request label May 20, 2023
@manusimidt manusimidt self-assigned this May 20, 2023
@manusimidt
Copy link
Owner Author

#76 is related

@twright8
Copy link

Hi there - apologies for lack of knowledge of this, but for UK data i am frequently unable to pass statements because of:

  • 'LargeMedium-sizedCompaniesRegimeForAccounts'
  • 'dpl-frs'
  • 'core'
  • 'ThirdPartyAgentTypeDimension'

Was this fixed? I assume its a problem with the UK data.

Thanks very much

@manusimidt
Copy link
Owner Author

Hey, no worries.
Yes, it's true that these errors still occur often on UK submissions because many companies do not properly define their schema_urls.
I searched the internet specifically for this "dpl-frs" taxonomy. However, I could not find an online entry point (only .zip folders containing the taxonomy).

But finding some solution to this problem is in the top 3 list of fixes I want to implement in the next month. Will keep you updated.

@twright8
Copy link

@manusimidt I realise that this is provided for free and its an excellent solution, but i wonder if the UK aspect is any closer to completion? I am close to completing a research report and would ideally like to use this (with attribution!)

@manusimidt
Copy link
Owner Author

The issue I still have with UK submissions is that many of them do not properly import their underlying taxonomies.
Additionally, the taxonomy endpoints are extremely hard to find and sometimes even not reachable.

If I would find the taxonomy schema files on the web it would be pretty easy to implement.

For example, the taxonomy endpoints for us submissions are pretty straightforward and easy to find:

Name Prefix Namespace Schema URL
DEI 2011 dei http://xbrl.sec.gov/dei/2011-01-31 https://xbrl.sec.gov/dei/2011/dei-2011-01-31.xsd
DEI 2012 dei http://xbrl.sec.gov/dei/2012-01-31 https://xbrl.sec.gov/dei/2012/dei-2012-01-31.xsd
DEI 2013 dei http://xbrl.sec.gov/dei/2013-01-31 https://xbrl.sec.gov/dei/2013/dei-2013-01-31.xsd
... ... ... ...
US GAAP 2022 us-gaap http://fasb.org/us-gaap/2022 https://xbrl.fasb.org/us-gaap/2022/elts/us-gaap-2022.xsd
US GAAP 2023 us-gaap http://fasb.org/us-gaap/2023 https://xbrl.fasb.org/us-gaap/2023/elts/us-gaap-2023.xsd

If I had the same table for the UK taxonomies, I could really easy fix this issue. However, I can't find proper taxonomy endpoints for the taxonomies where the parser fails:

Name Prefix Namespace Schema URL
?? dpl-countries ?? ??
FRC 2021 core or frc-core http://xbrl.frc.org.uk/fr/2021-01-01/core ??
FRC 2022 core or frc-core http://xbrl.frc.org.uk/fr/2022-01-01/core ??

So basically if I find the missing taxonomy schema files on the web (and basically complete the table for all the years), it would be relatively easy to fix this issue.

@twright8
Copy link

@manusimidt does this page contain what you need? https://www.frc.org.uk/accountants/frc-taxonomies#current-taxonomies-downloads

or

https://uk-taxonomies-tdp.corefiling.com/yeti/resources/yeti-gwt/Yeti.jsp#tax~(id~9*v~150)!net~(a~160*l~32)!lang~(code~en)!rg~(rg~4*p~2)

Realise they're not end points but have the xsd files

@manusimidt
Copy link
Owner Author

https://uk-taxonomies-tdp.corefiling.com/yeti/resources/yeti-gwt/Yeti.jsp
Is a great way to visualize the taxonomy structure. They also have the FRC 2022 - 2024 Taxonomies, however just visualized for the user and not machine readable.

https://www.frc.org.uk/accountants/frc-taxonomies#current-taxonomies-downloads
Is the correct location for downloading the FRC taxonomy. But as far as I can see they only provide .zip folder downloads containing the entire taxonomy and supporting documents. For py-xbrl I ideally would need a direct endpoint (.xds file on server) to easily fetch it. There are basically two workarounds:

  1. Upload the extracted .zip folder to some server and map the namespace to this url
  2. Manually implement functionality to download the .zip folder, extract the necessary documents and map the prefixes, namespaces and (local) schema URLs.

But this would not even solve the full problem, since there site provides only the FRC Taxonomies (as far as I can see). I still don't know to which the other prefixes and namespaces belong.. (i.e.: "dpl-countries", " dpl-frc"...)

@twright8
Copy link

On your last point, I think (but admit i dont know much about this) countries and frc are in the Zip folders, but named slightly differently. But yeah i see your point. How frustrating! I can try and raise this with companies house

@Cave-Johnson
Copy link

Was there any progress on this with companies house? Currently running in to the same issue
I have some of found the dpl files here : https://www.frc.org.uk/library/standards-codes-policy/accounting-and-reporting/frc-taxonomies/

They are in zip format but could we put them in a public s3 bucket or a dedicated github project and map them to the right namespace?

@Cave-Johnson
Copy link

Cave-Johnson commented Oct 18, 2023 •
edited
Loading

Are these any of the missing Schema URLs?

This also works for previous years ie
https://xbrl.frc.org.uk/fr/2022-01-01/core/frc-core-full-2022-01-01.xsd

@Cave-Johnson Cave-Johnson mentioned this issue Nov 17, 2023
Open
@Cave-Johnson
Copy link

hi @manusimidt - what is needed for me to create a PR with these additional taxonimies? Are you able to link to a similar PR demonstrating whats needed? Happy to have a go at getting these added (and anymore i can find)

@manusimidt
Copy link
Owner Author

@Cave-Johnson thanks for researching and providing the links. They indeed look promising! I am not aware of any PR which addressed this problem.

Keep in mind that most of the errors are caused not because the schema URL of the taxonomy is missing but because both the schema URL and the namespace of a given taxonomy are missing. This creates for example the rather frequent error KeyError: 'dpl-frs'.

Here a simple map between the namespace and schema URL is not enough since the namespace dpl-frs could refer to multiple taxonomies. In this case we basically have two options:

  1. Always map the prefix of the taxonomy to the newest taxonomy version that is available and hope for compatibility
  2. Map the prefix of the taxonomy to the most probable taxonomy (i.e.: filing submitted on 31.10.2019 will most probably use the taxonomy version that was released in January 2019).

@manusimidt
Copy link
Owner Author

Another option would be that the parser just throws a warning and just ignores all facts that were tagged with the taxonomy which could not be imported. However, this could lead to cascading problems since the parser for example also parses XBRL footnotes. If the fact which the footnote references is not present due to missing taxonomy then the assigning of the footnote to the fact will also fail.
This case could be avoided by a simple if-check, but must still be taken into account if such a solution were to be implemented.

@manusimidt manusimidt mentioned this issue Nov 21, 2023
Open
@manusimidt
Copy link
Owner Author

manusimidt commented Nov 21, 2023 •
edited
Loading

Probably the last option (to just ignore facts tagged with not locatable taxonomies) would be the best one. In the last two years 90% of my time I invested in py-xbrl was to implement a workaround because some submissions from some country doesn't strictly follow the XBRL standard. However, all these exceptions really bloat the library and make the code harder and harder to read and maintain.

I also tried to formulate this in #84. I hope that I have some time over Christmas to dive into this. In the meantime any PRs or suggestions are highly appreciated :)

@manusimidt
Copy link
Owner Author

I just ran a test with new UK submissions and noticed that the schema URL for the following namespace was often missing:

The taxonomy with namespace **http://www.hmrc.gov.uk/schemas/ct/dpl/2021-01-01** could not be found

@Cave-Johnson already found the DPL taxonomy for 2023: https://xbrl.frc.org.uk/dpl/2023-01-01/dpl-2023-01-01.xsd
If we find a 2021 version these errors simply could be resolved for now by adding it to the mapping

@Cave-Johnson
Copy link

Cave-Johnson commented Nov 21, 2023 via email

@Cave-Johnson
Copy link

A bit more digging i have found the following https://www.hmrc.gov.uk/schemas/ct/combined/2021-01-01/dpl-2021-frs-102-2021-v1.0.0.xsd which lead me to

Are these what was missing?

@keyboardcrimp
Copy link

Been having the same issue recently when parsing companies house docs, was wondering if there was any update on this please?

@manusimidt
Copy link
Owner Author

I have tried it with the taxonomies provided by @Cave-Johnson, however, unfortunately, they did not work for all of the failed submissions I tested on.
The instance documents were using tags like dpl-countries:CountriesDimension which are not present in the taxonomy schema.

@manusimidt
Copy link
Owner Author

I think the only viable option is to build in some "fail gracefully" mode into py-xbrl in which the parser returns a partially parsed xbrl instance document and just omits every fact/context that is labeled/uses a concept from a taxonomy which could not be located.

@Cave-Johnson
Copy link

understand your point RE fail gracefully logic and that makes sense. It would still be good to get the taxonomies linked where possible.

Ive done a bit more digging and come across the attached that references dpl countries in the dpl-2021-presentation.xml file - is this of any use?
DPL-2021-v1.1.0.zip

@twright8
Copy link

@manusimidt Thanks for all the looking into this! was just wondering if @Cave-Johnson solution could have helped?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@manusimidt manusimidt

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Cave-Johnson @manusimidt @twright8 @keyboardcrimp