xsd2py failing #83

thomas-mangin · 2016-03-22T14:46:08Z

Hello Felix,

I never had to use xsd/wsdl/soap before but I am trying to find a library which can help me implement correctly a tr-069 server (instead of just generating/parsing the XML 'by hand') like many available open source projects.

soapfish seems to be a good contender for the job but before I can even get started I have this issue.
Any help would be welcomed ! Thank you.

# xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd
Traceback (most recent call last):
  File "/usr/local/bin/xsd2py", line 9, in <module>
    load_entry_point('soapfish', 'console_scripts', 'xsd2py')()
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 171, in main
    code = generate_code_from_xsd(xmlelement, encoding='utf-8')
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 69, in generate_code_from_xsd
    standalone=True)
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 151, in schema_to_py
    return tpl.render(schema=schema, cwd=cwd, base_path=base_path)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/thomas/source/tr-69/soapfish/soapfish/templates/xsd", line 11, in top-level template code
    {{- resolve_import(i, known_files, schema.targetNamespace, cwd, base_path) }}
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 48, in resolve_import
    location = os.path.relpath(location, base_path)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 430, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 360, in abspath
    if not isabs(path):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 54, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'

The text was updated successfully, but these errors were encountered:

ngnpope · 2016-03-22T15:34:15Z

I have been working on a lot of fixes and this one might be solved in my improvements branch by the commit 828e4a5. I still have a few more things to finish before I submit a pull request.

According to the specification xsd:import[schemaLocation] and xsd:include[schemaLocation] are optional. Please try the branch to see if you have any success. There are also some other issues with generation of names for schemas based on location that I am currently attempting to fix.

FelixSchwarz · 2016-03-22T15:44:51Z

@thomas-mangin Can you try Nick's branch as suggest in his comment above?

thomas-mangin · 2016-03-22T16:09:01Z

I was pretty sure to run this branch but to make absolutely sure, I done this ( OSX )

# git branch -v
* improvements 58e7fc3 misc: drop support for django<1.6.0
  master       2407838 merge element reordering for correct XSD restriction/extension rendering (by Iuri de Silvio)

# git log | head -8
commit 58e7fc39e961ff7ccb644e36f5e3024345ba6eb8
Author: Nick Pope <[email protected]>
Date:   Mon Mar 21 16:32:36 2016 +0000

    misc: drop support for django<1.6.0

    License: BSD

# pip uninstall soapfish
Uninstalling soapfish-0.6.0.dev0:
  /usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg
Proceed (y/n)? y
  Successfully uninstalled soapfish-0.6.0.dev0

# cd  /usr/local/bin
# rm xsd2py wsdl2py py2xsd py2wsdl
# cd -

# python setup.py install
[...]
Using /usr/local/lib/python2.7/site-packages
Finished processing dependencies for soapfish==0.6.0.dev0

# find /usr/local/lib/python* | grep soapfish | cut -f1-7 -d/ | uniq
/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg

# find /usr/local/opt/python* | grep soapfish

# which xsd2py
/usr/local/bin/xsd2py

# cat /usr/local/bin/xsd2py
#!/usr/local/opt/python/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'soapfish==0.6.0.dev0','console_scripts','xsd2py'
__requires__ = 'soapfish==0.6.0.dev0'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('soapfish==0.6.0.dev0', 'console_scripts', 'xsd2py')()
    )

/usr/local/bin/xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd
Traceback (most recent call last):
  File "/usr/local/bin/xsd2py", line 9, in <module>
    load_entry_point('soapfish==0.6.0.dev0', 'console_scripts', 'xsd2py')()
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 171, in main
    code = generate_code_from_xsd(xmlelement, encoding='utf-8')
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 69, in generate_code_from_xsd
    standalone=True)
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 151, in schema_to_py
    return tpl.render(schema=schema, cwd=cwd, base_path=base_path)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/templates/xsd", line 11, in top-level template code
    {{- resolve_import(i, known_files, schema.targetNamespace, cwd, base_path) }}
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 48, in resolve_import
    location = os.path.relpath(location, base_path)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 430, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 360, in abspath
    if not isabs(path):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 54, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'

So it seems the bug is still present in this branch -- sorry 😢

thomas-mangin · 2016-03-22T16:13:08Z

And sorry - I should have said I was running Nick's branch - but looking at the recent tickets it looked like the code to run to try parse complex XSDs.

FelixSchwarz · 2016-03-22T16:29:08Z

Thank you for your quick response. Unfortunately I'm short on time (easter around the corner) so if you need something faster than 2 weeks you need either to debug this yourself or look at another library...

attached reproducer file so it can't go away: cwmp14.txt

thomas-mangin · 2016-03-22T17:56:34Z

Being an open source developer too, I can appreciate your position. As it seems your library is the right fit for us, you will most likely see me pop up back here in a few weeks - I hope you won't mind the pestering if your life an work got in the way of happy hacking 😄

Thank you for your help so far.

FelixSchwarz · 2016-03-22T19:33:19Z

Looking forward to your questions/contributions :-)
Btw: An open source(?) TR069 project sounds interesting. Feel free to include pointers once there is something to see.

iurisilvio · 2016-03-22T21:58:47Z

It is related to my changes to support relative paths and these imports.

  <xs:import namespace="http://schemas.xmlsoap.org/soap/envelope/" schemaLocation="http://schemas.xmlsoap.org/soap/envelope/"/>
  <xs:import namespace="http://schemas.xmlsoap.org/soap/encoding/" schemaLocation="http://schemas.xmlsoap.org/soap/encoding/"/>

When I had this problem, I just removed these imports and it worked. Of course it is not the fix, just a quick hack.

I'll try to fix it soon.

ngnpope · 2016-03-23T12:14:41Z

@iurisilvio I have further changes I was working on related to the relative path issues so I'll look at your changes and compare to mine. I should be able to push further updates later today.

FelixSchwarz · 2016-03-23T13:20:47Z

Based on what @pope1ni said is it ok if I wait a bit with merging?

@iurisilvio: thank you very much for your quick fix, much appreciated.

iurisilvio · 2016-03-23T14:08:18Z

Yes, wait for @pope1ni. Feel free to use my changes or ignore it if you already did something to fix this issue. 👍

iurisilvio · 2017-03-26T20:03:43Z

I tested this XSD file today and it failed for another reason. This issue is fixed.

xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd

Now it fails with another error.

$ xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd
Traceback (most recent call last):
  File "/home/default/.virtualenvs/soapfish/bin/xsd2py", line 9, in <module>
    load_entry_point('soapfish==0.6.0.dev0', 'console_scripts', 'xsd2py')()
  File "/code/soapfish/soapfish/xsd2py.py", line 184, in main
    code = generate_code_from_xsd(xml, encoding='utf-8', cwd=cwd)
  File "/code/soapfish/soapfish/xsd2py.py", line 77, in generate_code_from_xsd
    standalone=standalone)
  File "/code/soapfish/soapfish/xsd2py.py", line 162, in schema_to_py
    return tpl.render(schema=schema, cwd=cwd, base_path=base_path)
  File "/home/default/.virtualenvs/soapfish/local/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/default/.virtualenvs/soapfish/local/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/code/soapfish/soapfish/templates/xsd", line 11, in top-level template code
    {{- resolve_import(i, known_paths, known_types, schema.targetNamespace, cwd, base_path) }}
  File "/code/soapfish/soapfish/xsd2py.py", line 46, in resolve_import
    xml = open_document(path)
  File "/code/soapfish/soapfish/utils.py", line 38, in open_document
    with open(path, 'rb') as f:
IOError: [Errno 2] No such file or directory: '/code/soapfish/https:/www.broadband-forum.org/cwmp/../../../../../../../../cwmp/cwmp-1-3.xsd'

First the command failed because the site does not accept requests User-Agent. I hard coded changed it just to test the issue.

ngnpope · 2017-04-19T11:17:02Z

If I recall, the issue we had was to do with whether imported files were local or remote and this would be affected in different ways depending on whether the initial path provided on the command line was local or remote.

In this instance, although the initial path was a remote path, the code originally checked whether the path started with a slash to determine whether it was local. In the remote case, however, starting with a slash indicates that the path is relative to the root of the domain. Worse, if the file is then downloaded from the initial path and used locally, these paths will be incorrect as they should be rewritten to remote paths rather than looking like absolute local paths. But there is probably not much we can do about that...

iurisilvio · 2017-04-19T13:45:20Z

You are correct. I fixed the problem with the relative paths for remote files, that were handled as local paths.

If you download the file and run it, I think all WSDL/XSD readers will fail too, because you don't have the original URL reference. Maybe we can provide a way to reference it from command line, but I don't think it is a real issue, you have to download all your xsd schema tree.

ngnpope · 2017-04-19T16:08:40Z

PR #93 was irrelevant to this issue, but PR #96 resolved the issue in the latest stacktrace above.

In addition, I fixed two additional problems in 2ef18c8 and 3ac2a6b that affected the schema file in question.

Now we are blowing up because we always expect <xs:restriction> as a child of <xs:simpleType> but don't handle <xs:union> as a child which contains <xs:restriction> with <xs:enumeration> underneath. Development of support for <xs:union> is being tracked in #38.

I've attached all of the schema files for future reference:

cwmp-1-1.xsd.txt
cwmp-1-2.xsd.txt
cwmp-1-3.xsd.txt
cwmp-1-4.xsd.txt

There are also other weird things that haven't event been considered in some of these such as <xs:redefine>.

FelixSchwarz added bug waiting for additional information labels Mar 22, 2016

FelixSchwarz added patches welcome and removed waiting for additional information labels Mar 22, 2016

iurisilvio mentioned this issue Mar 23, 2016

Set cwd when xsd2py is executed as __main__ and ignore xsd imports #84

Closed

ngnpope added this to the 0.6.0 milestone Jul 6, 2016

iurisilvio mentioned this issue Mar 26, 2017

Fix extensions with imported bases #93

Merged

ngnpope mentioned this issue Apr 19, 2017

Fix relative imports with remote files. #96

Merged

ngnpope modified the milestones: 0.6.0, 0.7.0 May 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

xsd2py failing #83

xsd2py failing #83

thomas-mangin commented Mar 22, 2016

ngnpope commented Mar 22, 2016

FelixSchwarz commented Mar 22, 2016

thomas-mangin commented Mar 22, 2016

thomas-mangin commented Mar 22, 2016

FelixSchwarz commented Mar 22, 2016

thomas-mangin commented Mar 22, 2016

FelixSchwarz commented Mar 22, 2016

iurisilvio commented Mar 22, 2016

ngnpope commented Mar 23, 2016

FelixSchwarz commented Mar 23, 2016

iurisilvio commented Mar 23, 2016

iurisilvio commented Mar 26, 2017

ngnpope commented Apr 19, 2017

iurisilvio commented Apr 19, 2017

ngnpope commented Apr 19, 2017

xsd2py failing #83

xsd2py failing #83

Comments

thomas-mangin commented Mar 22, 2016

ngnpope commented Mar 22, 2016

FelixSchwarz commented Mar 22, 2016

thomas-mangin commented Mar 22, 2016

thomas-mangin commented Mar 22, 2016

FelixSchwarz commented Mar 22, 2016

thomas-mangin commented Mar 22, 2016

FelixSchwarz commented Mar 22, 2016

iurisilvio commented Mar 22, 2016

ngnpope commented Mar 23, 2016

FelixSchwarz commented Mar 23, 2016

iurisilvio commented Mar 23, 2016

iurisilvio commented Mar 26, 2017

ngnpope commented Apr 19, 2017

iurisilvio commented Apr 19, 2017

ngnpope commented Apr 19, 2017