Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compilers do not use django storage to save files #608

Open
miki725 opened this issue Dec 26, 2016 · 9 comments · May be fixed by #609
Open

compilers do not use django storage to save files #608

miki725 opened this issue Dec 26, 2016 · 9 comments · May be fixed by #609
Labels
bug Bug confirmed or can be reproduced

Comments

@miki725
Copy link
Contributor

miki725 commented Dec 26, 2016

For example when using S3 to serve static files with combination of less compiler, none of the less converted files are uploaded to S3 since less is a subprocess compile which simply calls lessc command which stores the file locally. Am I doing something wrong with using pipeline in combination with S3? I followed suggestions at https://django-pipeline.readthedocs.io/en/latest/storages.html#using-with-other-storages

here is my storage backend:

from django.contrib.staticfiles.storage import ManifestFilesMixin
from pipeline.storage import PipelineMixin
from storages.backends.s3boto import S3BotoStorage

class StaticS3BotoStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
    pass

from quick glance seems issue could be related to https://github.com/jazzband/django-pipeline/blob/master/pipeline/compilers/__init__.py#L40 since that simply calls the compiler but later on does not use the storage to save that compiled result via django storage.

any ideas?
@miki725
Copy link
Contributor Author

miki725 commented Dec 26, 2016

seems this is actually related when compiler is used with compressor. so lessc creates file locally but then when attempting to compress that file, compressor attempts to read it via storage which does not have it yet.

here is exact traceback btw:

Traceback (most recent call last):
  File "./manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle
    collected = self.collect()
  File "/usr/local/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 139, in collect
    for original_path, processed_path, processed in processor:
  File "/usr/local/lib/python3.5/site-packages/pipeline/storage.py", line 26, in post_process
    packager.pack_stylesheets(package)
  File "/usr/local/lib/python3.5/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
    variant=package.variant, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/pipeline/packager.py", line 107, in pack
    content = compress(paths, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/pipeline/compressors/__init__.py", line 75, in compress_css
    css = self.concatenate_and_rewrite(paths, output_filename, variant)
  File "/usr/local/lib/python3.5/site-packages/pipeline/compressors/__init__.py", line 139, in concatenate_and_rewrite
    content = self.read_text(path)
  File "/usr/local/lib/python3.5/site-packages/pipeline/compressors/__init__.py", line 223, in read_text
    content = self.read_bytes(path)
  File "/usr/local/lib/python3.5/site-packages/pipeline/compressors/__init__.py", line 217, in read_bytes
    file = staticfiles_storage.open(path)
  File "/usr/local/lib/python3.5/site-packages/django/core/files/storage.py", line 38, in open
    return self._open(name, mode)
  File "/usr/local/lib/python3.5/site-packages/storages/backends/s3boto.py", line 393, in _open
    raise IOError('File does not exist: %s' % name)
OSError: File does not exist: somepath/foo.css

@miki725
Copy link
Contributor Author

miki725 commented Dec 26, 2016 •
edited
Loading

so the solution would probably be somehow would have to save all the paths before compression in https://github.com/jazzband/django-pipeline/blob/master/pipeline/packager.py#L105

@miki725 miki725 linked a pull request Dec 26, 2016 that will close this issue
Open
@miki725
Copy link
Contributor Author

miki725 commented Dec 26, 2016 •
edited
Loading

just sent PR with a fix which seems to work - #609

@pythdasch
Copy link

Hi I think I have the same issue as you @miki725 . How did you make it work ?

@miki725
Copy link
Contributor Author

miki725 commented Feb 14, 2017

@pythdasch just using the fork at the moment

@pythdasch
Copy link

thanks :)

@pythdasch
Copy link

@miki725 Not working for me. I guess it's not a storage problem, but more a compiler problem. so annoying

@SverkerSbrg
Copy link

+1

@triat triat added the bug Bug confirmed or can be reproduced label Apr 2, 2017
@sharov sharov mentioned this issue Jun 21, 2018
Open
@sharov
Copy link

sharov commented Jun 21, 2018

I've sent PR #663, which work with original Django storages & storages from package 'django-storages'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Bug confirmed or can be reproduced
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

saving compiler result via storage backend miki725/django-pipeline
5 participants
@sharov @miki725 @triat @pythdasch @SverkerSbrg