freeze_support() line to be omitted..

[HardipinderS]

I'm getting this error

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Program Files\Python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Program Files\Python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\client-pypeln-pl.task.py", line 6, in <module>        
    import pypeln as pl
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\__init__.py", line 4, in <module>
    from . import thread
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\__init__.py", line 34, in <module>
    from .api.concat import concat
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\api\concat.py", line 8, in <module>
    from .to_stage import to_stage
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\api\to_stage.py", line 5, in <module>
    from ..stage import Stage
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\stage.py", line 7, in 
<module>
    from . import utils
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\utils.py", line 11, in <module>
    MANAGER = multiprocessing.Manager()
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 56, in Manager
    m.start()
  File "C:\Program Files\Python37\lib\multiprocessing\managers.py", line 563, in start
    self._process.start()
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__   
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data  
    _check_not_importing_main()
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "client-pypeln-pl.task.py", line 6, in <module>
    import pypeln as pl
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\__init__.py", line 4, in <module>
    from . import thread
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\__init__.py", line 34, in <module>
    from .api.concat import concat
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\api\concat.py", line 8, in <module>
    from .to_stage import to_stage
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\api\to_stage.py", line 5, in <module>
    from ..stage import Stage
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\stage.py", line 7, in 
<module>
    from . import utils
  File "C:\Users\HARDIPINDER\w00rkspace\tufin\lib\site-packages\pypeln\thread\utils.py", line 11, in <module>
    MANAGER = multiprocessing.Manager()
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 56, in Manager
    m.start()
  File "C:\Program Files\Python37\lib\multiprocessing\managers.py", line 567, in start
    self._address = reader.recv()
  File "C:\Program Files\Python37\lib\multiprocessing\connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "C:\Program Files\Python37\lib\multiprocessing\connection.py", line 306, in _recv_bytes      
    [ov.event], False, INFINITE)

I was trying the code from [https://medium.com/@cgarciae/making-an-infinite-number-of-requests-with-python-aiohttp-pypeln-3a552b97dc95] and was to hoping to get an error for server not on or something but it throwed me this.

I tried adding the call to each task in if clause but still this error.

if name == 'main':
pl.task.each(
fetch,
urls,
workers=limit,
on_start=lambda: dict(session=ClientSession(connector=TCPConnector(limit=None))),
on_done=lambda session: session.close(),
run=True,
)

can you please help out where I might be going wrong?

[cgarciae]

Hey, thanks for reporting!

Can you post your pypeln version + OS information?

[cgarciae]

Are you running exactly this example?

https://github.com/cgarciae/pypeln/blob/master/benchmarks/100_million_downloads/client-pypeln-io.py

[HardipinderS]

Hey @cgarciae

versions are as below:
pypeln 0.4.3
OS Windows 10 build 1909, Windows Server 2016

I'm trying the it on both the server and pc, as wanted to implement in an API data fetch script.

And yes that it is the same code I used, just with a difference of If main clause

[cgarciae]

Can you try to run this code on your OS? Sadly I cant test on windows.

import multiprocessing

MANAGER = multiprocessing.Manager()

[HardipinderS]

It worked, no errors as such.

[cgarciae]

Can you try to run the last line twice?

You caught a bug in which the threading module is creating a multiprocessing manager (it shouldn't), so there are 2 now.

[HardipinderS]

tried with and without name change still this

I was searching side by side and had found this article, maybe this might give some insight.
https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection

[cgarciae]

I am thinking of lazy loading the MANAGER object, I think this might solve the issue and its good in general. I'll create a branch.

[HardipinderS]

same thing is happening without if clause, I was trying if just for the sake of traceback. great, please let me know if I can help with anything.

being new to async, not sure where to look under the hood.

[cgarciae]

Thanks a lot!

Can you try the feature/fix-manager branch?

pip install git+https://github.com/cgarciae/pypeln.git@feature/fix-manager

Sadly this is more of a Windows thing related to the pl.process module that you aren't even using.

[HardipinderS]

sure, tried this, Working and showing successfully build.

[cgarciae]

@HardipinderS does the example work now?

[HardipinderS]

no still same error

[cgarciae]

@Davidnet is it possible to test on windows somehow?

[cgarciae]

@HardipinderS can you post the new error? The MANAGER = ... line should not be called upon import now if the installation was successful.

[HardipinderS]

Hey @cgarciae sorry for the delayed reply, still able to call the manager.

[cgarciae]

I mean, what is the new error on your sample code?

[HardipinderS]

same error is there

[cgarciae]

@HardipinderS are you using PyCharm by any chance? Checkout #52

[HardipinderS]

No VS code and one bash and other time cmd