python - 为 jupyter 覆盖 * 全局导入

问题描述

我在 Windows 上运行 jupyter lab 并fastai.vision.utils.verify_images(fns)给我带来了问题，因为它fastcore.parallel.parallel使用 default调用n_workers=8。有很多方法可以解决它，但我试图找出一个代码块，我可以将它放在任何笔记本上并拥有它，这样所有底层调用都parallel将使用n_workers=1.

我尝试了以下单元格：

import fastcore
import sys
_fastcore = fastcore
_parallel = lambda *args, **kwargs: fastcore.parallel.parallel(*args, **kwargs, n_workers=1)
_fastcore.parallel.parallel = _parallel
sys.modules['fastcore'] = _fastcore
fastcore.parallel.parallel

印刷

<function __main__.<lambda>(*args, **kwargs)>

但是当我尝试运行 verify_images 它仍然失败，好像补丁从未发生过

---------------------------------------------------------------------------
BrokenProcessPool                         Traceback (most recent call last)
<ipython-input-37-f1773f2c9e62> in <module>
      3 # from mock import patch
      4 # with patch('fastcore.parallel.parallel') as _parallel:
----> 5 failed = verify_images(fns)
      6 # failed = L(fns[i] for i,o in enumerate(_parallel(verify_image, fns)) if not o)
      7 failed

~\anaconda3\lib\site-packages\fastai\vision\utils.py in verify_images(fns)
     59 def verify_images(fns):
     60     "Find images in `fns` that can't be opened"
---> 61     return L(fns[i] for i,o in enumerate(parallel(verify_image, fns)) if not o)
     62 
     63 # Cell

~\anaconda3\lib\site-packages\fastcore\parallel.py in parallel(f, items, n_workers, total, progress, pause, threadpool, timeout, chunksize, *args, **kwargs)
    121             if total is None: total = len(items)
    122             r = progress_bar(r, total=total, leave=False)
--> 123         return L(r)
    124 
    125 # Cell

~\anaconda3\lib\site-packages\fastcore\foundation.py in __call__(cls, x, *args, **kwargs)
     95     def __call__(cls, x=None, *args, **kwargs):
     96         if not args and not kwargs and x is not None and isinstance(x,cls): return x
---> 97         return super().__call__(x, *args, **kwargs)
     98 
     99 # Cell

~\anaconda3\lib\site-packages\fastcore\foundation.py in __init__(self, items, use_list, match, *rest)
    103     def __init__(self, items=None, *rest, use_list=False, match=None):
    104         if (use_list is not None) or not is_array(items):
--> 105             items = listify(items, *rest, use_list=use_list, match=match)
    106         super().__init__(items)
    107 

~\anaconda3\lib\site-packages\fastcore\basics.py in listify(o, use_list, match, *rest)
     54     elif isinstance(o, list): res = o
     55     elif isinstance(o, str) or is_array(o): res = [o]
---> 56     elif is_iter(o): res = list(o)
     57     else: res = [o]
     58     if match is not None:

~\anaconda3\lib\concurrent\futures\process.py in _chain_from_iterable_of_lists(iterable)
    482     careful not to keep references to yielded objects.
    483     """
--> 484     for element in iterable:
    485         element.reverse()
    486         while element:

~\anaconda3\lib\concurrent\futures\_base.py in result_iterator()
    609                     # Careful not to keep a reference to the popped future
    610                     if timeout is None:
--> 611                         yield fs.pop().result()
    612                     else:
    613                         yield fs.pop().result(end_time - time.monotonic())

~\anaconda3\lib\concurrent\futures\_base.py in result(self, timeout)
    437                 raise CancelledError()
    438             elif self._state == FINISHED:
--> 439                 return self.__get_result()
    440             else:
    441                 raise TimeoutError()

~\anaconda3\lib\concurrent\futures\_base.py in __get_result(self)
    386     def __get_result(self):
    387         if self._exception:
--> 388             raise self._exception
    389         else:
    390             return self._result

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

我怀疑这与fastai.vision.utils使用 * 导入 fastcore 有关。有没有办法实现我想要的？

标签： pythonjupyter

python - 为 jupyter 覆盖 * 全局导入

问题描述

解决方案

推荐阅读