Skip to content

Spawned-thread imports recurse in importlib _blocking_on bookkeeping #7586

@smarcd

Description

@smarcd

Summary

On current RustPython tip, imports executed on a spawned VM thread fail with:

  • RecursionError: in comparison

This reproduces in bare RustPython embedding, outside any PyO3 integration.

RustPython version

Local cargo git checkout pinned at d201c48e1.

Minimal repro

use rustpython::InterpreterBuilderExt;
use rustpython_vm::InterpreterBuilder;

fn main() {
    let interpreter = InterpreterBuilder::new().init_stdlib().interpreter();
    interpreter.enter(|vm| {
        vm.start_thread(|vm| {
            let result = vm.import("collections", 0);
            match result {
                Ok(_) => println!("collections ok"),
                Err(err) => {
                    let mut rendered = String::new();
                    let _ = vm.write_exception(&mut rendered, &err);
                    println!(
                        "import err class={} msg={}",
                        err.class().name(),
                        rendered.trim_end()
                    );
                }
            }
        })
        .join()
        .unwrap();
    });
}

Observed traceback

Traceback (most recent call last):
  File ".../Lib/collections/__init__.py", line 29, in <module>
    import _collections_abc
  File ".../Lib/_collections_abc.py", line 35, in <module>
    from abc import ABCMeta, abstractmethod
  File ".../Lib/abc.py", line 85, in <module>
    from _abc import ...
  File "_frozen_importlib", line 1368, in _find_and_load
  File "_frozen_importlib", line 421, in __enter__
  File "_frozen_importlib", line 311, in acquire
  File "_frozen_importlib", line 170, in __enter__
RecursionError: in comparison

Narrowing

These succeed on a spawned VM thread:

  • direct dict.setdefault(...)
  • raw _weakref.ref(...)
  • import abc
  • import _collections_abc

These fail on a spawned VM thread:

  • import weakref
  • import collections
  • import array because it pulls in collections

The failure consistently lands in:

  • Lib/importlib/_bootstrap.py
  • _BlockingOnManager.__enter__()
  • _ModuleLock.acquire()
  • _blocking_on bookkeeping

Additional notes

  • This still reproduces when using vm.start_thread(...), so it is not just an arbitrary non-owner-thread embedding misuse.
  • Monkeypatch experiments suggest the bug is not just _WeakValueDictionary itself; the broader import-lock bookkeeping path seems to be involved.

Why this matters

I am working on a PyO3 frontend/backend split with a RustPython backend. A correct backend design needs a reliable spawned-thread/runtime-thread execution model, and this currently blocks that work.

Related context:

If useful, I can provide the standalone repro crate as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions