Frames can be skipped only for warnings issued by imported module code, not for warnings issued by import machinery itself.

I propose following algorithm:

Before executing module code push the current frame and the number of frames to skip in global stack in the warnings module, and pop it after execution (public context manager can help). In warnings.warn() skip stacklevel frames and if saved frame was skipped, skip corresponding additional number of frames. Repeat for all saved frames.