Author cykerway
Recipients barry, cykerway, methane, ncoghlan, terry.reedy, vstinner
Date 2018-08-07.21:40:42
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1533678043.0.0.56676864532.issue34296@psf.upfronthosting.co.za>
In-reply-to
Content
>   While this issue is "pre warming VM", VM startup is not significant part of your 500ms.

10-20ms should be OK for shell scripts. But a fork is still faster.  

>   You're talking about application specific strategy now. It's different of this issue.

Actually, this issue is created to look for a generic approach that can optimize the running time for most, or even all, python scripts. Different scripts may import different modules, but this doesn't mean there isn't a method that works for all of them.

>   And many ideas like yours are already discussed on ML, again and again.

I browsed about 6-7 threads on python-dev. I think 2-3 of them provide great information. But I don't think any of them gives concrete solutions. So we are still facing this problem today.

>   I want to close this issue. Please give us more concrete idea or patch with target sample application you want to optimize.

As said I'm looking for a generic approach. So optimizing specific applications isn't really the goal of this issue (though work on specific modules still helps). I did implement a proof of concept (link: <https://github.com/cykerway/pyforkexec>) for the fork-exec startup approach. It's still very rough and elementary, but proves this approach has its value. As Nick said:

>   ...the CPython runtime is highly configurable, so it's far from clear what, if anything, could be shared from run to run...

What I hope is we can inspect these configurations and figure out the invariants. This would help us make a clean environment as the forking base. If this is impossible, I think we can still fork from a known interpreter state chosen by the user script author. You may close this issue if nobody has more to say, but I hope the fork-exec startup can be standardized one day as I believe, for quick scripts, however much you optimize the cold start it can't be faster than a fork.
History
Date User Action Args
2018-08-07 21:40:43cykerwaysetrecipients: + cykerway, barry, terry.reedy, ncoghlan, vstinner, methane
2018-08-07 21:40:43cykerwaysetmessageid: <1533678043.0.0.56676864532.issue34296@psf.upfronthosting.co.za>
2018-08-07 21:40:42cykerwaylinkissue34296 messages
2018-08-07 21:40:42cykerwaycreate