On Wed, Feb 13, 2019 at 5:09 PM Steve Dower <report@bugs.python.org> wrote:
> This is why I'm keen to design the ideal *user* API first (that is, write the examples of how you would use it) and then figure out how we can make it fit.
> It's kind of the opposite approach from what you've been doing to adapt the existing code to suit particular needs.

That makes sense. :)

> For example, imagine instead of all the PySet*() functions followed by Py_Initialize() you could do this:
>
>     PyObject *runtime = PyRuntime_Create();

FYI, we already have a _PyRuntimeState struct (see
Include/internal/pycore_pystate.h) which is where I pulled in a lot of
the static globals last year.  Now there is one process-global
_PyRuntime (created in Python/pylifecycle.c) in place of all those
globals.  Note that _PyRuntimeState is in parallel with
PyInterpreterState, so not a PyObject.

>     /* optional calls */
>     PyRuntime_SetAllocators(runtime, &my_malloc, &my_realloc, &my_free);
>     PyRuntime_SetHashSeed(runtime, 12345);

Note that one motivation behind PEP 432 (and its config structs) is to
keep all the config together.  Having the one struct means you always
clearly see what your options are.  Another motivation is to keep the
config (dense with public fields) separate from the actual run state
(opaque).  Having a bunch of config functions (and global variables in
the status quo) means a lot more surface area to deal with when
embedding, as opposed to 2 config structs + a few initialization
functions (and a couple of helpers) like in PEP 432.

I don't know that you consciously intended to move away from the dense
config struct route, so I figured I'd be clear. :)

>     /* sets this as the current runtime via a thread local */
>     auto old_runtime = PyRuntime_Activate(runtime);
>     assert(old_runtime == NULL)

Hmm, there are two ways we could go with this: keep using TLS (or
static global in the case of _PyRuntime) or update the C-API to
require explicitly passing the context (e.g. runtime, interp, tstate,
or some wrapper) into all the functions that need it.  Of course,
changing that would definitely need some kind of compatibility shim to
avoid requiring massive changes to every extension out there, which
would mean effectively 2 C-APIs mirroring each other.  So sticking
with TLS is simpler.  Personally, I'd prefer going the explicit
argument route.

>
>     /* pretend triple quoting works in C for a minute ;) */
>     const char *init_code = """
> [snip]
>     """;
>
>     PyObject *globals = PyDict_New();
>     /* only UTF-8 support at this stage */
>     PyDict_SetItemString(globals, "argv0", PyUnicode_FromString(argv[0]));
>     PyRuntime_Initialize(runtime, init_code, globals);

Nice.  I like that this keeps the init code right by where it's used,
while also making it much more concise and easier to follow (since
it's Python code).

>     PyEval_EvalString("open('file.txt', 'w', encoding='gb18030').close()");

I definitely like the approach of directly embedding the Python code
like this. :)  Are there any downsides?

> Maybe it's a terrible idea?

Nah, we definitely want to maximize simplicity and your example offers
a good shift in that direction. :)

> Honestly I'd be inclined to do other big changes at the same time (make PyObject opaque and interface driven, for example).

Definitely!  Those aren't big blockers on cleaning up initialization
though, are they?

> My point is that if the goal is to "move the existing internals around" then that's all we'll ever achieve. If we can say "the goal is to make this example work" then we'll be able to do much more.

Yep.  I suppose part of the problem is that the embedding use cases
aren't understood (or even recognized) well enough.