As to not being able to delete a loaded DLL on Windows, a workaround that may help in some instances is to rename it to a temporary name on the same volume. This is useful for upgrading in place. If you have admin privileges you can even flag the renamed DLL to be deleted when the system reboots:

    MoveFileExW(renamed_dll_path, NULL, MOVEFILE_DELAY_UNTIL_REBOOT)

Renaming a loaded DLL succeeds because the system opens the file with delete sharing, which allows renaming (i.e. relinking) an open file. 

To verify this, let's check the associated file object that's used by the mapping for python35.dll. We can see this in a local kernel debugging session. The following is in 64-bit Windows 7.

Attach to the Python process address space:

    lkd> .tlist python.exe
     0n2296 python.exe
    lkd> !process 0n2296 0
    Searching for Process with Cid == 8f8
    Cid handle table at fffff8a003506000 with 951 entries in use

    PROCESS fffffa800712fb10
        SessionId: 0  Cid: 08f8    Peb: 7fffffdf000  ParentCid: 02b8
        DirBase: 20780d000  ObjectTable: fffff8a0189fe1a0  HandleCount:  67.
        Image: python.exe

    lkd> .process /p /r 0xfffffa800712fb10 
    Implicit process is now fffffa80`0712fb10
    Loading User Symbols
    ...................................

Get the base address for python35.dll.

    lkd> lm m python35
    start             end                 module name
    00000000`66fb0000 00000000`67393000   python35   (deferred)

Get the associated Virtual Address Descriptor (VAD).

    lkd> !vad 66fb0000
    VAD             level  start  end    commit
    fffffa800855f230 (-1)  66fb0  67392  151 Mapped  Exe
                                         EXECUTE_WRITECOPY
                                         \Program Files\Python35\python35.dll

Get the File object reference from the associated control area.

    lkd> ?? ((nt!_MMVAD *)0xfffffa800855f230)->Subsection->ControlArea->FilePointer
    struct _EX_FAST_REF
       +0x000 Object           : 0xfffffa80`093f1503 Void
       +0x000 RefCnt           : 0y0011
       +0x000 Value            : 0xfffffa80`093f1503

The low nibble in the above address is information that needs to be masked out, so the address is actually 0xfffffa80`093f1500. Verify we have the correct file object by checking the filename.
       
    lkd> ?? ((nt!_FILE_OBJECT *)0xfffffa80`093f1500)->FileName
    struct _UNICODE_STRING
     "\Program Files\Python35\python35.dll"
       +0x000 Length           : 0x48
       +0x002 MaximumLength    : 0x78
       +0x008 Buffer           : 0xfffff8a0`04b23690
                                 "\Program Files\Python35\python35.dll"

Finally, check the sharing mode. We see below that the file for a DLL mapping is loaded with read and delete sharing, but not write sharing.

    lkd> ?? ((nt!_FILE_OBJECT *)0xfffffa80`093f1500)->SharedRead
    unsigned char 0x01 ''
    lkd> ?? ((nt!_FILE_OBJECT *)0xfffffa80`093f1500)->SharedDelete
    unsigned char 0x01 ''
    lkd> ?? ((nt!_FILE_OBJECT *)0xfffffa80`093f1500)->SharedWrite
    unsigned char 0x00 ''

Thus when deleting a DLL fails, it's not due to ERROR_SHARING_VIOLATION (32). The sharing mode would actually permit deleting the file. Instead the error is ERROR_ACCESS_DENIED (5), which results from the system call NtSetInformationFile returning STATUS_CANNOT_DELETE (0xC0000121), because the file currently has a mapped image or data section. But that doesn't stop us from renaming it to get it out of the way of an in-place upgrade.