classification
Title: open 'PhysicalDriveN' on windows fails (since python 3.5) with OSError: [WinError 1] Incorrect function
Type: behavior Stage: resolved
Components: IO, Library (Lib), Windows Versions: Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: tempfile.TemporaryFile fails when dir option set to directory residing on host OS mount
View: 25717
Assigned To: Nosy List: benjamin.peterson, eryksun, paul.moore, pitrou, rokozh, steve.dower, stutzbach, tim.golden, vstinner, zach.ware
Priority: normal Keywords: 3.5regression

Created on 2015-11-16 21:42 by rokozh, last changed 2016-02-11 01:43 by eryksun. This issue is now closed.

Messages (5)
msg254756 - (view) Author: Roman Kozhemiakin (rokozh) Date: 2015-11-16 21:42
open('\\\\.\\PHYSICALDRIVE1','rb',0)
fails since python 3.5

At the end of _io_FileIO___init___impl function 
_Py_fstat call raise OSError: [WinError 1] Incorrect function
_Py_fstat result not used on windows in this place. 

     440     self->blksize = DEFAULT_BUFFER_SIZE;
---> 441     if (_Py_fstat(self->fd, &fdfstat) < 0)
     442         goto error;
     443 #if defined(S_ISDIR) && defined(EISDIR)
     444     /* On Unix, open will succeed for directories.
     445        In Python, there should be no file objects referring to
     446        directories, so we need a check.  */
     447     if (S_ISDIR(fdfstat.st_mode)) {
     448         errno = EISDIR;
     449         PyErr_SetFromErrnoWithFilenameObject(PyExc_IOError, nameobj);
     450         goto error;
     451     }
     452 #endif /* defined(S_ISDIR) */
     453 #ifdef HAVE_STRUCT_STAT_ST_BLKSIZE
     454     if (fdfstat.st_blksize > 1)
     455         self->blksize = fdfstat.st_blksize;
     456 #endif /* HAVE_STRUCT_STAT_ST_BLKSIZE */
msg254784 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-11-17 06:39
As a workaround you can open a file descriptor via os.open:

    >>> import os
    >>> fd = os.open(r'\\.\PhysicalDrive0', os.O_RDONLY | os.O_BINARY)
    >>> os.read(fd, 512)[:8]
    b'3\xc0\x8e\xd0\xbc\x00|\x8e'

> _Py_fstat result not used on windows in this place. 

While it's pretty much useless, it is possible to open a CRT file descriptor for a directory handle:

    >>> h = _winapi.CreateFile('C:\\', 0x80000000, 3, 0, 3, 0x02000000, 0)
    >>> fd = msvcrt.open_osfhandle(h, os.O_RDONLY)
    >>> open(fd)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IsADirectoryError: [Errno 21] Is a directory: 3

So calling _Py_fstat shouldn't be skipped on Windows.

The issue is that a raw disk device doesn't support querying information (IRP_MJ_QUERY_INFORMATION), as shown in this local kernel debugging session:

    lkd> !object \GLOBAL??\PhysicalDrive0
    [...]
        Target String is '\Device\Harddisk0\DR0'

    lkd> !object \Device\Harddisk0\DR0
    Object: ffffe001c4fa4500  Type: (ffffe001c3d7bd30) Device
    [...]

    lkd> !devobj ffffe001c4fa4500
    Device object (ffffe001c4fa4500) is for:
     DR0 \Driver\disk DriverObject ffffe001c4fa3310
    [...]

    lkd> !drvobj \Driver\disk 2
    [...]
    Dispatch routines:
    [...]
    [05] IRP_MJ_QUERY_INFORMATION  fffff8037251a06c  nt!IopInvalidDeviceRequest
    [06] IRP_MJ_SET_INFORMATION    fffff8037251a06c  nt!IopInvalidDeviceRequest
    [...]

Specifically WinAPI GetFileInformationByHandle calls the system function NtQueryInformationFile. The system call makes an IRP_MJ_QUERY_INFORMATION I/O request of the driver, which for a raw disk device is handled by IopInvalidDeviceRequest. This fails the request with STATUS_INVALID_DEVICE_REQUEST (0xC0000010), which translates to WinAPI ERROR_INVALID_FUNCTION (0x0001). 

Possibly in the case of ERROR_INVALID_FUNCTION, FileIO can just call PyErr_Clear and skip the code that uses fdfstat.
msg254790 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-17 10:47
I don't know the physical disk type on Windows. Can you read and write from such "file" type? If no, I don't think that it makes sense to support it in io.FileIO. What do you think?

It looks like the file must be opened with specific options, otherwise it doesn't work. See this question on CreateFile():
https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/fa2175fe-02f4-4adb-9cb9-5df3d1122cf6/readfile-works-when-handle-opened-with-drive-letterphysical-drive-symlink-but-not-with-device-path


io.FileIO() calls _Py_fstat() for two reasons:

1) raise an error if the path/file descriptor is a directory
2) get the block size

_Py_fstat() is implemented with two Windows calls: GetFileType() & GetFileInformationByHandle().

To solve this issue, we need to have a function telling that the path/file descriptor is a physical disk. In this case, we can skip both checks.

Documentation on Physical Drivers in CreateFile():
https://msdn.microsoft.com/en-us/library/aa363858%28VS.85%29.aspx#physical_disks_and_volumes
msg254797 - (view) Author: Roman Kozhemiakin (rokozh) Date: 2015-11-17 11:55
>I don't know the physical disk type on Windows. Can you read and write from such "file" type?

Yes this "files" can be readed and writed (with restriction - size of the data must be a multiple of the sector size)

in python 3.4.3 open,read,write,seek - works fine (with raw usb disk)
for python 3.5 as workaround I replaced these functions to os.xxxx

>2) get the block size
st_blksize is not implemented under windows
msg260065 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-02-11 01:43
I'm closing this as a duplicate of issue 25717. Martin's patch fixes this problem since it's just another case of _Py_fstat failing for a valid file. In this case the problem is volume and disk files (e.g. r"\\.\C:" and r"\\.\PhysicalDrive1") that don't support querying file information.
History
Date User Action Args
2016-02-11 01:43:11eryksunsetstatus: open -> closed
superseder: tempfile.TemporaryFile fails when dir option set to directory residing on host OS mount
messages: + msg260065

resolution: duplicate
stage: test needed -> resolved
2015-11-17 11:55:08rokozhsetmessages: + msg254797
2015-11-17 10:47:45vstinnersetmessages: + msg254790
2015-11-17 06:39:23eryksunsetnosy: + eryksun
messages: + msg254784
2015-11-16 22:09:20serhiy.storchakasetnosy: + vstinner
2015-11-16 21:45:11SilentGhostsetkeywords: + 3.5regression
nosy: + stutzbach, pitrou, paul.moore, tim.golden, benjamin.peterson, zach.ware, steve.dower

components: + Windows, IO
stage: test needed
2015-11-16 21:42:09rokozhcreate