classification
Title: struct.unpack generates wrong error in certain conditions
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: SylvainDe, knzsys, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-06-20 13:32 by knzsys, last changed 2017-06-23 07:25 by knzsys. This issue is now closed.

Messages (10)
msg296428 - (view) Author: knzsys (knzsys) Date: 2017-06-20 13:37
#Working code:
from struct import *
rawdata = 'A'*196
laserdata = list(unpack('2s2s29H',rawdata[14:76]))
laserdata += list(unpack('26f3L4s',rawdata[76:196]))					

#This code generates error:
from struct import *
rawdata = 'A'*196
laserdata = list(unpack('2s2s29H26f3L4s',rawdata[14:196]))
# Does not work due to python bug: unpack calculates wrong size 184 (should be 182) from format string and generates error: 

#struct.error: unpack requires a string argument of length 184
msg296438 - (view) Author: knzsys (knzsys) Date: 2017-06-20 14:24
the pack function has the same error: it adds additional 2 bytes
msg296444 - (view) Author: SylvainDe (SylvainDe) * Date: 2017-06-20 14:47
Could it be caused by alignment ?

Also, it could be interesting to explain a bit more what you expect and what you actually get.
msg296449 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-20 15:05
Don't forget about an alignment. If the first character is not one of '@', '=', '<', '>' or '!', '@' is assumed, which means native byte order, type sizes and alignments. If the 'f' format character require 4-byte alignment, 2 padding bytes are inserted before the first 'f' item.

Add the '=' character at the start of the format string if you need native byte order, "standard" type sizes and no alignments. Add the '<' or '>' characters if you need some specific byte order.
msg296521 - (view) Author: knzsys (knzsys) Date: 2017-06-21 05:48
Thanks a lot.
changing format string from '2s2s29H26f3L4s' to '=2s2s29H26f3L4s' worked for me.

But I still do not understand, why it's working if I splitt format string in 2 parts + use the (un)pack command 2 times ('2s2s29H' + '26f3L4s' are working separately, but '=2s2s29H26f3L4s' needs the alignment specified.

Do someone have a link, where the alignment options are deeply explained?
msg296522 - (view) Author: knzsys (knzsys) Date: 2017-06-21 06:11
just an example, how the pack function behaves with the same format string:

before: 13830108be00cafe....3e05ecbc5138000000043c9be14125359541bf338044000000007452e041.....
after:  13830108be00cafe....3e05ecbc51380000000400003c9be14125359541bf338044000000007452e041....

the pack function inserts 2 0x00 bytes at byte pos 62.

which means the first f item converts to 0x0000AABBCCDD instead to 0xAABBCCDD (float value AABBCCDD is an example).
if the formatstring starts with f it converts to 0xAABBCCDD.

Why this strange behavior?
msg296528 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-21 08:30
https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment

"""
Notes:
1. Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct.
"""
msg296542 - (view) Author: knzsys (knzsys) Date: 2017-06-21 10:48
many many thanks for your help!
and fast responses :-)
this explain my oberservations

why only the first 2 bytes are padded?
0xAABBCCDD (4 bytes = real32) to 0x0000AABBCCDD (6 bytes)
and not to 0x0000AABB0000CCDD (8 bytes)

it has something to do with the behavior of the  C compiler’s sizeof expression? and why?

since I need pack / unpack often in my projects, i want to have it completely clarified.
msg296544 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-21 11:03
Because the size of '2s2s29H' is 62 bytes. If the following C float needs to be aligned on 4 bytes boundary, it should start at offset 64, after 2 padding bytes.
msg296686 - (view) Author: knzsys (knzsys) Date: 2017-06-23 07:25
Ah OKAY!
That's the key which makes all clear to me!

I didn't thought at that!

Thanks a lot Serhiy Storchaka for your help!
History
Date User Action Args
2017-06-23 07:25:30knzsyssetmessages: + msg296686
2017-06-21 11:03:06serhiy.storchakasetmessages: + msg296544
2017-06-21 10:48:25knzsyssetmessages: + msg296542
2017-06-21 08:30:24serhiy.storchakasetmessages: + msg296528
2017-06-21 06:11:35knzsyssetmessages: + msg296522
2017-06-21 05:48:28knzsyssetmessages: + msg296521
2017-06-20 15:05:48serhiy.storchakasetstatus: open -> closed

nosy: + serhiy.storchaka
messages: + msg296449

resolution: not a bug
stage: resolved
2017-06-20 14:47:58SylvainDesetmessages: + msg296444
2017-06-20 14:38:15serhiy.storchakasetmessages: - msg296425
2017-06-20 14:24:55knzsyssetmessages: + msg296438
2017-06-20 13:53:32SylvainDesetnosy: + SylvainDe
2017-06-20 13:37:45knzsyssetmessages: + msg296428
2017-06-20 13:35:53knzsyssettype: crash -> behavior
2017-06-20 13:32:47knzsyscreate