This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Py_BuildValue("y#".... returns incomplete result
Type: performance Stage: resolved
Components: C API Versions: Python 3.8
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, dwoodjunkmail, eric.smith
Priority: normal Keywords:

Created on 2021-03-08 15:31 by dwoodjunkmail, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (13)
msg388268 - (view) Author: David Wood (dwoodjunkmail) Date: 2021-03-08 15:31
I have a c function to encrypt values which returns an array of bytes.  The function returns proper values outside of python.  When used as a python function, the result is incomplete usually 10-20% of the time.  If I add a sleep(1) call before returning from the function, my success rate goes to 100%.  While this works, it is unacceptable as it will create enormous latency in my application.

static PyObject *method_encrypt(PyObject *self, PyObject *args) {
	char *keyval, *str = NULL, output[512];
	Py_ssize_t count=0;
	PyObject *retval;

	if(!PyArg_ParseTuple(args, "ss", &str, &keyval)) {
		return NULL;

	encryptBlowfishCfb(str, &count, output, keyval);

	retval = Py_BuildValue("y#", output, count);
	return retval;
msg388269 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-08 15:37
What do you mean by "incomplete"? Does it return less data or invalid data? Could you please paste your implementation of encryptBlowfishCfb(), too?
msg388270 - (view) Author: David Wood (dwoodjunkmail) Date: 2021-03-08 16:00
I have not gone to the extent of comparing the direct output against what python gets, although it appears to be incomplete.  The following is my encrypt function which has been used successfully for many years in a separate c program, so I have high confidence in it.

char * encryptBlowfishCfb(const char * inStr, Py_ssize_t *count, char * output, char * encrkey) {
	char IV[10];
	int len, i;
	char block_buffer[512];
	time_t t;

	srand((unsigned) time(&t));
	strcpy(IV, "");
	for (i = 0 ; i < 8 ; i++ ) {
		IV[i] =  mset[(rand() % 62)];
	td = mcrypt_module_open(MCRYPT_BLOWFISH, NULL, MCRYPT_CFB, NULL);
	mcrypt_generic_init(td, encrkey, strlen(encrkey), IV);
	memset(block_buffer, 0, sizeof(block_buffer));
	strcpy(block_buffer, inStr);
	i = strlen(inStr);
	mcrypt_generic(td, &block_buffer, i);
	block_buffer[i] = '\0';
	strcpy(&block_buffer[i], IV);

	len = i + 8 + 1;
	memset(output, 0, 512);
	strncpy(output, block_buffer, len -1);
	*count = i + 8;

	return output;
msg388273 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-08 17:47
Does mcrypt_generic() output base64 or ASCII-only data? Since you are converting the output to bytes, I assume the output may contain any byte. In that case strcpy() is not safe. You have to use memcpy().

Fun fact: Nintendo had a similar bug many years ago, check out "trucha bug"
msg388306 - (view) Author: David Wood (dwoodjunkmail) Date: 2021-03-08 21:31
Christian -

Thank you for this.  I did as you suggested however, I still have the same problem.  As I pointed out in my original message, the problem does not exist if I insert a sleep(1) statement prior to returning from the function.  Additional to that, I previously had printf('.') statement in place of the sleep statement.  When I ran 10,000 iterations in python and printed my pass/fail totals, I still had a few hundred dots print which tells me that there has to be some sort of timing or buffer transfer issue between c and python.  Does that make sense?
msg388308 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-08 21:43
Py_BuildValue("y#", output, count) is equivalent to PyBytes_FromStringAndSize(output, count). The function returns a copy of the input string as a new bytes object. It's very unlikely that the code is broken. It's been around for ages and it's a core feature.
msg388310 - (view) Author: David Wood (dwoodjunkmail) Date: 2021-03-08 21:52
Still no bueno.
msg388367 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-03-09 15:52

If you could give us an example showing the inputs, the actual outputs, and how they differ from what you expect, that would be helpful. Otherwise it's a lot of guessing on our part.
msg388369 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-09 16:07
A sleep(1) call affects exactly one aspect of the program: the state of the PRNG rand(). You re-initialize the process globale RNG in every function call with srand((unsigned) time(&t)). time() has a granularity of one second.
msg388402 - (view) Author: David Wood (dwoodjunkmail) Date: 2021-03-10 01:37
Attached are the basic files.  As you can see in the example, I am generating a random string, encrypting it, decrypting it, and comparing the decrypted result with the original value.

This is intended to run on linux and requires the mcrypt library (libmcrypt-dev on debian based distro).  I'm at a loss and any help is greatly appreciated.
msg388411 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-10 08:46
Could you please give us an example for an incorrect output and corresponding correct output as bytes representation?
msg388460 - (view) Author: David Wood (dwoodjunkmail) Date: 2021-03-10 21:03
Christian/Eric -

Thank you both so much for taking time on this.  Christian had pointed out the use of memcpy vs strncpy.  It turns out that while strncpy is designed to copy a specific number of chars, it turns out that it stops on null.  I did make the change Christian suggested however, it turns out that strncpy was also used in the decrypt function which was also subject to the same problem.
msg388461 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-10 21:08
Don't feel bad about it. Nintendo made a very similar mistake. The trucha bug made it trivial to bypass DRM of Wii games.
Date User Action Args
2022-04-11 14:59:42adminsetgithub: 87601
2021-03-11 13:27:38dwoodjunkmailsetfiles: - crypt.tar.gz
2021-03-10 21:08:19christian.heimessetmessages: + msg388461
2021-03-10 21:03:08dwoodjunkmailsetstatus: open -> closed
resolution: not a bug
messages: + msg388460

stage: resolved
2021-03-10 08:46:38christian.heimessetmessages: + msg388411
2021-03-10 01:47:48dwoodjunkmailsetfiles: - crypt.tar.gz
2021-03-10 01:39:46dwoodjunkmailsetfiles: + crypt.tar.gz
2021-03-10 01:37:50dwoodjunkmailsetfiles: + crypt.tar.gz

messages: + msg388402
2021-03-09 16:07:53christian.heimessetmessages: + msg388369
2021-03-09 15:52:34eric.smithsetnosy: + eric.smith
messages: + msg388367
2021-03-08 21:52:40dwoodjunkmailsetmessages: + msg388310
2021-03-08 21:43:56christian.heimessetmessages: + msg388308
2021-03-08 21:31:52dwoodjunkmailsetmessages: + msg388306
2021-03-08 17:47:04christian.heimessetmessages: + msg388273
2021-03-08 16:00:00dwoodjunkmailsetmessages: + msg388270
2021-03-08 15:37:40christian.heimessetnosy: + christian.heimes
messages: + msg388269
2021-03-08 15:31:33dwoodjunkmailcreate