New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RotatingFileHandler rollover doesn't work on QNX shmem filesystems #60653
Comments
logging.handlers.RotatingFileHandler.doRollover fails on QNX /dev/shmem filesystems (seen on a 6.4.0-based system). QNX RAM filesystems don't support rename() (see http://www.qnx.com/developers/docs/6.4.0/neutrino/sys_arch/fsys.html#DEVSHMEM - it's a 'big filesystem' feature). So for example, rename("/dev/shmem/foo", "/dev/shmem/bar") fails with EXDEV. This is easily fixed by using shutils.move rather than os.rename where appropriate, falling back to copying if a rename() fails. It's not sufficient to set the rotator attribute, since there are other os.rename calls in in doRollover. |
I'm not sure why you would setup logging on a RAM filesystem (I assume we're talking about normal volatile RAM)? |
I agree with Antoine. It seems to me that it is very important to the semantics of rollover that the rename be atomic, even if we ignore the issue of existing other readers. If it were not atomic, you might end up with lost log messages. So I don't think there is anything to do here: you just can't use rollover on a filesystem that doesn't support atomic rename. I'll leave it Vinay to close the issue if he agrees with us. |
Thanks for the patch, but I'm closing this as 'wontfix', as per the points made by Antoine and David. If you need logging from an embedded system, please consider using one of the socket-based logging handlers, if that's feasible in the specific situation. |
What about the "rotator" attribute. |
I'm not convinced that it matters whether the rename or move is atomic. Can anyone come up with a quick concrete example? I see two scenarios:
While the first case isn't ideal, I don't think there can be any loss of logs. |
Serhiy, there are also calls to os.rename in RotatingFileHandler.doRollover |
The current implementation was written with an expectation of working rename functionality in the stdlib. As such, while this issue might be categorised as being of type "enhancement", I don't see how you can categorise it as being of type "behaviour". What's the problem with subclassing the relevant handlers to implement your own doRollover() method? You only need to do that once for all the projects in your QNX environment. Given that it's not a mainstream environment, that's not IMO an unreasonable suggestion. If you publish such a handler, I'll mention it on http://plumberjack.blogspot.co.uk/p/handlers-for-logging.html which has a whole bunch of handlers written by people for specific requirements which don't warrant adding functionality directly in the stdlib. |
I've updated the type to enhancement (it seems like a grey area to me - it's a behavioural fix for a niche use case). I suggested a patch rather than simply subclassing RotatingFileHandler since:
Thanks for the offer of having an extension linked from your blog. I don't think it's appropriate for this case (since it's just a slightly modified copy of the stdlib code - I'll probably just keep the patch along with a few other compatibility hacks we have) although I may have something for you in future (we do have a subclass that compresses old logs). |
You may just as well monkeypatch os.rename() to fallback on From a code quality and readability standpoint, os.rename() conveys the |
Doesn't the "rotator" attribute break atomicity? A careful rotator should first rename the source to the temporary file, process the data and save it to other temporary file, and then rename the result to the destination and remove the first temporary file. |
Which rotator do you mean? The default rotator is None, which leads to os.rename being called. If you're referring to the example in the documentation (cookbook) - it was intended purely as an example, and the paragraph under the snippet does say it's only intended for illustrative purposes. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: