Using tokenize.py to tokenize the attached file yields:
0,0-0,0: ENCODING 'utf-8'
1,0-1,2: NAME 'if'
1,3-1,4: NAME 'a'
1,4-1,5: OP ':'
1,5-1,7: NEWLINE '\r\n'
2,0-2,4: INDENT ' '
2,4-2,5: NAME 'b'
2,6-2,7: OP '='
2,8-2,9: NUMBER '1'
2,9-2,11: NEWLINE '\r\n'
3,0-3,2: NL '\r\n'
4,0-4,6: COMMENT '# test'
4,6-4,6: NL ''
4,6-4,7: NEWLINE ''
5,0-5,0: DEDENT ''
5,0-5,0: ENDMARKER ''

This output is wrong in that it adds 2 newlines one as a NL which is a correct and one as a NEWLINE which is not since there is no preceding code.

If a new line is added at the end of the file, one gets:

0,0-0,0: ENCODING 'utf-8'
1,0-1,2: NAME 'if'
1,3-1,4: NAME 'a'
1,4-1,5: OP ':'
1,5-1,7: NEWLINE '\r\n'
2,0-2,4: INDENT ' '
2,4-2,5: NAME 'b'
2,6-2,7: OP '='
2,8-2,9: NUMBER '1'
2,9-2,11: NEWLINE '\r\n'
3,0-3,2: NL '\r\n'
4,0-4,6: COMMENT '# test'
4,6-4,8: NL '\r\n'
5,0-5,0: DEDENT ''
5,0-5,0: ENDMARKER ''

Similarly if code is added before the comment, a single NEWLINE is generated (with no text since it is fake).

The extra NEWLINE found when tokenizing the attached file can cause issue when parsing the file. It was found in we-like-parsers/pegen#11 (comment) where a pure python parser based on pegen is being built. The extra NEWLINE is an issue since the grammar does not accept NEWLINE at the end of a block and cause parsing to fail using the same rules as the python grammar while the cpython parser can handle this file without any issue.

I believe this issue stems from https://github.com/python/cpython/blob/3.9/Lib/tokenize.py#L605 where the check does not account for a last line limited to comments. Adding a check to determine if the line starts with a # should be sufficient to avoid emitting the extra NEWLINE.

New changeset b6bde9f by Pablo Galindo Salgado in branch 'main':
bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499)
b6bde9f

New changeset 33a4010 by Miss Islington (bot) in branch '3.10':
bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499) (GH-27500)
33a4010

New changeset 2d11797 by Miss Islington (bot) in branch '3.9':
bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499) (GH-27501)
2d11797

This appears to have been a duplicate of bpo-35107, where the failing example was '#' and it was NL, NEWLINE pair was noted. So this either predates 3.9 or was re-introduced. In any case, thanks for the fix.