`shlex.push_source()` fails to reset lexer state, causing new stream to be ignored if called after EOF

[T90REAL]

Bug report

Bug description:

Calling shlex.push_source() after the current input has reached EOF leaves the lexer in state == None, so the newly pushed source is never read. Subsequent get_token() calls immediately return EOF instead of consuming the new input.

from shlex import shlex

lexer = shlex('a')
print(lexer.get_token())
print(lexer.get_token())  # EOF -> state is None

lexer.push_source('b', None)
print(lexer.get_token())  # actural '' (EOF), expected: 'b'

a

CPython versions tested on:

3.12

Operating systems tested on:

Linux

[zangjiucheng]

Hello :)

I was able to reproduce this behaviour and can confirm the issue described.
The root cause is that after reading to EOF, get_token() sets self.state = None.
When a new source is pushed afterward, the lexer state is never reset, so the next
call to get_token() still returns EOF instead of starting to read from the new stream.

This behaviour seems inconsistent with the documentation shlex objects,
which states that a pushed source should be read immediately before returning
to the original stream.

I suggest the following change to push_source() to align with the documented semantics.
By resetting state and clearing the pushback buffers, the lexer will start reading
from the new source right away:

    def push_source(self, newstream, newfile=None):
        "Push an input source onto the lexer's input source stack."
        if isinstance(newstream, str):
            newstream = StringIO(newstream)
        self.filestack.appendleft((self.infile, self.instream, self.lineno))
        self.infile = newfile
        self.instream = newstream
        self.lineno = 1
+       self.state = ' '
+       self.token = ''
+       self.pushback.clear()
+       if hasattr(self, "_pushback_chars"):
+           self._pushback_chars.clear()
        if self.debug:
            if newfile is not None:
                print('shlex: pushing to file %s' % (self.infile,))
            else:
                print('shlex: pushing to stream %s' % (self.instream,))

This change maintains backward compatibility while fixing the post-EOF push behavior.
After applying it, push_source() correctly switches to the new source immediately,
matching the documented behavior.

[zangjiucheng]

Yes, one more sentence here.

To make the solution to this problem more robust, we need to store more useful variables on the stack, but I am not sure if this is worthwhile.

self.filestack.appendleft(
    (self.infile, self.instream, self.lineno, self.state,
     list(self.pushback), list(self._pushback_chars), self.token)
)