Tokens needs to be a generator, not a list.

Tokens needs to be a generator, not a list.

At first I thought they were pretty much the same thing.
But generators pick up from where they left off,
while lists don't.

This fixes the edge-case infinite loop
that was happening last commit.

from the python repl

Lists
>>> data = [1,2,3,4,5]
>>> for i in data:
...     print(i)
...     if i == 2:
...         for ii in data:
...             print('>', ii)
...             if ii == 4:
...                 break
...
1
2
> 1
> 2
> 3
> 4
3
4
5

Generators
>>> data = [1,2,3,4,5]
>>> gen = (d for d in data)
>>> for i in gen:
...     print(i)
...     if i == 2:
...         for ii in gen:
...             print('>', ii)
...             if ii == 4:
...                 break
...
1
2
> 3
> 4
5