Unfortunately this might be tricky to do correctly as the implementation from d3ce840 doesn't consider that for and if can be interleaved! I'll go ahead and finish rebasing this but then we'll have another bug to fix. I still think the only good solution here is to add an AST node that encapsulates the for … in … [if …] instead of trying to put it all into GeneratorExpr.

Why does mypy still use its own parser anyhow? Couldn't the Python 3 parser be used and just transform the ast to something that's suitable for mypy?

Rebased. I worked around the if thing by using nested lists but haven't written tests for interleaved for/if yet. I'll do that separately from this.

Looks good! Thanks for the implementation!

And about the parser: that's just a legacy thing. The parser was originally implemented in Alore, a programming language that I created. Alore had no tools for writing parsers, so I wrote the thing from scratch.

If somebody wants to replace parser, I'm totally fine with it... it's currently pretty klunky. However, it would be good to preserve the ability to pretty print the original source code, including whitespace and comments.

I think it would be pretty difficult to reconstruct comments from the ast module for pretty printing, but maybe a good compromise would be to use the tokenize module and write the parser/unparser in terms of tokens?

Tokenize also throws away comments.

There's a parser for Python syntax in the stdlib though, as part of lib2to3.

On Wed, Jul 30, 2014 at 2:10 PM, Bob Ippolito [email protected]
wrote:

I think it would be pretty difficult to reconstruct comments from the ast
module for pretty printing, but maybe a good compromise would be to use the
tokenize module and write the parser/unparser in terms of tokens?

—
Reply to this email directly or view it on GitHub
#324 (comment).

--Guido van Rossum (python.org/~guido)

Try using various forms of continuation line syntax.

On Wed, Jul 30, 2014 at 2:37 PM, Bob Ippolito [email protected]
wrote:

$ printf '# hello world\ndef x(): pass' | python3 -mtokenize1,0-1,13: COMMENT '# hello world'1,13-1,14: NL '\n' 2,0-2,3: NAME 'def' 2,4-2,5: NAME 'x' 2,5-2,6: OP '(' 2,6-2,7: OP ')' 2,7-2,8: OP ':' 2,9-2,13: NAME 'pass' 3,0-3,0: ENDMARKER ''

—
Reply to this email directly or view it on GitHub
#324 (comment).

--Guido van Rossum (python.org/~guido)

I did some profiling and it seems that lexing parsing take approx. ~40% of the total time needed for type checking. Speeding up the parser could give a useful performance boost.

However, there are probably more effective ways of speeding up type checking, such as caching the type checked interfaces of modules (e.g., via JSON serialization) and reusing the cached interface description when the module and its dependencies have not changed since the previous type checking run.

Also, the above also means that any rewrite of the parser should not make it slower.