Add `prism` parser support #387

viralpraxis · 2025-04-10T15:24:44Z

Resolves #385

Related Prism bug: ruby/prism#3540 (merged)

test/corpus/literal/before/34.rb contains expressions rejected by Prism (and regexp-related bug, see #385 (comment)):

$ ASDF_RUBY_VERSION=3.4.2 ruby --parser prism test/corpus/literal/before/34.rb
test/corpus/literal/before/34.rb:1: syntax errors found (SyntaxError)
>  1 | retry
     | ^~~~~ Invalid retry without rescue
>  2 | in {"#{"a"}": 1} then
     | ^~ unexpected 'in', ignoring it
     |                  ^~~~ unexpected 'then', ignoring it
     |                  ^~~~ unexpected 'then', expecting end-of-input
   3 |   true
   4 | /\c*a/
   5 | /\c*a\c*/
   6 | /\c*\c*\c*/
>  7 | (break foo) || a
     |  ^~~~~~~~~ Invalid break
     |  ^~~~~~~~~ unexpected void value expression
>  8 | (return foo) || a
     |  ^~~~~~~~~~ unexpected void value expression
>  9 | a = b || break
     |          ^~~~~ Invalid break
> 10 | a = b || next
     |          ^~~~ Invalid next
> 11 | a || (break foo)
     |       ^~~~~~~~~ Invalid break
> 12 | b or break
     |      ^~~~~ Invalid break
> 13 | b or next
     |      ^~~~ Invalid next
> 14 | break or b
     | ^~~~~ Invalid break
     | ^~~~~ unexpected void value expression
> 15 | next or b
     | ^~~~ Invalid next
     | ^~~~ unexpected void value expression
> 16 | return or a
     | ^~~~~~ unexpected void value expression

The prism gem is temporary pointed to the main since Prism::Translation::ParserCurrent is not yet released. Let me know if we want to support prism 1.4.

mbj · 2025-04-16T22:56:35Z

@viralpraxis ^^ there are some CI errors, you have energy to look into them?

viralpraxis · 2025-04-17T05:13:18Z

Yeah, these were due to invalid YAML array with ruby versions 🤦🏻 Lets try again

mbj · 2025-04-17T22:14:43Z

Gemfile


 source 'https://rubygems.org'

+gem 'prism', github: 'ruby/prism'


@viralpraxis JFYI: I cannot merge / release this until we have a prism release that contains the fix for the bug you found.

@viralpraxis Once we have a prism release with the fix, unparser needs to depend on that prism version. So once we have a release of prism with the fix: Can you add a dependency to prism in the unparser.gemfile ?

Its fine for me to depend on prism even if we do not use it for the ruby < 3.4 cases. And afaik we cannot do "ruby version dependent dependencies".

Alright, but just out of curiosity: can't we just depend on prism >= 1.4 and temporary ignore this (extremely rare) edge case?

@viralpraxis I do not like "knowingly" releasing bugs. On the other hand: 3.4 is bugged "already" in the latest release. So "ignoring" the bug in prism is still better than current state of affairs where parser is more bugged on 3.4.

Maybe we can just ask prism devs kindly to do a release? It will allow go add unparser as supported library to prism, and there are lots of code gen cases for unparser.

Lets ask kindly for a release first? If not possible I'm happy to release an unparser on current state.

@mbj

Can you add a dependency to prism in the unparser.gemfile

unparser.gemspec, right?

Maybe we can just ask prism devs kindly to do a release? It will allow go add unparser as supported library to prism, and there are lots of code gen cases for unparser. Lets ask kindly for a release first? If not possible I'm happy to release an unparser on current state.

Yeah, I think it makes sense. But I also think it makes even more sense coming from you -- since you maintain this (and mutant) gem, your request has a much better chance of succeeding :-)

Alright, but just out of curiosity: can't we just depend on prism >= 1.4 and temporary ignore this (extremely rare) edge case?

I actually missed that we currently also depend on Prism::Translation::ParserCurrent API which is also unreleased yet (see this PR). So depending on prism >= 1.4 is not an option

--

By the way, it looks like unparser can't be installed on MRI 3.1:

$ ASDF_RUBY_VERSION=3.1.6 bundle Fetching gem metadata from https://rubygems.org/........... Resolving dependencies... Could not find compatible versions Because mutant >= 0.13.0 depends on Ruby >= 3.2 and Gemfile depends on mutant ~> 0.13.0, Ruby >= 3.2 is required. So, because current Ruby version is = 3.1.6, version solving has failed.

perhaps required_ruby_version should be set to >= 3.2?

@viralpraxis Yes, I want to drop 3.1 support (also in mutant).

But I also think it makes even more sense coming from you -- since you maintain this (and mutant) gem, your request has a much better chance of succeeding :-)

@viralpraxis See: ruby/prism#3553.

mbj · 2025-04-17T22:23:30Z

@viralpraxis Only merge blocking comment here: https://github.com/mbj/unparser/pull/387/files#r2049710390

Feel free to bump version and add a changelog crediting yourself. I'll release immediately after ^^ is resolved.

viralpraxis · 2025-04-22T09:32:07Z

I updated this PR so it should now work with Prism (1.4) so you can merge it if Prism does not release soon:

Prism::Translation::ParserCurrent is changed to Prism::Translation::Parser34
Prism-failing tests are moved to test/corpus/literal/before/34.rb

viralpraxis · 2025-04-22T20:58:25Z

I'm still unable to trigger CI even after merging #388 (weird), @mbj could you please trigger it?

viralpraxis · 2025-04-22T21:36:54Z

Alright, no idea why it still fails. Gonna look into it later

viralpraxis · 2025-04-23T11:53:50Z

@mbj It seems like I was too optimistic (and naive) about this PR. Round trip tests do not work with Prism since it uses different testing engine (Test::Unit instead of minitest). I'll try to make it working, meanwhile we can wait for the Prism's team response

mbj · 2025-04-23T12:09:38Z

@mbj It seems like I was too optimistic (and naive) about this PR. Round trip tests do not work with Prism since it uses different testing engine (Test::Unit instead of minitest). I'll try to make it working, meanwhile we can wait for the Prism's team response

Hehe, sorry I've not yet dug deep into your code and was happy to see CI green. It makes sense that unparser has to hook the prism test suite, just like unparser used to hook parser test suite.

Writing unparsers that actually round trip significant bodies is not an easy feat, ty for helping.

Extracted from mbj#387 ``` $ bundle exec bin/unparser -e 'foo = 1, 2 rescue nil' warning: parser/current is loading parser/ruby34, which recognizes 3.4.0-dev-compliant syntax, but you are running 3.4.3. Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri. (string) Original-Source: foo = 1, 2 rescue nil Generated-Source: foo = [1, 2] rescue nil Original-Node: (rescue (lvasgn :foo (array (int 1) (int 2))) (resbody nil nil (nil)) nil) Generated-Node: (lvasgn :foo (rescue (array (int 1) (int 2)) (resbody nil nil (nil)) nil)) Node-Diff: @@ -1,7 +1,7 @@ -(rescue - (lvasgn :foo +(lvasgn :foo + (rescue (array (int 1) - (int 2))) - (resbody nil nil - (nil)) nil) + (int 2)) + (resbody nil nil + (nil)) nil)) Error: (string) ```

Extracted from mbj#387

Extracted from mbj#387 This one is tricky: looks like we have to always (as a general rule) append comma to mhls and omit them when they are inside `def` node (as an exception).

Resolves mbj#385 Related Prism bug: ruby/prism#3540 (merged) `test/corpus/literal/before/34.rb` contains expressions rejected by Prism (and regexp-related bug, see mbj#385 (comment)): ```bash $ ASDF_RUBY_VERSION=3.4.2 ruby --parser prism test/corpus/literal/before/34.rb test/corpus/literal/before/34.rb:1: syntax errors found (SyntaxError) > 1 | retry | ^~~~~ Invalid retry without rescue > 2 | in {"#{"a"}": 1} then | ^~ unexpected 'in', ignoring it | ^~~~ unexpected 'then', ignoring it | ^~~~ unexpected 'then', expecting end-of-input 3 | true 4 | /\c*a/ 5 | /\c*a\c*/ 6 | /\c*\c*\c*/ > 7 | (break foo) || a | ^~~~~~~~~ Invalid break | ^~~~~~~~~ unexpected void value expression > 8 | (return foo) || a | ^~~~~~~~~~ unexpected void value expression > 9 | a = b || break | ^~~~~ Invalid break > 10 | a = b || next | ^~~~ Invalid next > 11 | a || (break foo) | ^~~~~~~~~ Invalid break > 12 | b or break | ^~~~~ Invalid break > 13 | b or next | ^~~~ Invalid next > 14 | break or b | ^~~~~ Invalid break | ^~~~~ unexpected void value expression > 15 | next or b | ^~~~ Invalid next | ^~~~ unexpected void value expression > 16 | return or a | ^~~~~~ unexpected void value expression ```

viralpraxis · 2025-05-15T22:35:21Z

@mbj Alright, looks like I've fixed all the (visible) bugs and we now round-trip the entire prism corpus 🙂 (as we discussed earlier, some heredoc-related counterexamples are temporary excluded)

Extracted from #387 Without this change, the following code snippets ```ruby a or b and c ``` and ```ruby a[:x] = b[:x] || c[:x] || d(:new) ``` do not round trip.

Extracted from #387 Without this change, the following code snippet ```ruby class A class << self rescue else ensure end end ``` leads to ```bash ``` error during RT

Extracted from #387 Parser represents %-encoded array patterns with the `ArrayNode`, not `ArrayPatternNode`. We can infer required %-notation variant given array's children elements. ```bash $ bundle exec bin/unparser -e 'a => %i[a b]' (string) Original-Source: a => %i[a b] Generated-Source: a => [:a, :b] Original-Node: (match-pattern (send nil :a) (array (sym :a) (sym :b))) Generated-Node: (match-pattern (send nil :a) (array-pattern (sym :a) (sym :b))) Node-Diff: @@ -1,5 +1,5 @@ (match-pattern (send nil :a) - (array + (array-pattern (sym :a) (sym :b))) Error: (string) ```

Extracted from #387

Extracted from #387 ```bash bundle exec bin/unparser -e 'not a...b' (string) Original-Source: not a...b Generated-Source: !a...b Original-Node: (send (eflipflop (send nil :a) (send nil :b)) :!) Generated-Node: (erange (send (send nil :a) :!) (send nil :b)) Node-Diff: @@ -1,4 +1,4 @@ -(send - (eflipflop - (send nil :a) - (send nil :b)) :!) +(erange + (send + (send nil :a) :!) + (send nil :b)) ```

Extracted from #387

Extracted from #387 This solution aint perfect (there are still some bugs in this area), but at least it fixes some of them. I'll try to find some time to revisit it later.

Extracted from Extracted from #387 ```bash $ bundle exec bin/unparser -e 'module A; x = 1; rescue; end' warning: parser/current is loading parser/ruby34, which recognizes 3.4.0-dev-compliant syntax, but you are running 3.4.3. Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri. (string) Original-Source: module A; x = 1; rescue; end Generated-Source: module A x = 1 rescue end Original-Node: (module (const nil :A) (rescue (lvasgn :x (int 1)) (resbody nil nil nil) nil)) Generated-Node: ```

Extracted from #387

Extracted from #387 ``` $ bundle exec bin/unparser -e 'foo = 1, 2 rescue nil' warning: parser/current is loading parser/ruby34, which recognizes 3.4.0-dev-compliant syntax, but you are running 3.4.3. Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri. (string) Original-Source: foo = 1, 2 rescue nil Generated-Source: foo = [1, 2] rescue nil Original-Node: (rescue (lvasgn :foo (array (int 1) (int 2))) (resbody nil nil (nil)) nil) Generated-Node: (lvasgn :foo (rescue (array (int 1) (int 2)) (resbody nil nil (nil)) nil)) Node-Diff: @@ -1,7 +1,7 @@ -(rescue - (lvasgn :foo +(lvasgn :foo + (rescue (array (int 1) - (int 2))) - (resbody nil nil - (nil)) nil) + (int 2)) + (resbody nil nil + (nil)) nil)) Error: (string) ```

Follow-up to mbj#387 ```shell bundle exec bin/unparser -e '`#$G`' (string) Original-Source: `#$G` Generated-Source: lib/unparser/emitter/xstr.rb:56:in 'Unparser::Emitter::XStr#escape_xstr' lib/unparser/emitter/xstr.rb:52:in 'Unparser::Emitter::XStr#emit_string' lib/unparser/emitter/xstr.rb:45:in 'block in Unparser::Emitter::XStr#emit_xstr' lib/unparser/emitter/xstr.rb:41:in 'Array#each' lib/unparser/emitter/xstr.rb:41:in 'Unparser::Emitter::XStr#emit_xstr' lib/unparser/emitter/xstr.rb:16:in 'Unparser::Emitter::XStr#dispatch' lib/unparser/generation.rb:13:in 'block in Unparser::Generation#write_to_buffer' lib/unparser/generation.rb:44:in 'Unparser::Generation#with_comments' lib/unparser/generation.rb:13:in 'Unparser::Generation#write_to_buffer' <internal:kernel>:91:in 'Kernel#tap' lib/unparser/generation.rb:236:in 'Unparser::Generation#visit_deep' lib/unparser/generation.rb:132:in 'block in Unparser::Generation#emit_body' lib/unparser/generation.rb:138:in 'Unparser::Generation#with_indent' lib/unparser/generation.rb:122:in 'Unparser::Generation#emit_body' lib/unparser/emitter/root.rb:11:in 'Unparser::Emitter::Root#dispatch' lib/unparser/generation.rb:13:in 'block in Unparser::Generation#write_to_buffer' lib/unparser/generation.rb:44:in 'Unparser::Generation#with_comments' lib/unparser/generation.rb:13:in 'Unparser::Generation#write_to_buffer' lib/unparser.rb:148:in 'block in Unparser.unparse_ast' <internal:kernel>:91:in 'Kernel#tap' Original-Node: (xstr (gvar :$G)) Generated-Node: undefined Error: (string) ```

viralpraxis mentioned this pull request Apr 10, 2025

Using prism translation instead of parser/current at 3.4 #385

Closed

viralpraxis force-pushed the add-prism-support branch 4 times, most recently from 58478d7 to 3665cc3 Compare April 16, 2025 20:45

viralpraxis force-pushed the add-prism-support branch from 3665cc3 to e2ac5e0 Compare April 17, 2025 05:11

viralpraxis marked this pull request as ready for review April 17, 2025 05:13

mbj reviewed Apr 17, 2025

View reviewed changes

viralpraxis force-pushed the add-prism-support branch from e2ac5e0 to a263dae Compare April 18, 2025 10:20

This was referenced Apr 18, 2025

Requesting new release ruby/prism#3553

Closed

Modifying AST and generating source code ruby/prism#3466

Closed

viralpraxis force-pushed the add-prism-support branch from a263dae to aaa64ad Compare April 22, 2025 09:21

viralpraxis mentioned this pull request Apr 22, 2025

Update development dependencies #388

Merged

viralpraxis force-pushed the add-prism-support branch from aaa64ad to 231d9a7 Compare April 22, 2025 20:55

viralpraxis force-pushed the add-prism-support branch from 040c083 to 231d9a7 Compare April 22, 2025 21:32

viralpraxis force-pushed the add-prism-support branch 4 times, most recently from e7aec57 to 95e9a04 Compare April 22, 2025 22:11

viralpraxis force-pushed the add-prism-support branch 3 times, most recently from 60533f6 to ce586a7 Compare April 23, 2025 22:46

viralpraxis mentioned this pull request May 14, 2025

Fix mass assign and rescue precedence during unparsing #404

Merged

viralpraxis force-pushed the add-prism-support branch from fc57a20 to 40b7eb8 Compare May 15, 2025 13:34

viralpraxis added a commit to viralpraxis/unparser that referenced this pull request May 15, 2025

Fix masgn with nested mlhs unparsing

9415e90

Extracted from mbj#387

viralpraxis mentioned this pull request May 15, 2025

Fix masgn with nested mlhs unparsing #405

Merged

viralpraxis force-pushed the add-prism-support branch from 40b7eb8 to 6f1b443 Compare May 15, 2025 22:31

viralpraxis force-pushed the add-prism-support branch from 6f1b443 to 1e4963e Compare May 15, 2025 22:32

viralpraxis requested a review from mbj May 15, 2025 22:35

mbj pushed a commit that referenced this pull request May 25, 2025

Fix and and or operators RT broken associativity

c0ad489

Extracted from #387 Without this change, the following code snippets ```ruby a or b and c ``` and ```ruby a[:x] = b[:x] || c[:x] || d(:new) ``` do not round trip.

mbj pushed a commit that referenced this pull request May 25, 2025

Fix rescue in class << self unparsing

f630c57

Extracted from #387 Without this change, the following code snippet ```ruby class A class << self rescue else ensure end end ``` leads to ```bash ``` error during RT

mbj pushed a commit that referenced this pull request May 25, 2025

Fix resbody with multiple indexes assignment unparsing

27a3357

Extracted from #387

mbj pushed a commit that referenced this pull request May 25, 2025

Fix rescue without body unparsing

0ce30ae

Extracted from #387

mbj pushed a commit that referenced this pull request May 25, 2025

Fix irange/erange edgecases unparsing

d926a28

Extracted from #387

mbj pushed a commit that referenced this pull request May 25, 2025

Fix not with binary operator unparsing

a8dab26

Extracted from #387 This solution aint perfect (there are still some bugs in this area), but at least it fixes some of them. I'll try to find some time to revisit it later.

mbj pushed a commit that referenced this pull request May 25, 2025

Fix irange/erange with missing rhs unparsing

f9ba255

Extracted from #387

mbj merged commit b4d2ae5 into mbj:main May 25, 2025
16 checks passed

viralpraxis mentioned this pull request May 29, 2025

Fix xstr with plain gvar unparsing #409

Merged

viralpraxis deleted the add-prism-support branch May 29, 2025 07:11


		source 'https://rubygems.org'

		gem 'prism', github: 'ruby/prism'

Uh oh!

Add prism parser support #387

Add prism parser support #387

Uh oh!

Conversation

viralpraxis commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mbj commented Apr 16, 2025

Uh oh!

viralpraxis commented Apr 17, 2025

Uh oh!

mbj Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

mbj Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

viralpraxis Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

mbj Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

viralpraxis Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbj Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

mbj Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

mbj commented Apr 17, 2025

Uh oh!

viralpraxis commented Apr 22, 2025

Uh oh!

viralpraxis commented Apr 22, 2025

Uh oh!

viralpraxis commented Apr 22, 2025

Uh oh!

viralpraxis commented Apr 23, 2025

Uh oh!

mbj commented Apr 23, 2025

Uh oh!

viralpraxis commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add `prism` parser support #387

Add `prism` parser support #387

viralpraxis commented Apr 10, 2025 •

edited

Loading

viralpraxis Apr 18, 2025 •

edited

Loading

viralpraxis commented May 15, 2025 •

edited

Loading