Using prism translation instead of `parser/current` at 3.4

[viralpraxis]

parser is no compatible with Ruby > 3.4:

The parser gem is only compatible with the syntax of Ruby 3.3 and lower. For Ruby 3.4 and later, please use the Prism::Translation::Parser instead. Starting in Ruby 3.4, Prism is the parser used in Ruby itself and can produce AST that is identical to the output of the parser gem. If you only need to parse Ruby 3.3 (or greater) and don't require compatibility with the parser gem AST, also consider using the native Prism AST. See this whitequark/parser#1046 for more details.

(https://github.com/whitequark/parser?tab=readme-ov-file#parser)

I wonder if the gem should use Prism::Translation::Parser instead?

[mbj]

I wonder if the gem should use Prism::Translation::Parser instead?

Just like rubocop: Unparser needs to work with "older" ruby versions that do not yet have prism support. This means unparser can use Prism::Translation::Parser but it'll be behind a branch that keeps using parser for the ruby versions that do not yet have prism support.

[viralpraxis]

Just like rubocop: Unparser needs to work with "older" ruby versions that do not yet have prism support

Yeah, I meant to add optional support for Prism (i.e. if RUBY_VERSION >= 3.4)

This means unparser can use Prism::Translation::Parser but it'll be behind a branch that keeps using parser for the ruby versions that do not yet have prism support.

Are there any plans on implementing this? I can take a look at this by myself

[mbj]

PRs are welcome ;)

[mbj]

I'll "eventually" get to it, but OSS time is limited.

[viralpraxis]

I took a brief look and ATM there are 36 failing unit tests, most of them due to AST mismatch between Prism (1.4) and parser/current, e.g. for foo[index] += 2 expression:

1. prism

=> s(:op_asgn,
  s(:send,
    s(:send, nil, :foo), :[],
    s(:send, nil, :index)), :-,
  s(:int, 2))

2. whitequark parser

  => s(:op_asgn,
  s(:indexasgn,
    s(:send, nil, :foo)), :-,
  s(:int, 2))

[viralpraxis]

Nevermind, that was due to a misconfiguration. There are still some problems, but it might be due to a bug in the prism itself.

[mbj]

Yeah modern AST format only. Feel free to push what you have so I can have a look ;) you may have sufficiently nerd sniped me :P

[viralpraxis]

I've opened #387 @mbj, feel free to take a look :P

[mbj]

@viralpraxis Travelling currently so I cannot do a deep dive, but it indeed appears that the translation has a bug at this point.

[mbj]

Also OT: What are you using unparser for?

[viralpraxis]

Travelling currently so I cannot do a deep dive, but it indeed appears that the translation has a bug at this point.

Yeah, I've already opened a PR at prism's side

Also OT: What are you using unparser for?

I (we) don't use unparser directly -- it's a transitive dependency. We're planning to migrate to MRI 3.4 so I volunteered to make sure all the relevant gems are capable of using prism

[mbj]

I (we) don't use unparser directly -- it's a transitive dependency. We're planning to migrate to MRI 3.4 so I volunteered to make sure all the relevant gems are capable of using prism

nice, may I ask what direct dependency you are using?

[viralpraxis]

nice, may I ask what direct dependency you are using?

We're using rspec-parameterized, which depends on rspec-parameterized_core, which depends on proc_to_ast, which depends on unparser :-)

[viralpraxis]

Here are 4 remaining bugs:

1

Original-Node:
(begin
  (regexp
    (regopt))
  (regexp
    (str "foo")
    (regopt))
  (regexp
    (begin
      (send nil :foo))
    (regopt))
  (regexp
    (begin
      (dstr
        (str "line_1\n")
        (str "line_2\n")
        (str "line_3\n")
        (str "line_4\n")
        (str "line_5\n")
        (str "line_6\n")
        (str "line_7\n")
        (str "line_8\n")))
    (regopt))
  (regexp
    (str "[^-+',./:@[:alnum:]\\[\\]]+")
    (regopt))
  (regexp
    (str "foo")
    (regopt))
  (regexp
    (str "foo")
    (begin
      (ivar :@bar))
    (regopt))
  (regexp
    (str "foo")
    (begin
      (ivar :@bar))
    (regopt :i :m :x))
  (regexp
    (begin
      (str "\u0000"))
    (regopt))
  (regexp
    (str "\\n")
    (regopt))
  (regexp
    (str "\\n")
    (regopt))
  (regexp
    (str "\\n")
    (regopt :x))
  (regexp
    (str "//")
    (regopt :x))
  (regexp
    (str "\n")
    (regopt))
  (regexp
    (str "\\x0Aa")
    (regopt))
  (regexp
    (str "a\n")
    (regopt))
  (regexp
    (str "\\x0Aa\\x0A")
    (regopt))
  (regexp
    (str "\\x0A\\x0A\\x0A")
    (regopt))
  (regexp
    (str "\n")
    (str "a\n")
    (regopt)))
Generated-Node:
(begin
  (regexp
    (regopt))
  (regexp
    (str "foo")
    (regopt))
  (regexp
    (begin
      (send nil :foo))
    (regopt))
  (regexp
    (begin
      (dstr
        (str "line_1\n")
        (str "line_2\n")
        (str "line_3\n")
        (str "line_4\n")
        (str "line_5\n")
        (str "line_6\n")
        (str "line_7\n")
        (str "line_8\n")))
    (regopt))
  (regexp
    (str "[^-+',./:@[:alnum:]\\[\\]]+")
    (regopt))
  (regexp
    (str "foo")
    (regopt))
  (regexp
    (str "foo")
    (begin
      (ivar :@bar))
    (regopt))
  (regexp
    (str "foo")
    (begin
      (ivar :@bar))
    (regopt :i :m :x))
  (regexp
    (begin
      (str "\u0000"))
    (regopt))
  (regexp
    (str "\\n")
    (regopt))
  (regexp
    (str "\\n")
    (regopt))
  (regexp
    (str "\\n")
    (regopt :x))
  (regexp
    (str "//")
    (regopt :x))
  (regexp
    (str "\n")
    (regopt))
  (regexp
    (str "\\x0Aa")
    (regopt))
  (regexp
    (str "a\n")
    (regopt))
  (regexp
    (str "\\x0Aa\\x0A")
    (regopt))
  (regexp
    (str "\\x0A\\x0A\\x0A")
    (regopt))
  (regexp
    (str "\n")
    (str "a\n")
    (regopt)))
Source-Diff:
@@ -1,33 +1,33 @@
 //
 /foo/
 /#{foo}/
 /#{<<-HEREDOC}/
 line_1
 line_2
 line_3
 line_4
 line_5
 line_6
 line_7
 line_8
 HEREDOC
 %r{[^-+',./:@[:alnum:]\[\]]+}
 /foo/
 /foo#{@bar}/
 /foo#{@bar}/imx
 /#{"\u0000"}/
 /\n/
 /\n/
 /\n/x
 %r{//}x
 /
 /
-/\c*a/
+/\x0Aa/
 /a
 /
-/\c*a\c*/
-/\c*\c*\c*/
+/\x0Aa\x0A/
+/\x0A\x0A\x0A/
 /
 a
 /

2

test/corpus/literal/control.rb
Original-Source:
next
return
break
retry
redo
return 1
return 1, 2
return true ? 1 : 2
break true ? 1 : 2
next true ? 1 : 2
return true, if true
  1
else
  2
end

Generated-Source:
undefined
Original-Node:
#<Parser::SyntaxError: Invalid retry without rescue>

3

test/corpus/literal/binary.rb
Original-Source:
(a || b).foo
(break foo) || a
(return foo) || a
a && b && c
a = b || break
a = b || next
a = b || return
a and return foo
a or return
a or return foo
a || (b || c)
a || (break foo)
a || (return foo)
b or break
b or next
break or b
next or b
return or a
a = b or c
a = b and c

Generated-Source:
undefined
Original-Node:
#<Parser::SyntaxError: unexpected void value expression>

4

test/corpus/literal/pattern.rb
Original-Source:
case foo
in A[1, 2, *a, 3] then
true
in [1, 2, ] then
y
in A(x:) then
true
in {**a} then
true
in {} if true then
true
in [x, y, ] then
true
in {a: 1, aa: 2} then
true
in {} then
true
in {**nil} then
true
in {"#{"a"}": 1} then
true
in 1 | 2 then
true
in 1 => a then
true
in ^x then
true
in 1
in 2 then
true
else
true
end
case foo
in A[1, 2, a, 3]
end
case foo
in A
else
end
1 in [a]
1 => [a]
1 => []
1 in [, 42, ]
1 in [, a, *foo]

Generated-Source:
undefined
Original-Node:
#<Parser::SyntaxError: symbol literal with interpolation is not allowed>

Not sure yet how to fix them