The Fanciest REPL History in the Land #59819

tecosaur · 2025-10-12T18:58:03Z

The Fanciest REPL History in the Land ✨

Do you dread typing out code for the second time? Are you a particular enjoyed of REPL history?

Well, I know I am, and for years I have yearned for something better than the current readline-style completion, better than OhMyREPL.jl's fzf-driven completion, better than any REPL history I've seen before!

It's not quite finished baking, but we're onto the final stretch 😀

repl_history_demo.webm

Thanks to @kdheepak, @jakobnissen, and @digital-carver for helping me design the UI and UX over on Zulip (#repl > Revamped REPL history).

Features

Zippy searching
- Event-driven asynchronous filtering UI
- Incremental, resumable searching with dynamic batch sizes
- Log-structured search checkpoints
Multi-selection
Faster histfile parsing (~2x)
Multiple search modes
A friendly help page
Syntax highlighting
Save multiple items to a file or your clipboard

TODO

Introduce annotation-preserving replace method
Thoroughly test the new replace method
Ask somebody more compiler-y about the performance pitfalls of the replace method (see: the REVIEW: ... code comments in annotated_io.jl)
Implement flashy REPL history
Restore up/down arrow history rotation in the REPL (collateral damage of over-zealous deleting)
Create a new test set for the new history
Make sure that enough is precompiled to be relatively snappy

This PR is on top of #59778, because I think I can safely assume that will be merged first.

KristofferC · 2025-10-13T10:53:06Z

If possible, it would be nice to cut this up into some orthogonal pieces. For example, the AnnotatedString perf improvements could be a separate PR that (with benchmarks) could be merged and would make the diff here smaller and easier to review.

stdlib/REPL/src/History/search.jl

kdheepak · 2025-10-13T15:09:26Z

stdlib/REPL/src/History/search.jl

+        filename = try
+            readline(term.in_stream)
+        catch err
+            if err isa InterruptException
+                ""
+            else
+                rethrow()
+            end
+        end
+        isempty(filename) && (println(out, S"\e[F\e[2K{light,grey:{bold:history>} {red:×} History selection aborted}\n"); return)
+        write(filename, "# Julia REPL history excerpt\n\n", content)


Maybe it should default write to a location? The prompt could default to ~/.julia/history.jl (or a more appropriate location).

Ideally, the prompt for the filename would also allow scrolling up and down with the up arrow and down arrow, typing some characters of the file and hitting up arrow to narrow the search, have a ghost text of the last file name or the default file name so right arrow completes it etc. But I can imagine this being awkward the way things are set up in the REPL?

Ideally, the prompt for the filename would also allow scrolling up and down with the up arrow and down arrow, typing some characters of the file and hitting up arrow to narrow the search, have a ghost text of the last file name or the default file name so right arrow completes it etc. But I can imagine this being awkward the way things are set up in the REPL?

I've had exactly the same thought. I just can't be bothered to make it happen 😅

tecosaur · 2025-10-13T16:06:22Z

If possible, it would be nice to cut this up into some orthogonal pieces.

I'm happy to split this up, but there are really just three parts to it:

A refactor to _replace_final (needed to reuse some code for annotation-preserving replacement)
Introducing support for annotation-preserving replacement (needed for changing multi-line annotated strings into single-line with linebreak symbols)
Replacing the old history system with the fancy new one

I could split 1-2 off into a separate PR to review, if that sounds like a nice idea?

the AnnotatedString perf improvements

There's only suspicious performance here, not perf improvements, unfortunately 😞. I say "suspicious" because if I remove the keyword argument from replace skip reconstructing a named tuple with a range shifed the total time of a small replace op drops from ~4400ns to ~200ns. Given that an equivalent annotation-less replace takes ~65ns a time of ~200ns would be decent I think, but ~4400ns is strange — I can't figure it out though. If I can interest anyone else in investigating, that would be great.

For now, I've left these REVIEW code comments:

julia/base/strings/annotated_io.jl

Lines 202 to 226 in cc7ed22

    
           # REVIEW: For some reason, construction of `newannot` 
        
           # can be a significant contributor to the overall runtime 
        
           # of this function. For instance, executing: 
        
           # 
        
           #     replace(AnnotatedIOBuffer(), S"apple", 
        
           #             'e' => S"{red:x}", 'p' => S"{green:y}") 
        
           # 
        
           # results in 3 calls to `_insert_annotations!`. It takes 
        
           # ~570ns in total, compared to ~200ns if we push `annot` 
        
           # instead of `newannot`. Commenting out the `_insert_annotations!` 
        
           # line reduces the runtime to ~170ns, from which we can infer 
        
           # that constructing `newannot` is somehow responsible for 
        
           # a ~30ns -> ~400ns (~13x) increase in runtime!! 
        
           # This also comes with a marginal increase in allocations 
        
           # (compared to the commented out version) of 2 -> 14 (250b -> 720b). 
        
           # 
        
           # This seems quite strange, but I haven't dug into the generated 
        
           # LLVM or ASM code. If anybody reading this is interested in checking 
        
           # this out, that would be brilliant 🙏. 
        
           # 
        
           # What I have done is found that "direct tuple reconstruction" 
        
           # (as below) is several times faster than using `setindex`. 
        
           newannot = (region = start+offset:stop+offset, 
        
                       label = annot.label, 
        
                       value = annot.value)

julia/base/strings/annotated_io.jl

Lines 236 to 240 in cc7ed22

    
           # REVIEW: For some reason the `Core.kwcall` indirection seems to cause a 
        
           # substantial slowdown here. If we remove `; count` from the signature 
        
           # and run the sample code above in `_insert_annotations!`, the runtime 
        
           # drops from ~4400ns to ~580ns (~7x faster). I cannot guess why this is. 
        
           function replace(out::AnnotatedIOBuffer, str::AnnotatedString, pat_f::Pair...; count = typemax(Int))

tecosaur · 2025-10-13T18:50:24Z

I've just added back ~400 loc of the original code for using up/down arrow keys to go through recent history without ^R.

If you start a new REPL and immediately press up arrow, it does nothing the first time. Not quite sure why... (any ideas?)

I would like to add session-awareness. Maybe I'll try to drop that in this PR too?

tecosaur · 2025-10-14T13:43:20Z

Rebased to the latest REPL Syntax Highlighting HEAD.

tecosaur · 2025-10-14T13:45:35Z

Fixed some typos.

tecosaur · 2025-10-15T16:23:17Z

The precompilation situation is looking pretty decent, this is all I'm seeing with --trace-compile:

#=   19.4 ms =# precompile(Tuple{Type{Base.IOContext{IO_t} where IO_t<:IO}, Base.GenericIOBuffer{Memory{UInt8}}, Base.TTY})
#=   12.9 ms =# precompile(Tuple{typeof(Base.get), Base.Dict{Tuple{Symbol, Any}, Int64}, Tuple{Symbol, Symbol}, Int64})
#=   19.2 ms =# precompile(Tuple{typeof(Base.setindex!), Base.Dict{Tuple{Symbol, Any}, Int64}, Int64, Tuple{Symbol, Symbol}})
#=    2.5 ms =# precompile(Tuple{typeof(Base.print), Base.IOContext{Base.GenericIOBuffer{Memory{UInt8}}}, Base.AnnotatedString{String}})
#=    1.6 ms =# precompile(Tuple{typeof(Base._str_sizehint), UInt64})
#=    1.9 ms =# precompile(Tuple{typeof(Base.print), Base.GenericIOBuffer{Memory{UInt8}}, UInt64})
#=    2.5 ms =# precompile(Tuple{typeof(Base.AnnotatedDisplay.ansi_write), typeof(Base.write), Base.GenericIOBuffer{Memory{UInt8}}, Base.AnnotatedString{String}})
#=   14.6 ms =# precompile(Tuple{typeof(Base.getindex), Base.JuliaSyntax.GreenNode{Base.JuliaSyntax.SyntaxHead}, Int64})
#=    7.8 ms =# precompile(Tuple{typeof(Base.Terminals.cmove_up), Base.Terminals.TerminalBuffer})

tecosaur · 2025-10-15T16:28:36Z

Rebased now that #59778 has been merged.

tecosaur · 2025-10-15T16:50:25Z

If you start a new REPL and immediately press up arrow, it does nothing the first time. Not quite sure why... (any ideas?)

Fixed.

tecosaur · 2025-10-16T18:22:49Z

Thanks to Miguel, Camillo, and Sundar for taking this for a test-drive and providing feedback 🙂

Changes:

Clarified that ; uses and logic in quick help
Strip meaningless whitespace around filter segments
Changed "negative" → "negated" search term
Arrow keys (and others) no longer accidentally abort save
Prevent runaway redisplays
More minimal display updates
Make mmap work on Windows

tecosaur · 2025-10-17T18:21:10Z

Rebase (get the REPL precompilation improvements)

tecosaur · 2025-10-23T14:39:37Z

(hopefully) fix 32-bit CI

Extract the replacement loop body into `_replace_once` to ease future annotation tracking during string replacement operations. The new function returns match information (pattern index, match range, bytes written) that will be needed to properly adjust annotation positions when replacements occur.

Implement `replace` function for `AnnotatedString` that properly handles annotation regions during pattern replacement operations. The function tracks which bytes are replaced versus preserved, maintaining annotations only on original content and adding new annotations from replacement text. - Supports AnnotatedChar, AnnotatedString, and SubString replacements - Drops, shifts, and splits existing annotations appropriately - Refactored `_insert_annotations!` to work with annotation vectors directly - Adjacent replacements with identical annotations are merged into single regions - Lots of tests (thanks Claude!) Performance is strangely poor. For the test case mentioned in the REVIEW comment within `_insert_annotations!` we should be able to perform the replacement in ~200ns (compared to ~70ns for the equivalent unannotated case). However, for two reasons that are beyond me instead it takes ~4400ns. See the REVIEW comments for more details, help would be much appreciated.

Discarding the annotations can come as a bit of a surprise best avoided.

tecosaur · 2025-10-23T14:55:42Z

Update the branch

tecosaur · 2025-10-23T15:19:26Z

Prevent jitter when holding TAB at the end of a series of selected items

stdlib/REPL/src/LineEdit.jl

Since the dawn of the Julia REPL, history completion has been limited to a readline-style interface. OhMyREPL improved the experience with fzf, but those who yearned for a delightful history completion experience (me) were left underwhelmed. With this overhaul, I now find myself spending more time looking through my history because it's just *so nice* to do so. The new history system is organised as a standalone module in stdlib/REPL/src/History with a clear separation of concerns: 1. History file management 2. Event-driven prompt/UI updating 3. Incremental filtering 4. UI display 5. Search coordination (prompt + display + filter) I've attempted to pull out all the (reasonable) stops to make history searching as fluid and snappy as possible. By memory mapping the history file in the initial read, and optimising the parser, we can read ~2 million history items per second. Result filtering is incremental and resumable, performed in dynamically sized batches to ensure responsiveness. Rapid user inputs are debouced. We store a log-structured record of previous search result, and compare search strictness to resume from prior partial results instead of filtering the history from scratch every time. Syncronisation between the interface and filtering is enabled via a Channel-based event loop. Enjoy! (I know I am)

tecosaur · 2025-10-23T15:44:40Z

I imagined that prompt_prefix took an argument

rfourquet · 2025-10-23T15:55:20Z

It seems you fixed the problem with keeping a reference to the first history entry in the session, great! But now it starts at 0, surely not acceptable for julia ;-)

(But maybe you again fixed it since yesterday)

tecosaur · 2025-10-23T15:57:01Z

It seems you fixed the problem with keeping a reference to the first history entry in the session, great! But now it starts at 0, surely not acceptable for julia ;-)

I did? This is (happy) news to me. That said, I never quite got to the bottom of why it wasn't working.

KristofferC · 2025-10-25T11:00:29Z

Let's merge this now so we get a "checkpoint" when CI is green and people can test it more, the remaining niggles shouldn't be a problem addressing post-merge.

topolarity · 2025-10-28T11:43:38Z

This is great fun. Feels very polished generally, and I'm very impressed by how responsive it feels.

Couple of early thoughts:

Is it possible / advisable to filter repeated entries?

I wonder if the REPL mode filtering shouldn't just be pkg> / julia> / shell> (i.e. as an exact match to the "restriction" text). I wasn't able to understand how to use > without the example, and it does collide with some valid function invocations:

Escaping doesn't quite seem to work the way I'd expect:

- What does `words [12/12]` mean?

KristofferC · 2025-10-28T19:08:22Z

Is it possible / advisable to filter repeated entries?

I think this is a good idea, it is often that I end up with the whole history being identical entries.

tecosaur · 2025-10-28T19:17:44Z

Is it possible / advisable to filter repeated entries?

I'm a fan of this idea, I'll put it in a follow-up PR.

I wonder if the REPL mode filtering shouldn't just be pkg> / julia> / shell>

I also wonder, however I'm wary the inconsistency from making a search mode that relies on a suffix.

Escaping doesn't quite seem to work the way I'd expect

I've noticed that the filtering isn't entirely working how I think it should. I think this is escaping working as you expect, but filtering not.

What does words [12/12] mean?

That you're currently doing a word-based search, and have result 12 out of 12 focused.

topolarity · 2025-10-30T13:31:59Z

I wonder if the REPL mode filtering shouldn't just be pkg> / julia> / shell>

I also wonder, however I'm wary the inconsistency from making a search mode that relies on a suffix.

FWIW, I don't think of this as a suffix really - I think of it as an exact match for "julia>", "pkg>", or "shell>"

Custom REPL modes complicate that a bit. I don't think "suffix" is the right way to refer to it in the user help / documentation (even though it'd be accurate)

That you're currently doing a word-based search

What are the other type of searches? Is there a discovery path for the user to learn what that means?

I was really pleased that everything else (e.g. the colored dots for prompt mode) is pretty direct to learn through interactive usage - that seems like great design to me. The "words" was the one bit of TUI that I couldn't learn how to read on my own

tecosaur · 2025-10-31T05:34:37Z

I'm a fan of this idea, I'll put it in a follow-up PR.

Done. You'll see this in #59953 soon

I wonder if the REPL mode filtering shouldn't just be pkg> / julia> / shell>

I've worked out something I'm happy with, making a xyz> prefix in word mode become a mode filter (e.g. pkg> add becomes a search for add with mode=:pkg)

What are the other type of searches? Is there a discovery path for the user to learn what that means?

Did you look at the help? Based on the way you escaped ; in the screenshot you shared, I would have guessed so.

topolarity · 2025-10-31T14:54:36Z

I'm a fan of this idea, I'll put it in a follow-up PR.

Done. You'll see this in #59953 soon

I'd recommend a separate PR probably. Makes it easier to tweak / take in feedback for individual features, etc.

I've worked out something I'm happy with, making a xyz> prefix in word mode

Nice, I like that! Feels very intuitive.

Did you look at the help?

I did yeah! I think what might have tripped me up is that none of the help documentation mentions "words"

In comparison, the "regexp" / "exact" / "fuzzy" / "initialism" statuses are quite clear and correspond directly to the help too. "words" is just a very generic term for the default search mode.

I wonder if maybe "words" / "separator" are unnecessary statuses and we could just leave them blank and display only the "special" modes?

tecosaur · 2025-10-31T15:01:11Z

I'd recommend a separate PR probably. Makes it easier to tweak / take in feedback for individual features, etc.

I expect I'll end up doing that, but while I'm adding tweaks to the current behaviour it's good to see what accumulates before making a PR out of it: I don't want to do half a dozen individual PRs.

Nice, I like that! Feels very intuitive.

Yea, it "feels right" to me (which is what I was waiting for before going ahead with an option). I do wonder if we should pre-populate the query for a search made in mode XYZ with xyz>?

I did yeah! I think what might have tripped me up is that none of the help documentation mentions "words"

This is good feedback. We could not mention words, but I'm more inclined to better mention words in the help.

This comment was marked as resolved.

Sign in to view

kdheepak reviewed Oct 13, 2025

View reviewed changes

stdlib/REPL/src/History/search.jl Outdated Show resolved Hide resolved

kdheepak reviewed Oct 13, 2025

View reviewed changes

tecosaur force-pushed the fancy-repl-history branch from cc7ed22 to 75fd56d Compare October 13, 2025 18:48

tecosaur force-pushed the fancy-repl-history branch from 75fd56d to 221b4e5 Compare October 14, 2025 13:42

tecosaur force-pushed the fancy-repl-history branch from 221b4e5 to ee23322 Compare October 14, 2025 13:45

tecosaur mentioned this pull request Oct 15, 2025

Make banner size depend on terminal size #51811

Merged

KristofferC mentioned this pull request Oct 15, 2025

add support for automatic bracket completion in REPL input #59847

Merged

tecosaur force-pushed the fancy-repl-history branch from ee23322 to 3a6d8d7 Compare October 15, 2025 16:28

tecosaur force-pushed the fancy-repl-history branch from 3a6d8d7 to 4bac24f Compare October 15, 2025 16:50

tecosaur force-pushed the fancy-repl-history branch from 4bac24f to c7a66cb Compare October 16, 2025 18:22

tecosaur force-pushed the fancy-repl-history branch 2 times, most recently from 7126dd7 to 0da2a86 Compare October 17, 2025 18:20

tecosaur force-pushed the fancy-repl-history branch from 03ca9f0 to 4072ade Compare October 23, 2025 14:39

tecosaur added 3 commits October 23, 2025 22:54

Preserve annots of case transformed annot substr

6694598

Discarding the annotations can come as a bit of a surprise best avoided.

tecosaur force-pushed the fancy-repl-history branch from 4072ade to dd8bf6e Compare October 23, 2025 14:55

tecosaur force-pushed the fancy-repl-history branch from dd8bf6e to cc2f31f Compare October 23, 2025 15:19

rfourquet reviewed Oct 23, 2025

View reviewed changes

stdlib/REPL/src/LineEdit.jl Outdated Show resolved Hide resolved

tecosaur and others added 4 commits October 23, 2025 23:44

Fix Mac precompile hang

aed28c3

Add news and docs for the new history searcher

448a3a1

Tell typos CI that "indexin" is valid

24a5ca2

tecosaur force-pushed the fancy-repl-history branch from cc2f31f to 24a5ca2 Compare October 23, 2025 15:44

KristofferC merged commit d1e2d25 into JuliaLang:master Oct 25, 2025
7 checks passed

This was referenced Oct 25, 2025

Record and display more REPL history details #59953

Draft

Meta-issue for improving the REPL experience #50817

Open

Lazy/async loading of REPL history #28187

Closed

Uh oh!

The Fanciest REPL History in the Land #59819

The Fanciest REPL History in the Land #59819

Conversation

tecosaur commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!