Directory history breaks accents in 8-bit locales

[egmontkob]

Is there an existing issue for this?

• I have searched the existing issues

Midnight Commander version and build configuration

4.8.33

Operating system

Linux

Is this issue reproducible using the latest version of Midnight Commander?

• I confirm the issue is still reproducible with the latest version of Midnight Commander

How to reproduce

With an 8-bit locale, navigate to a directory which has some accents (probably in UTF-8, thus right now showing up incorrectly). Quit mc.

Notice that .local/share/mc/history, under Dir Hist New [Left/Right] Panel, contains this entry encoded in "double UTF-8", each byte represented by the corresponding Unicode codepoint (up to U+00FF) encoded in UTF-8.

Start mc with an UTF-8 locale, click on the dropdown entry at the panel's upper right corner to open the directory listing. This recent entry is shown incorrectly. Selecting it doesn't take you to that directory.

Unix treats filenames as byte sequences, rather than Unicode character sequences. Accordingly, we should store unmodified byte sequences in the history file (possibly backslash-escaping high bytes).

(I'm not quite sure whether other sections in that file, e.g. mc.shared.search, inpmc.fm.mkdir or cmdline should also be byte-safe, or if here it's more important to keep the user-visible string the same across locale changes even if it means changing the underlying byte sequence.)

Expected behavior

.

Actual behavior

.

Additional context

No response

[mc-worker]

Unix treats filenames as byte sequences, rather than Unicode character sequences. Accordingly, we should store unmodified byte sequences in the history file (possibly backslash-escaping high bytes).

#391
GLib wants valid Unicode in ini-files.

[egmontkob]

Then I guess we should use some kind of escaping. Either for invalid UTF-8 portions only, or for every high byte.

This will probably be another layer above glib's \n-escaping, thus glib would convert our \HH to \\HH, but meh. We can choose anything else, like quoted-printable, if we wish to.

[zyv]

Then I guess we should use some kind of escaping. Either for invalid UTF-8 portions only, or for every high byte.

This will probably be another layer above glib's \n-escaping, thus glib would convert our \HH to \\HH, but meh. We can choose anything else, like quoted-printable, if we wish to.

Maybe g_strescape / g_strcompress could be used? My understanding is that it does too much, but at least it's kind of readable for the ASCII part and doesn't involve much work:

https://docs.gtk.org/glib/func.strescape.html

I think it would be nice to not write any own code and not use external libraries other than glib, but I don't think it can provide much. It has base64 encoding/decoding, but I find non-transparent encoding very inconvenient if one would want to open these files in the editor... Also, it would be nice to be able to read the old history and avoid writing migration code.

[egmontkob]

Maybe g_strescape / g_strcompress

Sounds perfect to me.

It has base64 encoding/decoding, but I find non-transparent encoding very inconvenient if one would want to open these files in the editor...

Fully agree. Let's use an encoding that leaves ASCII characters as-is.

Also, it would be nice to be able to read the old history and avoid writing migration code.

Honestly, if we once break entries with accented characters in the history, I don't really care. (If we broke non-accented ones (like with base64) that'd be a problem.)