terminal not handling color sequences properly

[augiedoggie]

Running a 'cat' on certain files will show colors were there shouldn't be any. Emacs syntax highlighting is not using the right colors anymore and is sometimes drawing color where there should be none. The 256-color demo script isn't rendering properly. It continues colors to the end of the line when it shouldn't. And so on...

All of these problems have shown up within the past couple of days. It was working fine before.

Attaching a couple screenshots of the bad behavior.

[augiedoggie]

running cat on a source file

[augiedoggie]

the 256-color test script

[mmadia]

Adrien, would you mind looking into this?

[pulkomandy]

What happens is :

• I changed the behaviour of termView to be more valid with regard to escape sequences. This means that every character position on screen must be tied to an attribute word. Unfortunately, this is not how the backend worked, it added attributes only to characters that had something "custom". This makes no sense, since the other chars still had an attribute word, but zeroed out.
• What's left is handling chars that spans over multiple columns. This include tabulations (up to 8 columns), cursor move sequences (should fill the space they skip with the latest used color), and end of line. The end of line is the most tricky to handle. It goes as this : the last char attribute should be replicated, unless there was an ANSI attribute change in between, eg : set color to blue ; draw text ; newline > line is blue from the set color to the end, whereas set color to blue ; draw text ; set color to red ; newline > line is red from the second 'set color' to the end of the line. This second case is not handled as the newline is not stored as a character in the termView buffer. This also mix up the beginning of the following line in some cases because the start of a line tries to figure out the attribute from the last char in the previous line.

[bonefish]

Adrien, are you going to work on this anytime soon? My fingers are itching to simply revert what you have done. The behavior might have been less correct with respect to (whatever) escape sequences, but at least I didn't see any graphics glitches in everyday use before.

[pulkomandy]

I'll have a look. The current state is working better for many apps, but still isn't 100% perfect.

[pulkomandy]

The first screenshot/bug (the most annoying and dangerous) should now be fixed. Tab chars at the start of a line tried to read something in the previous char when there was none, leading to unitialized memory being read.

The second one is not fixed. Basically, when the sequence is 'insert a color change', followed by 'insert a newline', the new color isn't taken into account until after the newline was inserted. I believe the same happens with some commands that make the cursor move around (ansi escape moves). This is because a newline only moves the cursor, but doesn't insert any character. I tried inserting an \0 (null char) but that didn't work. So I'm not sure what to do for this one.

[pulkomandy]

As of hrev39578, both problems shouldbe fixed. 256colors.pl is working properly. I did notice some glitches left in vim, I thnk they come from ANSI escape commands for moving the cursor. Will check that next, but I think the issues here should mostly be fixed.

Sorry for the long delay.

As for Emacs, I'm not sure how the color can become wrong. Maybe it tries to use the extra 256-color more but with sequences to set the palette we're not handling yet.

[siarzhuk]

I'm working on supporting OSC at the moment and have to dig in all this ANSI soup so deeply that I must either clean these "Augean stables" or perish in the fight. So wish me luck.

[tidux]

I've been having color issues for ages, so I tried messing with the $TERM setting. For example, Vim syntax highlighting still doesn't work properly with the default TERM=xterm, and irssi on my Linux server got garbled as heck. Fortunately, setting TERM=xterm-color fixed every issue I've had apart from GNU screen, which I think I can chalk up to GNU screen being a buggy pile of spaghetti code. My initial workaround was to put "export TERM=xterm-color" in my ~/.bash_profile - but that's a crappy fix. I poked around the Terminal source code and found the place it was set in Shell.cpp. The file I'm about to attach is a simple patch to put that fix in the source code instead of requiring editing config files like we're on Unix.

[tidux]

fixes TERM setting for Terminal

[xray7224]

This still exists but ONLY in tmux (tmux changes some environment variable). I have tried exporting $TERM (which is one of the ones which gets changed) however that doesn't help.

[xray7224]

256 colours perl script in a tmux session

[tidux]

irssi gets absolutely mangled by Terminal

[tidux]

SSH seems to bring out the worst in Terminal. Pictures related to some very simple, basic failures.

[siarzhuk]

Replying to tidux:

SSH seems to bring out the worst in Terminal. Pictures related to some very simple, basic failures.

The issue with irssi looks like the same as observed in #3164 with silc application in ssh session. Try to play with "stty -echonl" before starting irssi.

BTW, I have fixed lot of issues in the branch Terminal project available at ​https://gitorious.org/sz-sandbox/terminal - could, you, please test that version of Terminal? Note that replacement of /etc/termcap is required. Thank you!

[siarzhuk]

Test version of actual termcap database.

[tidux]

The version of Terminal from Gitorious works great with the new termcap! I have to make sure my screen sessions on remote systems are run with "stty -echonl" but other than that the only remaining problem is that some sequences of characters that include multibyte "wide" glyphs get truncated early. Take a random line of Japanese text and open it in StyledEdit (or just copy it from Wikipedia in WebPositive), and then paste that line of text into Terminal. It gets truncated early, and I think it has to do with the algorithm for determining whitespace characters.

[siarzhuk]

Replying to tidux:

The version of Terminal from Gitorious works great with the new termcap!

Nice to hear. Do you have possibility to check current bold attribute rendering in this version? I have implemented R5-like "left-down shadow" one but it looks like a bit "fuzzy" and non-clear nowadays in our opinion.

I have to make sure my screen sessions on remote systems are run with "stty -echonl" but other than that the only remaining problem is that some sequences of characters that include multibyte "wide" glyphs get truncated early. Take a random line of Japanese text and open it in StyledEdit (or just copy it from Wikipedia in WebPositive), and then paste that line of text into Terminal. It gets truncated early, and I think it has to do with the algorithm for determining whitespace characters.

Looks like the problem described in #6717. I still did nothing about it - just want to finish set of problems with background color erasing ATM.

[tidux]

The bold aspect does look a bit fuzzy but it's much better than the old behavior where bold text just broke a lot of things.

[siarzhuk]

Most of observed issues fixed in hrev45364.