Opened 10 years ago

Closed 8 years ago

#6510 closed bug (fixed)

terminal not handling color sequences properly

Reported by: augiedoggie Owned by: siarzhuk
Priority: normal Milestone: R1
Component: Applications/Terminal Version: R1/Development
Keywords: Cc:
Blocked By: Blocking: #9258
Platform: All

Description

Running a 'cat' on certain files will show colors were there shouldn't be any. Emacs syntax highlighting is not using the right colors anymore and is sometimes drawing color where there should be none. The 256-color demo script isn't rendering properly. It continues colors to the end of the line when it shouldn't. And so on...

All of these problems have shown up within the past couple of days. It was working fine before.

Attaching a couple screenshots of the bad behavior.

Attachments (9)

screenshot1.png (89.9 KB ) - added by augiedoggie 10 years ago.
running cat on a source file
screenshot2.png (111.4 KB ) - added by augiedoggie 10 years ago.
the 256-color test script
Shell.cpp.patch (88 bytes ) - added by tidux 9 years ago.
fixes TERM setting for Terminal
256-colour-fail.png (80.2 KB ) - added by xray7224 8 years ago.
256 colours perl script in a tmux session
step1.png (77.2 KB ) - added by tidux 8 years ago.
step2.png (58.5 KB ) - added by tidux 8 years ago.
step3.png (607.4 KB ) - added by tidux 8 years ago.
laststep.png (91.4 KB ) - added by tidux 8 years ago.
irssi gets absolutely mangled by Terminal
termcap (3.5 KB ) - added by siarzhuk 8 years ago.
Test version of actual termcap database.

Download all attachments as: .zip

Change History (26)

by augiedoggie, 10 years ago

Attachment: screenshot1.png added

running cat on a source file

by augiedoggie, 10 years ago

Attachment: screenshot2.png added

the 256-color test script

comment:1 by mmadia, 10 years ago

Owner: changed from jackburton to pulkomandy
Status: newassigned

Adrien, would you mind looking into this?

comment:2 by pulkomandy, 10 years ago

What happens is :

  • I changed the behaviour of termView to be more valid with regard to escape sequences. This means that every character position on screen must be tied to an attribute word. Unfortunately, this is not how the backend worked, it added attributes only to characters that had something "custom". This makes no sense, since the other chars still had an attribute word, but zeroed out.
  • What's left is handling chars that spans over multiple columns. This include tabulations (up to 8 columns), cursor move sequences (should fill the space they skip with the latest used color), and end of line. The end of line is the most tricky to handle. It goes as this : the last char attribute should be replicated, unless there was an ANSI attribute change in between, eg : set color to blue ; draw text ; newline > line is blue from the set color to the end, whereas set color to blue ; draw text ; set color to red ; newline > line is red from the second 'set color' to the end of the line. This second case is not handled as the newline is not stored as a character in the termView buffer. This also mix up the beginning of the following line in some cases because the start of a line tries to figure out the attribute from the last char in the previous line.

comment:3 by bonefish, 10 years ago

Adrien, are you going to work on this anytime soon? My fingers are itching to simply revert what you have done. The behavior might have been less correct with respect to (whatever) escape sequences, but at least I didn't see any graphics glitches in everyday use before.

comment:4 by pulkomandy, 10 years ago

I'll have a look. The current state is working better for many apps, but still isn't 100% perfect.

comment:5 by pulkomandy, 10 years ago

The first screenshot/bug (the most annoying and dangerous) should now be fixed. Tab chars at the start of a line tried to read something in the previous char when there was none, leading to unitialized memory being read.

The second one is not fixed. Basically, when the sequence is 'insert a color change', followed by 'insert a newline', the new color isn't taken into account until after the newline was inserted. I believe the same happens with some commands that make the cursor move around (ansi escape moves). This is because a newline only moves the cursor, but doesn't insert any character. I tried inserting an \0 (null char) but that didn't work. So I'm not sure what to do for this one.

comment:6 by pulkomandy, 10 years ago

As of hrev39578, both problems shouldbe fixed. 256colors.pl is working properly. I did notice some glitches left in vim, I thnk they come from ANSI escape commands for moving the cursor. Will check that next, but I think the issues here should mostly be fixed.

Sorry for the long delay.

As for Emacs, I'm not sure how the color can become wrong. Maybe it tries to use the extra 256-color more but with sequences to set the palette we're not handling yet.

comment:7 by siarzhuk, 9 years ago

Owner: changed from pulkomandy to siarzhuk

I'm working on supporting OSC at the moment and have to dig in all this ANSI soup so deeply that I must either clean these "Augean stables" or perish in the fight. So wish me luck.

comment:8 by tidux, 9 years ago

I've been having color issues for ages, so I tried messing with the $TERM setting. For example, Vim syntax highlighting still doesn't work properly with the default TERM=xterm, and irssi on my Linux server got garbled as heck. Fortunately, setting TERM=xterm-color fixed every issue I've had apart from GNU screen, which I think I can chalk up to GNU screen being a buggy pile of spaghetti code. My initial workaround was to put "export TERM=xterm-color" in my ~/.bash_profile - but that's a crappy fix. I poked around the Terminal source code and found the place it was set in Shell.cpp. The file I'm about to attach is a simple patch to put that fix in the source code instead of requiring editing config files like we're on Unix.

by tidux, 9 years ago

Attachment: Shell.cpp.patch added

fixes TERM setting for Terminal

comment:9 by tidux, 9 years ago

patch: 01

comment:10 by diver, 8 years ago

Blocking: 9258 added

comment:11 by xray7224, 8 years ago

This still exists but ONLY in tmux (tmux changes some environment variable). I have tried exporting $TERM (which is one of the ones which gets changed) however that doesn't help.

I just realised I ran the test with FonzoTerm though I double checked after and it works exactly the same with the haiku stock terminal. If you'd like I'll post a screenshot of it happening in that too (but it's exactly the same thing)

Last edited 8 years ago by xray7224 (previous) (diff)

by xray7224, 8 years ago

Attachment: 256-colour-fail.png added

256 colours perl script in a tmux session

by tidux, 8 years ago

Attachment: step1.png added

by tidux, 8 years ago

Attachment: step2.png added

by tidux, 8 years ago

Attachment: step3.png added

by tidux, 8 years ago

Attachment: laststep.png added

irssi gets absolutely mangled by Terminal

comment:12 by tidux, 8 years ago

SSH seems to bring out the worst in Terminal. Pictures related to some very simple, basic failures.

in reply to:  12 comment:13 by siarzhuk, 8 years ago

Replying to tidux:

SSH seems to bring out the worst in Terminal. Pictures related to some very simple, basic failures.

The issue with irssi looks like the same as observed in #3164 with silc application in ssh session. Try to play with "stty -echonl" before starting irssi.

BTW, I have fixed lot of issues in the branch Terminal project available at https://gitorious.org/sz-sandbox/terminal - could, you, please test that version of Terminal? Note that replacement of /etc/termcap is required. Thank you!

by siarzhuk, 8 years ago

Attachment: termcap added

Test version of actual termcap database.

comment:14 by tidux, 8 years ago

The version of Terminal from Gitorious works great with the new termcap! I have to make sure my screen sessions on remote systems are run with "stty -echonl" but other than that the only remaining problem is that some sequences of characters that include multibyte "wide" glyphs get truncated early. Take a random line of Japanese text and open it in StyledEdit (or just copy it from Wikipedia in WebPositive), and then paste that line of text into Terminal. It gets truncated early, and I think it has to do with the algorithm for determining whitespace characters.

Last edited 8 years ago by tidux (previous) (diff)

in reply to:  14 comment:15 by siarzhuk, 8 years ago

Replying to tidux:

The version of Terminal from Gitorious works great with the new termcap!

Nice to hear. Do you have possibility to check current bold attribute rendering in this version? I have implemented R5-like "left-down shadow" one but it looks like a bit "fuzzy" and non-clear nowadays in our opinion.

I have to make sure my screen sessions on remote systems are run with "stty -echonl" but other than that the only remaining problem is that some sequences of characters that include multibyte "wide" glyphs get truncated early. Take a random line of Japanese text and open it in StyledEdit (or just copy it from Wikipedia in WebPositive), and then paste that line of text into Terminal. It gets truncated early, and I think it has to do with the algorithm for determining whitespace characters.

Looks like the problem described in #6717. I still did nothing about it - just want to finish set of problems with background color erasing ATM.

comment:16 by tidux, 8 years ago

The bold aspect does look a bit fuzzy but it's much better than the old behavior where bold text just broke a lot of things.

comment:17 by siarzhuk, 8 years ago

Resolution: fixed
Status: assignedclosed

Most of observed issues fixed in hrev45364.

Note: See TracTickets for help on using tickets.