Sniffer rule for video/mp2t too simple

[dsuden]

[See comment 2 to reveal the mystery]

I had a flash video (downloaded with UberTuber) sitting on my desktop, and then deleted it, so it was no longer on the desktop. Later that day, I saved a text file from Pe to the desktop. I didn't look at it at the time, but this morning I noticed my text document had the same icon as the flash video I had trashed. I was able to use filetypes to change the document back to a text file, which allowed it to again be opened normally in Pe, and its text looked fine. But that seemed like a strange enough turn-of-events that I thought I'd better report it. Note, the trashed video file also had a completely different filename than the text file on the desktop.

[dsuden]

I just saved another text file to the desktop, and it happened again. This time I double-checked the filetype right away and it's not a flash as I thought, but an "MPEG-2 transport stream." So the deleted flash file might be a red herring. Still...how strange to have Pe saving files as video!

[humdinger]

Hold on, let me contact the spirits beyond... <insert dark atmospheric music>

I get a "G".... Does the contents of your text file start with a "G" by any chance? I bet it does. Culprit is the naive sniffer rule for video/mp2t filetypes. It just checks if the first byte is 0x47. Apparently transport streams do always beginn with 0x47, but as that's just a "G", anything starting with that qualifies.
​ReadMe.IntroductionToHaiku in the root of the repository suffers that same issue.

You can "solve" this for the time being by removing the rule in the Filetypes preferences.

Can anyone come up with a better sniffer rule? It looks like after that first sync byte, there isn't much to analyze (s. ​wikipedia), or is there?

[axeld]

Indeed, I think the format spec is too weak to be sniffed, at least with a high probability of failure. We could remove the sniffer, and only detect the type by extension.

We could also try to make its support level lower, and demand that the bytes following are not ASCII. Maybe this would be good enough already, but it might be the source of problems in the future again, so probably not worth pursuing.

[jscipione]

Patch that removes the mp2t sniffer rule that is causing the problem reported by this bug.

[jscipione]

Fixed in hrev44886 by removing the sniffer rule

[Giova84]

hrev48250

I have found again this bug: text files which start with a "g" character are mistakenly marked as video/mp2t. This occurs on BFS, too.

[phoudoin]

Replying to humdinger:

Can anyone come up with a better sniffer rule? It looks like after that first sync byte, there isn't much to analyze (s. ​wikipedia), or is there?

Our sniffer rule syntax don't support a repeat mode, otherwise a way better rule for MPEG files would be simply to check the sync 0x47 ('g') byte every 188 bytes, not just first one.

While a repeat pattern could be hard to implement, what could be done quickly is to support combining patterns with & (AND), not just with | (OR) as today. That would make this rule :

"0.9 ('g' & [188] 'g' & [376] 'g' & [564] 'g')"

...detecting better MP2+ content.

[X512]

Mayble having higher priority for plain text detection solve problem?

[phoudoin]

Replying to X512:

Mayble having higher priority for plain text detection solve problem?

That would be a trick. The issue is that our current sniffer rule syntax is not enough flexible to express a type identification rule for format that can be identified only by repeated pattern, not just some signature at begining.