Unselecting multiple selected list items difficult/impossible

[humdinger]

This is hrev53552.

• Open e.g. the Locale prefs
• Drag a couple of languages from the left to the right list
• Select a couple of items in the right list by clicking the 2nd and shift-click the 5th

Unselecting this block of items is difficult. You'd expect clicking on any item in that block would unselect all but the selected item. It doesn't. You have to click outside the selected items.
This becomes impossible even, if you have selected all items in the list...

If unselecting the block by clicking an item collides with drag&drop detection, unselecting at mouse-up seems acceptable in this case.