This is not just about Linux: The same behaviour exists on OS X and I presume Windows.

The latest dev build supports this. Ctrl+0 moves you to the sidebar, and then the arrow keys and enter work as expected.

This is now possible in build 2111.