Instead of using the @, # or : character you could always use the modifier key bindings. That would provide a single key binding regardless of context. For line number you would always know Command-G. I find that I am often looking at the preview of a file that the Goto Anything menu has open, so my brain is wired for file key bindings.

The @, # and : characters are only useful in the context of the menu– it seems more consistent and intuitive, to me, to just use the modifier based key bindings.

That's all :)