Your comments

can't we at least have a way of changing the behavior with keybindings?

something like:
{ "keys": ["up"], "command": "move", "args": {"by": "characters", "forward": false }, "context": 
    [ 
{ "key": "something", "operator": "something", "operand": "something" } 
    ] }
?
I cannot find the right context.