Your comments

I agree that an option to make the arrow keys end the search and move left/right/up/down at the current position would be desirable. To me, this would remove the need for ESC placing the cursor at the currently highlighted point.

I feel this behaviour is desirable, as it allows you to cancel a search with 'Esc' and go back to where you were before. This also mirrors how incremental search works in Emacs.

I agree with your comments about the cursor keys cancelling the search however (at least the up and down keys)