For long time I was frustrated form inability to use arrow keys in Adobe acrobat reader to move up and down the page.
Instead of that the cursor was moving in the text.
Finally I realized that this is used to select text. And surprisingly enough I was in Select mode.
Right click and change to Hand tool and I now have the desired behavior.
Hope this will help someone.