I'd be worried, but being able to handle the ball is a skill that can very easily be improved upon and especially if you get the right trainers and have all the time in the world to develop it.
I can almost guarantee you that you didn't see Jimmy Butler handling the ball as much now, or as well as he is now, when he was a rookie. He was also a older rookie as well.