There is always one that is a little braver! We have foxes living in our garden, and they had 7cubs! One would always come up and see me(it has a scar so I can tell it's always her) but the others where always a little scared. She still comes in my garden now, the others don't though.
I bet your squirrel will get more brave and take one from your hand eventually! :)