It won't hurt or bother them. I do it all the time. Sometimes nest aren't made right and they can't "shoot" over the side. I just take it all out and add new material(straw/pine needles) and put the babys back in.
Yeah, not very good nest makers.. and i didn't give them a little bowl to use, so there are droppings all down the side. However, I found one baby on the ground yesterday. I was concerned that by getting my scent on the babies that the parents might reject them. I rubbed hay on my hands before putting the baby back, and mom is with her babies today.