Either the pieces are too big and the chicks ignore them until they grow, or the pieces are an acceptable size and the chicks eat them. Either way, the chicks are not in any danger.Nothing harmful using adult grit at 5 weeks then?
Most likely, the chicks pick through the grit you give them, throw most of it on the floor, and eat a few bits that are the right size. As they get bigger, they choose somewhat bigger pieces, but even then they may have personal preferences about what size or shape they want.
The smallest pieces of grit usually fall to the bottom of the package. So if you open the bag from the bottom, or dump it all into a container, you can give your chicks the small pieces from the bottom now, and the bigger pieces are mostly left for later. You can even dump the grit into a bowl or bucket, scoop some from the bottom for the chicks, and dump the rest back into the original bag if you want to store it there. It's not a big deal, just something you can do if you want to. You can just scoop from the top of the bag, and you will still get a mix of sizes, just with a few more big pieces and a few less little pieces.