I say go for it. If anyone put nuclear waste of anything in them, they wouldn't be empty right now. They'd be buried under a mountain somewhere. Nuclear waste isn't something you can just dump out on the side of the road...
As for the aggregate/sand; it might have been the kind they use for making molds of for sand-blasting. That's the only thing I can think of. See what it says on the labels when you go to get them.
"Eating 'Que isn't making me fat- it's the beer I drink while lounging by the pit."