The term translated as rubble
is literally soil.
Over the span of the century and a half since the wall was dismantled, vegetation had grown around and over the stones and had in turn decomposed and returned to dirt. Retrieving the stones required digging through soil bound together by the roots of further shrubs and trees. That is serious toil!
10 In Judah it was said, “The strength of those who bear the burdens is failing. There is too much rubble. By ourselves we will not be able to rebuild the wall.”