Ezra 3:8–9 (ESV)

8 Now in the second year after their coming to the house of God at Jerusalem, in the second month, Zerubbabel the son of Shealtiel and Jeshua the son of Jozadak made a beginning, together with the rest of their kinsmen, the priests and the Levites and all who had come to Jerusalem from the captivity. They appointed the Levites, from twenty years old and upward, to supervise the work of the house of the LORD.

Under the supervision of the Levites, the work of construction on the foundation of the temple begins. Zerubbabel and Joshua lead and encourage the people to participate in the work. All the exiles who returned from Babylon were involved somehow even if they were not all directly involved in digging holes and carrying stones.