The phrase could refer to the common folk who would do the actual reconstruction work. Given that this group of people is listed alongside persons of influence (the priests, nobles and officials), it is more likely that with this phrase Nehemiah is thinking of those who would have crews available to do the work (Nehemiah 3:1–32).
16 And the officials did not know where I had gone or what I was doing, and I had not yet told the Jews, the priests, the nobles, the officials, and the rest who were to do the work.