Around 50,000 convict men and boys passed through the barracks between 1819 and 1848. Most had been charged with property crimes in British courts and served terms of life, 7 or 14 years transportation. Typically, their lives were governed by rigid rules, discipline and hard, monotonous work although for some, good conduct or useful skills brought rewards, indulgences and positions of responsibility.

Barracks convicts were mustered daily and marched to worksites around town. In gangs, they built the docks, roads, churches, hospitals, quarries, bridges and fortifications of early Sydney. Their tools, equipment, food and clothing were supplied by gangs of convict cobblers, weavers, bakers, hat makers, grass cutters, gardeners, wheelwrights, carpenters and blacksmiths. From 1830 courts of General Sessions were held at the Barracks. Convicts and employers put their complaints to visiting magistrates who determined various penalties. Punishments could include the cruel treadmill, flogging, a stint in leg irons, solitary confinement, reassignment to a distant road gang or, worst of all, Cockatoo Island.