There was also significant movement of population within Australia itself. Of particular concern was the movement of people from VDL. Transportation to NSW and Victoria had ceased by the time gold was discovered in 1851. Nonetheless, VDL remained a penal colony with a very high proportion of transportees to free settlers. On the mainland they were disparagingly known as ‘Vandiemonians’ or ‘Vandemonians’ referring to the place where they had often served time blended with ‘demon’. They flooded into Victoria in the early days of the gold rushes and in the second half of 1851 there were more recorded immigrants from VDL than NSW and South Australia combined. A significant proportion of these were ex-convicts, but it is probable that many more crossed the Bass Strait. The majority of contemporary observers considered them as a severe threat to law and order on the goldfields and Geoffrey Serle suggested Vandemonians were largely responsible for the increase in recorded crime.  Efforts to stem their flow led to the Convicts Prevention Act of 1852. Criticised by some at the time as ‘illiberal’ and ‘arbitrary’, this legislation stopped convicts with conditional pardons from landing in Victoria. The fear and hatred these new immigrants inspired also helped generate support for the anti-transportation movement, headed by the Australasian League that successfully lobbied for the end of transportation to VDL by the end of 1852.
The population of Australia doubled in less than ten years because of gold. Over half a million people sailed from the United Kingdom during the 1850s, more than went to the United States. Over half paid their own fares and in Victoria seventy per cent were unassisted. A high proportion of men who came to Australia were educated, had professional experience and good connections at home. Peter Lalor, a civil engineer, came from Ireland to Melbourne in 1852 and Joseph Reed, a Cornish architect left England to look for gold in Victoria. Thousand crossed the Pacific from California and China. Some came from continental Europe such as Raffaello Carboni, a colonel under Garibaldi and since 1849 an exile in Rome, attracted to Melbourne by articles in the Illustrated London News.
Gold worth £3 million was mined in NSW and more than £9 million in Victoria by the end of 1852 long before most emigrants from England landed. Although gold continued to be found in significant quantities in Victoria during the 1850s, it gradually declined after 1860 and the number of individual diggers plummeted. Many of those who rushed to the goldfields turned to other occupations either to support the diggers or in areas that benefited from the growing wealth of the colony. Joseph Reed, for instance, began to design the grand civic buildings in Melbourne that gold made possible. Peter Lalor and Raffaello Carboni became involved in goldfields politics in the period leading up to Eureka Stockade. Francis Hare, cousin of the governor of Singapore who arrived in Melbourne in mid-1852 found that more reliable money was to be made in guarding the gold as it travelled to Melbourne than digging for it. Some emigrants were successful; most were not.
What image do we have of diggers? Though they only made up a significant proportion of Australia’s workforce only in the 1850s, nostalgia for the freedom of the digger developed rapidly once the early rushes was over. This was, in part, fostered by the emerging labour movement as a time when working men were, at least temporarily, free from the master-servant relationship. By the late nineteenth century, the 1850s increasingly looked like a lost heroic or ‘golden’ age during which, according to W. E. Adcock ‘our pioneers were founding a nation.’ The Australian writer Henry Lawson played a major part in communicating this nostalgia and his pessimism about the Australian present was developed in relation to a weakening collective memory of the gold decade.  His writing on the gold rush was largely a memory of greatness as well as a radical critique of the loss of independence since suffered by working men and a contrast between the confining anonymity of the city and the freedom of the diggings and the ‘mateship’ of the diggers. Gold-seeking was a liberating experience leading to independence and freedom that anti-democratic imperial officials sought to smother. This optimistic view of experience is problematic for two reasons. First, it seems to read back some of the nationalist concerns of the 1890s into the 1850s. Secondly, it gives a false impression of the way gold was actually understood in the 1850s when reactions across the political spectrum were either pessimistic or deeply ambivalent in the aftermath of the chaos of self-interest that gold seemed to have released. Conservative memory was not of freedom but disorder and the gold rushes were remembered as a vast fantasy that lured and trapped thousands of immigrants in an alien and hostile land.
This pessimistic view of gold-rush society was gradually erased from radical memory and the dominant nationalist historiography. Disruption was temporary and soon forgotten. What was remembered was the beneficial increase in population, urban growth and civic pride, self-government and effective nation building. Gold became the keystone on which Australia’s new society was based and mainstream history has in general celebrated the achievements of the gold years. The self-seeking acquisitiveness of the diggers was refashioned as initiative and enterprise and their creative energy was favourably compared to the attitudes of convicts and assisted immigrants. It was on them that the nation was built. This view was strongly expressed in the 1930s with almost eugenic pride when Australians were concerned about the quality of their racial stock
...the diggers as their Pilgrim Fathers, the first authentic Australians, the founders of their self-respecting, independent, strenuous national life, the fathers of their soldiers....[the legend] does not distort the facts. 
White Australia was the central defence of national identity and this, in Hancock’s view, justified the restriction of non-Europeans.
What [Australians] fear is not physical conquest by another race, but rather the internal decomposition and degradation of their own civilisation. They have gloried in their inheritance of free institutions, in their right to govern themselves and freely make their own destiny.
This view was reiterated in the Australian volume of the Cambridge History of the British Empire, originally published in 1933 and W.P. Morrell concluded
...tens of thousands of men took part, and though many faltered or fell by the wayside, the best of them evolved a new type of self-reliant character, a new free, careless social life. 
The only qualification to this general approval of the nation-building capacity of gold was that the large number of immigrants swamped the Australian-born population and this delayed the rise of nationalism to the end of the century. Goodman argued that
The realisation that Australian nationalism played little part in the consciousness of the 1850s had not prevented a popular reading of the gold rushes into the story of national development. 
Despite this nationalist historiography using the past to justify the present, there is a modicum of truth in its view of the independent diggers. What it neglects is that their independence was interspersed with intervals of wage-labour as they exhausted their resources before striking gold. It also romanticises the nature of the egalitarianism present on the goldfields. There was certainly a sense of social equality that led to the jettisoning of social pretensions but it was a consequence of the particular conditions that existed as all scrambled for their own gold. Many contemporary observers commented that the goldfields were not place for people unwilling to put their physical energy into manual labour. Egalitarianism was a consequence of the nature of the labour market during the 1850 when workers could demand their own rates of pay. Gold refashioned Australia but the legacy of the diggers is greater in legend than in reality.
 Ibid, Robson, Lloyd, A History of Tasmania, Vol. 1, pp. 465-467.
 Serle, pp. 126-127.
 This is examined in ibid, Goodman, D., Gold seeking: Victoria and California in the 1850s, pp. 5-10, 20-24.
 Adcock, W. E., The Gold Rushes of the Fifties, (Cole), 1912, p. 11.
 Matthews, Brian, ‘Lawson, Henry (1867-1922)’, ADB, Vol. 10, pages 18-22 and Clark, M., In Search of Henry Lawson, (Melbourne University Press), 1978 provide valuable, if idiosyncratic, biographical material.
 Ibid, Goodman, D., Gold seeking: Victoria and California in the 1850s, pp. 10-11 and ibid, Lyons, Martyn, and Russell, Penny, (eds.), Australia’s History: themes & debates, pp. 24, 36, 54-55, 103-105 and 121.
 This can be seen especially in ibid, Handcock W. K., Australia, pp. 35-36.
 Ibid, Hancock, W. K., Australia, p. 80.
 Portus, G.V., ‘The Gold Discoveries’, in Scott, Ernest, (ed.), Cambridge History of the British Empire: volume VII (part I) Australia, (Cambridge University Press), 1933, reprinted with a new introduction, 1988, pp. 245-271.
 Morrell, W. P., The Gold Rushes, (A & C Black), 1941, p. 415.
 Ward, Russel, Australia since the coming of man, (Melbourne University Press), 1987, p. 117.
 Ibid, Goodman, D., Gold seeking: Victoria and California in the 1850s, p. 11.