Letters circulated to then prime minister Harold Macmillan recommended nuclear force as the only real alternative to abandoning the territory in the event of an attack by the neighbouring Chinese.
British officials discussed how to ensure that Beijing understood any attack would be met by the United States dropping nuclear bombs on China.
At the same time, the plan needed to avoid laying Hong Kong open to the claim that it was becoming a military outpost of the United States.
Details of the discussions emerged from records of Macmillan's office between 1957 to 1961, which were made public by the National Archives.
The suggested nuclear strategy followed communications on how best to strengthen Hong Kong's defences amid growing uncertainty about the intentions of its communist neighbour.
Hong Kong was particularly vulnerable, not least because its water and food supplies from China could be cut off at any time.
On February 22 1961, then foreign secretary Sir Alec Douglas-Home wrote a letter marked "top secret" to defence minister Harold Watkinson and the prime minister.
In it, he said: "It must be fully obvious to the Americans that Hong Kong is indefensible by conventional means and that in the event of a Chinese attack, nuclear strikes against China would be the only alternative to complete abandonment of the colony.
"In these circumstances it is perhaps not so much formal staff talks with the Americans that we need so much as an informal exchange of views involving a discussion of the use of nuclear strikes.
"I need hardly say, however, that I agree entirely with your view that while we should encourage the Chinese to believe that an attack on Hong Kong would involve nuclear retaliation, we must avoid anything that would allow the Chinese to claim that the colony is a military outpost of the Unites States."
Secret meetings with US officials took place in Hawaii and it was advised in 1961 that further talks should be held on board a US naval carrier, which frequently visited Hong Kong.