Thousands of documents detailing some of the most shameful acts and crimes committed during the final years of the British empire were systematically destroyed to prevent them falling into the hands of post-independence governments, an official review has concluded.