Hollywood thrillers love conspiracy theories (the movie business largely consists of numberless ad hoc conspiracies, so participants enjoy dreaming up more permanent, all-encompassing ones), while academics do not. Academic Jeffrey M. Bale complains of this in:

Political paranoia v. political realism: on distinguishing between bogus conspiracy theories and genuine conspiratorial politics

Patterns of Prejudice 41.1 (2007): 45-60.

Very few notions nowadays generate as much intellectual resistance, hostility and derision within academic circles as a belief in the historical importance or efficacy of political conspiracies. Even when this belief is expressed in a very cautious manner, limited to specific and restricted contexts, supported by reliable evidence and hedged about with all sorts of qualifications, apparently it still manages to transcend the boundaries of acceptable discourse and to violate unspoken academic taboos. The idea that particular groups of people meet together secretly or in private to plan various courses of action, and that some of these plans actually exert a significant influence on particular historical developments, is typically rejected out of hand and assumed to be the figment of a paranoid imagination. 

Most academic researchers clearly prefer to ignore the implications of conspiratorial politics altogether rather than deal directly with such controversial matters. A number of complex cultural and historical factors contribute to this reflexive and unwarranted reaction, but it is perhaps most often the direct result of a simple failure to distinguish between ‘conspiracy theories’ in the strict sense of the term, which are essentially elaborate fables even though they may well be based on kernels of truth, and the activities of actual clandestine and covert political groups, which are a common feature of modern politics. For this and other reasons, serious research into genuine conspiratorial networks has at worst been suppressed, as a rule discouraged, and at best looked on with condescension by the academic community. An entire dimension of political history and contemporary politics has thus been consistently neglected. ... 

If certain parties were to say, for example, that a secret Masonic lodge in Italy had infiltrated all of the state’s security agencies and was involved in promoting or at least exploiting acts of neo-fascist terrorism in order to help condition the political system and strengthen its own influence in the corridors of government, most readers would probably assume that that they were joking or accuse them of having taken leave of their senses. Twenty-five years ago this author might have had the very same reaction. Nevertheless, although the above statement greatly oversimplifies a far more complex pattern of interaction between the public and private spheres, not to mention between visible political institutions (‘the overground’ or ‘the Establishment’) and covert political groups (‘the underground’), such a lodge did in fact exist. It was known as Loggia Massonica Propaganda Due (P2), was affiliated with the Grand Orient branch of Italian Freemasonry, and was headed by a former Fascist militiaman named Licio Gelli.

I can recall reading about the various scandalous Italian conspiracies in The Economist in the early Eighties. Most enlightening.

(By the way, beginning one minute after I put up this post about "Propaganda Due" I've been flooded with spam comments on my own site in Italian offering genuine Gucci and Prada merchandise. Coincidence? You be the judge.)

Likewise, if someone were to claim that an Afrikaner secret society founded in the early decades of this century had played a key role in promoting the system of apartheid in South Africa, and in the process helped to ensure the preservation of ultraconservative Afrikaner cultural values and Afrikaner political dominance until the early 1990s, some readers would undoubtedly believe that that person was exaggerating. Yet this organization also existed. It was known as the Afrikaner Broederbond (AB), and it formed a powerful ‘state within a state’ in that country by virtue, among other things, of its exercise of covert influence over elements of the security services.

I hadn't heard of that before.

Here's a theory I just made up: these kind of deep state conspiratorial organizations are related in some sense to the relatively early retirement ages for soldiers and cops. Lots of colonels and police lieutenants get pensioned off while still in vigorous middle age, so in some countries they are enthusiastic about continuing to toil with their former colleagues as planners and organizers (i.e., conspirators). 

Bale goes on to explain that academics pay very little attention to such matters, while investigative reporters love this kind of stuff. So, this knowledge exists in a twilight world that screenwriters go nuts over, but doesn't seem respectable to enlightened opinion.

This leads to academic histories missing big chunks of the story. For example, Project Ultra, the giant WWII British codebreaking effort at Bletchley Park, which employed 9,000 people at one site by war's end, remained virtually unknown until 1974 (although traces of the story began appearing in the press from 1967 onward). I can recall it being a very big deal when it was revealed 29 years after the war ended. Was this hubbub due to the Watergate Era's hunger for conspiracy theories? Probably, but I wouldn't be surprised if journalists hadn't been telling each other about it in smoky bars for 29 years too, so they were well past primed to jump in when the official shackles were lifted in 1974.

In hindsight, I've noticed that Admiral Samuel Eliot Morison's official history of the Navy in WWII from the 1950s and 1960s contains a passage where he more or less taunts the reader to guess that the Allies had inside information on what the German U-boats were up to, but apparently nobody much noticed at the time.

Similarly, if you read newspapers and magazines closely, the whole NSA spying thing (which is a continuation of sorts of Ultra) was old news. Back in the day, the president of France was always denouncing the "Anglo-Saxon powers" and their perfidious Echelon for listening in on his phone calls.