General Philip Breedlove, Supreme Allied Commander Europe, recently gently but unmistakably reprimanded the US intelligence community for its “lack of ability to see into Russia, especially at the operational and tactical level.” While he acknowledged change was under way, even then he made it clear that this was very, very much a work in progress: “We’re gently turning the nose of this ship to get back to what we need to be looking at.” Is Russia befuddling US intelligence, and if so what should be done about it?

Outside the movies, intelligence rarely gets a good press. On screen, the perceptive analyst or gung-ho field agent gets that one scrap of world-changing intel, realizes what it means, and suddenly—typically, just in the nick of time—policy spins on the proverbial dime, and all is put to rights in time for the closing credits. In practice, it is never quite so neat and clear, and the intelligence community tends not only to have to juggle multiple possible interpretations and “best truths” but they are also just a few voices in the mighty and often discordant choir of government.

Precisely because information and timely warning is its core business, the intelligence agencies tend to get the blame when governments are caught by surprise. And, let’s be honest, Washington has been caught by surprise again and again when Moscow is concerned, from the seizure of Crimea to most recently the Russian deployment to Syria.

Needless to say, the spooks have rushed to their own defense and affirmed that they were on top of all these developments and briefed to that end. To an extent, this is entirely true, but not necessarily the whole truth. Modern intelligence products often cover a range of possibilities, but there is a world of difference between including something as a potential option and clearly identifying it as the likely one. Consider, for example, the real and evident confusion which reigned when the “little green men” were taking Crimea while Moscow flatly disavowed responsibility: were they local militias, were they mercenaries, were they soldiers working for maverick local commanders? The answer was the simplest one – that Moscow was lying – but the period of uncertainty allowed Russia’s special forces to seize the peninsula in a smooth fait accompli. This did not suggest a strong and confident grasp of the unfolding situation in Washington.

Director of National Intelligence Clapper’s response to the charge of intelligence failure is instructive.

In an interview, he said: “We tracked [the situation in Ukraine] pretty carefully and portrayed what the possibilities were and certainly portrayed the difficulties we’d have, because of the movements of Russian troops and provided anticipatory warning of their incursion into Crimea.”

Likewise, a CIA spokesman said: “Since the beginning of the political unrest in Ukraine, the CIA has regularly updated policymakers to ensure they have an accurate and timely picture of the unfolding crisis. These updates have included warnings of possible scenarios for a Russian military intervention in Ukraine. Any suggestion otherwise is flat wrong.”

That’s all well and good, but tracking stuff happening, potential difficulties, possible scenarios and the like do not represent clear and unambiguous predictions, and the “anticipatory warning” does seem to have been pretty much as things happened, not early enough to do anything potentially to forestall the invasion. (Although in fairness, probably nothing could have done so.) Certainly on the eve of the invasion, US intel sources were briefing The Daily Beast that “From an intelligence perspective we don’t have any reason to think it’s more than military exercises.”

Of course often the problem is that smart and shrewd insights from the intelligence community get lost in the political process. The making of foreign policy is, after all, an arena in which diplomats and lobbyists, op. ed. writers and lawyers, soldiers and senators, overseas allies and domestic sentiment all get to pitch in. Given how rarely the intel products really can speak with the absolute confidence any good lobbyist or ignoramus can muster, no wonder they can get drowned out by other voices.

But it’s not quite that simple. There does seem to be a genuine intelligence problem with Moscow.

In part, this is because the Russians are very, very good at counter-intelligence. Just as they managed to fly their bombers into Syria undetected with transponders off, hidden beneath a larger cargo plane, so too they kept their Crimean operation off the US intelligence radar. A military exercise masked the movement of troops; orders were transmitted on paper, to sidestep America’s extraordinary signals intelligence capabilities; soldiers were even instructed to keep their cellphones and radios off, again to prevent the leakage of radioelectronic indications. The Russians may not be able to match most American intel capacities, but they are aware of them and put considerable thought into working out how to minimize them.

This is exacerbated by the extraordinarily small, tight circle within which most policy and especially security policy is made. We do not even know for sure exactly whose advice Putin takes. My own suspicion is that neither Foreign Minister Lavrov nor Defense Minister Shoigu are in the innermost circle, and instead we have to look to figures such as Presidential Chief of Staff Sergei Ivanov, Security Council Secretary Nikolai Patrushev and Federal Security Service Director Alexander Bortnikov. These are close-lipped loyalists, hardly likely to make incautious comments in public, nor easy to bring under human or electronic surveillance.

Furthermore, given that this is a regime which expects the elite and masses alike to rally round whatever is the policy of the day, it feels no need to signal policy in advance, to float ideas to gauge their response, or do any of the other kind of systematic foreshadowing exercises that might otherwise give us meaningful clues.

Finally, it takes time to come to terms with real, human-level change. Today’s Putin is not the Putin of his first two presidencies, when outspoken nationalist rhetoric was tempered by a much more pragmatic approach. Whether because we are seeing the real Vladimir or, more likely, because like most authoritarian leaders he has over the time become more insulated from reality, more steeped in his own mythology, this is a different man, heading a different team, for a different national purpose.

So what can be done to help “turn the nose of the ship”?

Given that the issue is human more than technical, I suspect that – for all the many challenges that poses – a greater concentration on building up HUMINT assets in Russia is a must. This poses risks of all kinds, from the potential for further embarrassing incidents such as the 2013 Ryan Fogle “wig-gate” case, through to the actual risk to agents and handlers. But if we are in a war of sorts with Russia – and the Russians certainly seem to feel so – then this cannot be without some danger and cost.

Secondly, play the analysis. Just as with so many other tectonic shifts which seem to have caught the USA and the West by surprise, from the collapse of the USSR to the Arab Spring, there are often no magic documents, no secret communiqués that would have revealed the future. Instead, what was needed was and is now an analytic capacity that is at least as strong as the technical intelligence capacity developed. It’s all very well building a $1.7 billion NSA computer facility in Utah, or planning a $2-4 billion next-generation spy satellite constellation – arguably you’d get vastly more bang for buck spending half as much on the best analysts around and giving them access to the huge amounts of open source information available. Predicting Russia’s next move will come by sneaking into Putin’s head; all the spy satellites will show is what he has decided as it starts to happen.

Thirdly, this means there needs to be as much creativity as possible in the intelligence process. If one accepts Clapper’s assertion that Putin is “kind of winging it, day to day,” then this becomes all the more important. One key area is the interaction with outside experts and perspectives, something which certainly happens, but often only under complex (and expensive) cut-outs which may help security but slow and reduce the flow of information. Furthermore, it is harder to be sure that iconoclastic insights actually inform the intel process; just as the CIA’s Red Cell is an attempt to challenge the groupthink that so often emerges, there is the scope to treat the outside analytic community – from journalists to academics to random bloggers – more often as analytic partners rather than just a passive resource.

Finally, the US government needs to listen more to its spooks, but also demand more from them. Consider the disastrous “reset” which, inter alia, put great emphasis on cultivating seat-warmer-in-chief, President-for-Halflife Dmitri Medvedev, something that helped infuriate and alienate Putin. As I understand it, this very much came out of the White House and State, without meeting with great enthusiasm from the intel community. At present, the spooks may not be listened to much, but then again there is a certain comfort for them in that. Time for them to take a more central role, but also to be expected to abandon the defensive tendency to offer ranges of possibilities like a fan of cards and asking the policy makers to pick whichever one they choose.

Of course, there is a corollary. I honestly don’t know – only insiders can – but I get the sense that just as Putin’s spooks seem to be competing for favour by pandering to his paranoiac and persecutionist world view, there may be more of a touch of that in Washington, too. It doesn’t matter how good your case officers and analysts, if the final intelligence products are smoothed down to suit a political consensus. One of the perennial problems has been how to manage and maximize the value of US intelligence for the policy process, and it is hard to believe that this has yet been cracked.

The Russian challenge is a bit of a Potemkin one, not — for all some of the over-the-top rhetoric of an “existential threat” — one that perhaps is likely to be so serious for years to come. So why is this such a concern? Put it this way: don’t crack the intel/policy problem this time round, and the USA will be scrambling to do it with a newly-resurgent Iran tomorrow, or perhaps China the week after…

This article was originally posted in In Moscow’s Shadows.