The extent of government data mining of communications

[gumboot]

How significantly different is this from ECHELON, a program that the NSA reportedly used during the Clinton years?

Completely different. Echelon involves the monitoring of international satellite and radio traffic. That's a fraction of a percent of the amount of traffic skeptigirl is talking about.

There's no technically feasible way to monitor this volume of traffic.

 

[gumboot]

I already clarified this. You data mine all traffic with the basic program, you single out less for more detailed data mining, then you use the voice recognition data mining on the smaller subset and actually wiretap the smallest subset.

Um... this makes absolutely no sense at all. I don't think you have thought this through.

If the raw data is live audio (say a phone call) you only have two options for your initial data mining; either you capture everything or you run real-time speech recognition software.

You seem to be arguing that you would only use speech recognition on a smaller pre-mined subset. How exactly would you achieve that subset without speech recognition?

In addition, while speech recognition can give reasonable real-time performance in optimum controlled conditions, this would not include probably 90+% of phone calls. Once you take into account background noise, accents, jargon/dialects, language changes, interference and microphone quality (just to name a few) you're going to have failed scanning on the overwhelming majority of calls.

 

[Skeptic Ginger]

I bolded part you forgot!Internet is not simple network,where you can tap a single(multiple) central nodes and get entire traffic for network.

It is often forgotten that part of internet is routing and various interconnections.Another part is fragmentation.As soon as two fragments from same IP packet will travel different routes,there is problem for surveillance.

And third even if there is simple screening it will take big time or wil miss target.and enocded/encrypted transfer is more and more used/popular.It is quite nice chunk of traffic for analysis.(You have to store entire encoded/encrypted traffic and it is large)

One doesn't need both ends of a communication line to tap one side. There was the article on the satellite transmission capture in the other thread. There was a reference in the other thread to the joint cooperation of a number of other countries. There was a discussion of the one telecom, Qwest, who opted out but I believe opted in after the 2007 revision of the Patriot Act. And there was the evidence of the equipment installation which intercepted all Internet transactions at various interception points.


Where is the hole in the coverage you refer to?

 

[Skeptic Ginger]

Um... this makes absolutely no sense at all. I don't think you have thought this through.

If the raw data is live audio (say a phone call) you only have two options for your initial data mining; either you capture everything or you run real-time speech recognition software.

You seem to be arguing that you would only use speech recognition on a smaller pre-mined subset. How exactly would you achieve that subset without speech recognition?

In addition, while speech recognition can give reasonable real-time performance in optimum controlled conditions, this would not include probably 90+% of phone calls. Once you take into account background noise, accents, jargon/dialects, language changes, interference and microphone quality (just to name a few) you're going to have failed scanning on the overwhelming majority of calls.

The information on the capability of the voice recognition software said in 2005 it had an 85% accuracy. We don't really know the capability of the software the NAS is currently using.

The links also described the method of screening and triaging calls to wiretap. The testimony by the whistle blower was that vast amounts of call data was being mined. I posted a link to a ppt sales presentation describing pattern recognition capable software that could detect "small anti-social groups".

Draw any conclusions from this you want to. It's the evidence that matters.

 

[mortimer]

It isn't a theory. It is an evidence based political discussion. You are welcome to point out anything that is not supported by evidence.

This is absolutely a Conspiracy Theory.

Your theory is

the Bush administration was data mining all communications. That presumes all calls are basically screened, those flagged are further screened, and so on down to those which are actually wiretapped and transcribed by a person.

You have provided zero evidence that all calls were basically screened. It wouldn't even be technically feasible at this point, but even if it were doesn't mean it happened.

 

[Cobalt]

The information on the capability of the voice recognition software said in 2005 it had an 85% accuracy. We don't really know the capability of the software the NAS is currently using.

Which, of course, means it's completely flawless. Bush is bad!

The links also described the method of screening and triaging calls to wiretap. The testimony by the whistle blower was that vast amounts of call data was being mined. I posted a link to a ppt sales presentation describing pattern recognition capable software that could detect "small anti-social groups".

Draw any conclusions from this you want to. It's the evidence that matters.

I've long since concluded you see what you want to.

You've not shown evidence to support anything, really. Prove it was possible to monitor all calls, then prove that they did, mmmkay?

 

[Klimax]

One doesn't need both ends of a communication line to tap one side. There was the article on the satellite transmission capture in the other thread. There was a reference in the other thread to the joint cooperation of a number of other countries. There was a discussion of the one telecom, Qwest, who opted out but I believe opted in after the 2007 revision of the Patriot Act. And there was the evidence of the equipment installation which intercepted all Internet transactions at various interception points.


Where is the hole in the coverage you refer to?

How can inteligent person miss point of my post,is explainable only in Politics.
No matter how many taps are there,it will not solve major problems.

It is often forgotten that part of internet is routing and various interconnections.Another part is fragmentation.As soon as two fragments from same IP packet will travel different routes,there is problem for surveillance.

And third even if there is simple screening it will take big time or will miss target.and enocded/encrypted transfer is more and more used/popular.It is quite nice chunk of traffic for analysis.(You have to store entire encoded/encrypted traffic and it is large)

I bolded(and spell-checked) ignored part.
Until evidence that these has been solved without half of Google's total storage and super link to it from all taps and/or at least practical solution,entire thread is about CONSPIRACY THEORY!

(peer reviewed journals with concrete article would be good.)

 

[Skeptic Ginger]

I posted the evidence in the OP. Just as some people think the evidence supports Bush purposefully lied about the facts on Saddam's WMDs to gain public approval to invade Iraq and others think Bush merely made an error assessing the facts, we assess the evidence differently. "Wanting" to see something is not what underlies the different conclusions. Underlying premises about what the Bush administration was motivated to do and what they were capable of is where the different conclusions come from.

 

[TragicMonkey]

Please. Any government capable of COINTELPRO in the past wouldn't be shy about mere eavesdropping in the present. What, did everyone suddenly reform and become decent people in the time between then and now? Human nature doesn't change.

I just hope they confine their crapitude to spying and don't go around doing human experimentation again.

 

[gumboot]

The information on the capability of the voice recognition software said in 2005 it had an 85% accuracy.

It's called speech recognition, not voice recognition (voice recognition is infinitely less accurate, and functionally speaking is science fiction). Speech recognition has impressive real-time levels of accuracy - in optimal controlled conditions. But we're talking about monitoring calls that are about 99% not optimal controlled conditions.

The links also described the method of screening and triaging calls to wiretap. The testimony by the whistle blower was that vast amounts of call data was being mined.

Your "evidence" is nothing more than blind speculation:

It's becoming more and more clear that some new technological capability is at the heart of the NSA wiretapping scandal. I think we can piece together what that capability is.

I'm sorry, but some guy speculating on an internet blog is not evidence, not by a long shot.

I posted a link to a ppt sales presentation describing pattern recognition capable software that could detect "small anti-social groups".

That's post-event analysis of call logs, not real-time monitoring. The difference is substantial.

Draw any conclusions from this you want to. It's the evidence that matters.

The problem is you haven't actually presented any evidence. All you have a speculation and paranoia.

 

[Klimax]

I posted the evidence in the OP. Just as some people think the evidence supports Bush purposefully lied about the facts on Saddam's WMDs to gain public approval to invade Iraq and others think Bush merely made an error assessing the facts, we assess the evidence differently. "Wanting" to see something is not what underlies the different conclusions. Underlying premises about what the Bush administration was motivated to do and what they were capable of is where the different conclusions come from.

Sorry.Not evidence.And certainly not proof I ask about!

So far fail!

 

[INRM]

Skeptigirl,

How does the software determine if the group is anti-social? Does it have the potential of labling non antisocial groups as antisocial?

 

[jj]

It's called speech recognition, not voice recognition (voice recognition is infinitely less accurate, and functionally speaking is science fiction).

Um, actually, there was pretty good speaker recognition (is that what you mean by "voice recognition"???) working at Bell Labs in the late 1970's.

Both speech recognition and speaker recognition are for the general case very difficult problems, but word spotting, as well as spotting certain kinds of phonemes, is much more advanced, and pretty easily done for the case where it only serves to occasionally get a human into the loop.

 

[jj]

Skeptigirl,

How does the software determine if the group is anti-social? Does it have the potential of labling non antisocial groups as antisocial?

What part of "word spotting" is hard to understand?

I have no idea if any such thing is going on, but the technology is NOT over the top, and given the presence of dsp in every step of the phone chain, isn't even that difficult to do.

 

[jj]

The problem is you haven't actually presented any evidence. All you have a speculation and paranoia.

I suggest you pick up the proceedings of the IEEE Signal Processing society for the last 20 years.

There's evidence that you CAN do it.

 

[gumboot]

Skeptigirl,

How does the software determine if the group is anti-social? Does it have the potential of labling non antisocial groups as antisocial?

She's mixing links and not really paying attention to what she's talking about. The "anti-social groups" thing monitors call records after the fact and uses them to scan for specific call behaviour - such as a small group of people only ever calling each other and no one else.

 

[gumboot]

Um, actually, there was pretty good speaker recognition (is that what you mean by "voice recognition"???) working at Bell Labs in the late 1970's.

Both speech recognition and speaker recognition are for the general case very difficult problems, but word spotting, as well as spotting certain kinds of phonemes, is much more advanced, and pretty easily done for the case where it only serves to occasionally get a human into the loop.

Speaker verification is fairly straight forward, but speaker identification is another matter entirely, and while, again, it works well in controlled conditions, it does not work well in the uncontrolled conditions of telephone calls.

(Not to mention you have to have a pre-existing voice map to do it)

Seriously... a human is a vastly superior speaker identifier and speech identifier, yet how often do you have bad connections when you can't fully understand what is being said on the phone?

Ambient noise is a major hurdle for any sort of voice/speech monitoring software. As more and more phone calls are made on cellphones, the problem only becomes more severe.

The NSA probably could do the sorts of things being talked about here. It's theoretically possible. But the resources and the personnel would be hugely beyond what the NSA has, and would be impossible to hide due to the sheer scale (think power consumption, personnel, bandwidth, etc).

People think you can get around this with computers, but these still need people to maintain them and service them, and eventually you need humans to come in and filter the enormous number of false positives (because believe me, there would be millions of them).

 

[jj]

Speaker verification is fairly straight forward, but speaker identification is another matter entirely, and while, again, it works well in controlled conditions, it does not work well in the uncontrolled conditions of telephone calls.

(Not to mention you have to have a pre-existing voice map to do it)

Say like a particular known terrorist's voice?

And you can recognize some kinds of phonemes, too, yes? Without having any pre-existing template. Yep. consider the value of that.

Ambient noise is a major hurdle for any sort of voice/speech monitoring software. As more and more phone calls are made on cellphones, the problem only becomes more severe.

Not relevant. You don't have to catch 'every instance, all you have to do is flag a particular number and if it pans out, then you have a human involved.

People think you can get around this with computers, but these still need people to maintain them and service them, and eventually you need humans to come in and filter the enormous number of false positives (because believe me, there would be millions of them).

Actually, you can screen the millions of false positives with a second stage of automation, I'm fairly sure, using more sophisticated means.

Again, I don't know a thing about any system in place, or not in place, all I know is the state of the technology, at least until 2002, and in 2002 it was good enough to do some kind of thing like this if people wanted to spend enough bucks.

 

[Kevin_Lowe]

She's mixing links and not really paying attention to what she's talking about. The "anti-social groups" thing monitors call records after the fact and uses them to scan for specific call behaviour - such as a small group of people only ever calling each other and no one else.

It seems to me you're bent on attacking a straw man here. All you are really saying is "This straw man I made up is technically impossible. Therefore I don't have to discuss what Skeptigirl actually meant, let alone what her links said".

What's the technical difficulty in using stored records of who called who and when to flag people of interest, storing and electronically sifting a tiny subset of all the calls made per day for specific sounds (like "Plame" or "Guantanamo") to refine their list of interesting people, and actually tapping the calls of some of those people?

It wouldn't have to be perfect for a politician to want to try it.

 

[Klimax]

It seems to me you're bent on attacking a straw man here. All you are really saying is "This straw man I made up is technically impossible. Therefore I don't have to discuss what Skeptigirl actually meant, let alone what her links said".

What's the technical difficulty in using stored records of who called who and when to flag people of interest, storing and electronically sifting a tiny subset of all the calls made per day for specific sounds (like "Plame" or "Guantanamo") to refine their list of interesting people, and actually tapping the calls of some of those people?

It wouldn't have to be perfect for a politician to want to try it.

Then you are going to miss them.And we are back at square one.