Collarspace Discussion Forums


Home  Login  Search 

RE: Is the Govt. "tapping" the Internet?


View related threads: (in this forum | in all forums)

Logged in as: Guest
 
All Forums >> [Community Discussions] >> Dungeon of Political and Religious Discussion >> RE: Is the Govt. "tapping" the Internet? Page: <<   < prev  1 [2] 3 4 5   next >   >>
Login
Message << Older Topic   Newer Topic >>
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 6:19:49 AM   
Edwynn


Posts: 4105
Joined: 10/26/2008
Status: offline
quote:

ORIGINAL: JeffBC

tl;dr
Yes

OK, so this has come up in a wide variety of the current surveillance threads and those who don't really know what they are talking about continue to make assertions about either the technical capability or the motivation of the government to do such a thing. Y



You mean, like people who don't even know that the NSA and MI6 and German ant French intelligence and the Mossad, ect. always have and always will monitor every bit of data that crosses borders?

You mean those people?



< Message edited by Edwynn -- 6/15/2013 6:21:16 AM >

(in reply to JeffBC)
Profile   Post #: 21
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 9:13:10 AM   
JeffBC


Posts: 5799
Joined: 2/12/2012
From: Canada
Status: offline
quote:

ORIGINAL: DomKen
But you can't identify a torrent that way until you build up the torrent file section from the individual packets. So you need yet more processing power that does nothing but try and put together torrent pieces out of the packet stream.

I'll ask about that.

quote:

Anyway you have way over estimated the amount of bittorrent traffic out there. It is only just over 3% of total worldwide bandwidth and less than 2% of US bandwidth.
http://researchcenter.paloaltonetworks.com/app-usage-risk-report-visualization/#sthash.7wgpBVWi.dpbs

I'll ask about that also. That'd be a rather surprising mistake given the person I got it from but, you know, mistakes happen.

quote:

What the NSA is doing is what the PRISM slides say they are doing.

I agree. The leak is real. The slides are real. They are doing those things. I just happen to think that the other leaks we've heard about are also credible and they all fit together and reinforce each other and make a nice, coherent picture. Insofar as them doing "what is legal" that must be a joke, right? As we speak we have Clapper dancing on the head of a semantic pin about PRISM. It's abundantly clear he ought to go to jail for perjury. So I guess I'm not all that optimistic about them worrying about legalities.



_____________________________

I'm a lover of "what is", not because I'm a spiritual person, but because it hurts when I argue with reality. -- Bryon Katie
"You're humbly arrogant" -- sunshinemiss
officially a member of the K Crowd

(in reply to DomKen)
Profile   Post #: 22
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 9:33:08 AM   
DomKen


Posts: 19457
Joined: 7/4/2004
From: Chicago, IL
Status: offline

quote:

ORIGINAL: JeffBC

quote:

ORIGINAL: DomKen
But you can't identify a torrent that way until you build up the torrent file section from the individual packets. So you need yet more processing power that does nothing but try and put together torrent pieces out of the packet stream.

I'll ask about that.

The pieces of a torrent do not correspond to ip packets. The hashtag is simply a big number that is attached to each piece so it would be part of one or more packets in the data section.

quote:

quote:

Anyway you have way over estimated the amount of bittorrent traffic out there. It is only just over 3% of total worldwide bandwidth and less than 2% of US bandwidth.
http://researchcenter.paloaltonetworks.com/app-usage-risk-report-visualization/#sthash.7wgpBVWi.dpbs

I'll ask about that also. That'd be a rather surprising mistake given the person I got it from but, you know, mistakes happen.

quote:

What the NSA is doing is what the PRISM slides say they are doing.

I agree. The leak is real. The slides are real. They are doing those things. I just happen to think that the other leaks we've heard about are also credible and they all fit together and reinforce each other and make a nice, coherent picture. Insofar as them doing "what is legal" that must be a joke, right? As we speak we have Clapper dancing on the head of a semantic pin about PRISM. It's abundantly clear he ought to go to jail for perjury. So I guess I'm not all that optimistic about them worrying about legalities.

Show me these leaks that they say the NSA is monitoring all internet traffic.

You may not like the FISA court, I don't, but the process of getting a warrant there is not apparently any different from any other law enforcement organization seeking a warrant for a wiretap. I assume these judges know the law and the people receiving these warrants know the law as well and would resist warrants not inside the law. If you've been paying attention back under W the use of National Security Letters and warrantless wiretaps were reigned in by the courts.

(in reply to JeffBC)
Profile   Post #: 23
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 9:51:32 AM   
JeffBC


Posts: 5799
Joined: 2/12/2012
From: Canada
Status: offline
quote:

ORIGINAL: DomKen
The pieces of a torrent do not correspond to ip packets. The hashtag is simply a big number that is attached to each piece so it would be part of one or more packets in the data section.

I understand and I'm confused also Ken. Near as I can tell you'd have to reassemble the stream in order to throw it away. That operation, of course, would need to be done on the receiving end. That's what intuitively makes sense to me (and seems very difficult). That's why I wanted to ask the expert. I want to see if he's got a more credible answer for that.

quote:

Show me these leaks that they say the NSA is monitoring all internet traffic.

Been there, done that, got the T-shirt. Obviously I need to put it all together into a coherent picture because the individual pieces just aren't snapping into place in your mind. I have a few questions now queued up for my next round of playing "stump the experts" with my friends.

quote:

If you've been paying attention back under W the use of National Security Letters and warrantless wiretaps were reigned in by the courts.

What makes you say that? Is it because the folks like Clapper told you? I'm more inclined to believe the leakers than the known liars. I also think (and might be wrong) that the phrase "wiretapping" has specific legal meaning and does not include anything traversing over the internet... say... VOIP traffic... even possibly landline phones which have been routed over the backbones for longhaul transmission. So they can listen to a whole lot and not run afoul of any wiretapping issues. That doesn't fill me with comfort. It's a bit of legal sophistry and nothing more.


_____________________________

I'm a lover of "what is", not because I'm a spiritual person, but because it hurts when I argue with reality. -- Bryon Katie
"You're humbly arrogant" -- sunshinemiss
officially a member of the K Crowd

(in reply to DomKen)
Profile   Post #: 24
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 9:54:33 AM   
DomKen


Posts: 19457
Joined: 7/4/2004
From: Chicago, IL
Status: offline

quote:

ORIGINAL: JeffBC
quote:

If you've been paying attention back under W the use of National Security Letters and warrantless wiretaps were reigned in by the courts.

What makes you say that? Is it because the folks like Clapper told you? I'm more inclined to believe the leakers than the known liars. I also think (and might be wrong) that the phrase "wiretapping" has specific legal meaning and does not include anything traversing over the internet... say... VOIP traffic... even possibly landline phones which have been routed over the backbones for longhaul transmission. So they can listen to a whole lot and not run afoul of any wiretapping issues. That doesn't fill me with comfort. It's a bit of legal sophistry and nothing more.


You can read the court rulings on these issues. they are publicly available. Judges do take a dim view of people not obeying their court orders.

(in reply to JeffBC)
Profile   Post #: 25
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 10:14:03 AM   
njlauren


Posts: 1577
Joined: 10/1/2011
Status: offline

quote:

ORIGINAL: JeffBC

quote:

ORIGINAL: njlauren
So to 'tap' the entire internet traffic would be very difficult, because the packets that make up the traffic go by varying routes. You could pick it up by tapping the source or by the receiving machine, but in between would be very difficult because the packets go by way of coxies barn.


Undisputed tapping of San Jose node
I'll ask tomorrow how the short-haul traffic is captured.

Even assuming the NSA could 'tap the entire internet'
Again, we need not assume that. The above diagram pretty clearly illustrates how trivially easy the intercept part would be (pending the answer I need for short-haul)

even assuming they had server farms with disks big enough to store it, they would have a devil of a time trying to process it, it would probably take them many years to process 1 days total internet traffic, and that is being generous.
I did a quick bit of googling. Throughput on sustained write is no problem. I'll need to do some math on the capacity part but let's remember that you don't need to build out your SAN entirely before you start. You just need to keep adding storage faster than about 125TB/s (250/2 - assuming 50% is garbage like the bittorrent and netflix examples)

I doubt the government is monitoring people's internet traffic like this, I suspect the NSA is doing data profiling on general data coming off phone records, and maybe even profiling certain characteristics of traffic, and then if they find something drilling down, otherwise they would not be able to do much of anything, there is just too much data.
Well of course. There's going to be scanners checking for this and that with an attempt to check as many high level characteristics as possible before raising a flag to "drill down". It's that "drill down" operation that concerns me. One possible flag is, "He's running against me in the next election". I also suspect that you have no big data experience either but honestly neither do I and neither do any of my acquaintances... I'm afraid our relevant skill sets are around the security, network engineering and legal. However, it hardly matters. For now if they can just archive it all it wont' be long before the required processing power is available if it isn't now. And it's not an all or nothing proposition.

I don't think the NSA or anyone has the big brother capabilities some are afraid of, but they have enough power with what they can do that we should be concerned.
Again, I agree. They don't have enough to create my magic crystal ball viewing machine quite yet. They're scrambling to build it as fast as they can. You are right to be concerned though given that all this time we've been discussing tapping the entire internet. Obviously they could do much with only a tiny fraction of that. Crap, facebook and google alone (back to PRISM) would tell much of my life story.



Jeff-

ATT is just one communications backbone and the simplified backbone diagrams like yours make it seem easy, because you could say "Okay,I'll have tap points on the nodes on the graph map above, and will see any packets that come through any point". Problem is, this is just one backbone, there are fiber optic networks galore out there that carry the Internet, so a packet could go from My machine, over an ATT backbone, pass through a router on another machine, get routed through Verizon fiber, could pass another router and get sent to Canada, through a machine there, to another backbone to California....it is why tapping into the ISP's is more logical, if they have a general view of things, and drill down, and figure out someone is sending from someone using Comcast in upstate NY, they can use the taps they have in comcast to monitor their traffic.

My point simply was that on the backbone itself, the kind of monitoring being mentioned is not really possible, but in terms of analyzing data patterns and drilling down, no problem. If someone for example was running against someone and had influence with the NSA, it would be no big deal to get dirt. First of all, the NSA can monitor cell phone conversations (though they are not supposed to monitor domestic ones, they can), and it would be pretty easy to figure out that Joe Smith is on Optimum Online in Long Island, and get data from their servers on e-mail and browsing and so forth, doesn't even take much drilling down, to be honest.

Where I do have a problem is suppose we get some Jesus freak who decides they don't like kinky sex, and basically has the NSA do a broad search on people hitting certain kinky web sites (like this one), then drills down, traces back messages from point of origination, figures out who we are, and uses that power to out people, make their lives miserable..the tools they are using to find 'suspicious' terroristic activity could be used to try and track down people they don't like, whether it is tea party groups and their members or civil rights groups. During the 60's and 70's the FBI targeted the ACLU , including illegal wiretaps and monitoring, because Hoover and his successor thought the ACLU was a communist front group *shrug*.


I think the real concern is who is watching the watchers....among other things, the judges on FISA, and congressmen, are not great watchdogs, because guess what, if someone wanted to, they could make sure that key people were compromised and would rubber stamp their outrageous behavior. Again, might seem like a Robert Ludlum novel, but ask yourself how J.Edgar Hoover stayed in office so long, with all the shit he pulled, and you have a pretty good idea of why I am concerned.

(in reply to JeffBC)
Profile   Post #: 26
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 10:27:30 AM   
njlauren


Posts: 1577
Joined: 10/1/2011
Status: offline

quote:

ORIGINAL: DomKen


quote:

ORIGINAL: JeffBC

quote:

ORIGINAL: DomKen
But you can't identify a torrent that way until you build up the torrent file section from the individual packets. So you need yet more processing power that does nothing but try and put together torrent pieces out of the packet stream.

I'll ask about that.

The pieces of a torrent do not correspond to ip packets. The hashtag is simply a big number that is attached to each piece so it would be part of one or more packets in the data section.

quote:

quote:

Anyway you have way over estimated the amount of bittorrent traffic out there. It is only just over 3% of total worldwide bandwidth and less than 2% of US bandwidth.
http://researchcenter.paloaltonetworks.com/app-usage-risk-report-visualization/#sthash.7wgpBVWi.dpbs

I'll ask about that also. That'd be a rather surprising mistake given the person I got it from but, you know, mistakes happen.

quote:

What the NSA is doing is what the PRISM slides say they are doing.

I agree. The leak is real. The slides are real. They are doing those things. I just happen to think that the other leaks we've heard about are also credible and they all fit together and reinforce each other and make a nice, coherent picture. Insofar as them doing "what is legal" that must be a joke, right? As we speak we have Clapper dancing on the head of a semantic pin about PRISM. It's abundantly clear he ought to go to jail for perjury. So I guess I'm not all that optimistic about them worrying about legalities.

Show me these leaks that they say the NSA is monitoring all internet traffic.

You may not like the FISA court, I don't, but the process of getting a warrant there is not apparently any different from any other law enforcement organization seeking a warrant for a wiretap. I assume these judges know the law and the people receiving these warrants know the law as well and would resist warrants not inside the law. If you've been paying attention back under W the use of National Security Letters and warrantless wiretaps were reigned in by the courts.


There are problems with the FISA court. First of all, with warrants under FISA jurisdiction, they basically can go ahead and wiretap without a warrant and then later on 'present' the reasons they did the action. A big problem with that is obvious, let's say I decide I want to spy on someone for political reasons, we do the dirty..and then later on, with maybe a month between action and going to the court, I can cook up all kinds of crap that 'monitors' detected potentially 'suspicious' activity from said person, maybe terrorism, so we tapped down, but guess what, we were wrong, we found nothing..do you think FISA is going to say "wait a minute, that person is politically active, I think it is coincidence" or look at the 'proof' they use to justify the warrant, and say "okay, appears you had reasons, even if it didn't work out". Unfortunately, while broad descriptions have been written about FISA, it is interesting that no information about the way it really works has been published, or at least that I have read. The other problem is unlike real courts, all this stuff happens in secret, there is no one who has the authority to look at FISA and what comes across later, and say "ya know, this stinks". In a real court, judge's actions on search warrants are reviewed, and if they have a broad pattern of issuing warrants too easily, they can be censured for doing so, as far as I know, FISA has no checks on it. Especially when in effect 'issuing warrants after the fact', there is a lot more wiggle room for chicanery, and then claiming 'national security', which anyone who has been around any length of time (anyone remember the Pentagon Papers), often involve political embarrassment and political coverups more than national security reasons.

I don't believe the NSA has some mythical powers or the ability to monitor all internet traffic in real time and snoop on us all real time, but with the capabilities the spooks have right now, even using mundane technology, they can do a lot of really bad things and not for national security. Among other things, I don't think the NSA just has hooks into a relatively few big ISP and internet companies, I am pretty certain they have hooks into most ISP's in this country that they can use to get data directly, without even requesting it, I have gotten this from people with background in computer security, networking and data capture and analysis, that in a couple of cases meant they actually crossed paths with the NSA, and they all basically have said don't assume the NSA is limited to a relatively few companies, that their reach is quite broad and they can basically if they want get data from any system in the US any time they want it, and that few companies would challenge them, for fear of reprisals. None of these people are paranoid tin hat types, they said this almost as matter of course, with a shrug, like to them it wasn't a big deal particularly, it was something they assumed...

(in reply to DomKen)
Profile   Post #: 27
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 10:35:00 AM   
DomKen


Posts: 19457
Joined: 7/4/2004
From: Chicago, IL
Status: offline
I'm sure they have hooked into most ISP routers and I really don't like it.

My point in all this is they are not and will not soon have the capability to actually monitor all net activity.

It does concern me that they could monitor specific people's activity or monitor a specific website to determine who uses it.

(in reply to njlauren)
Profile   Post #: 28
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 12:35:27 PM   
Aswad


Posts: 9374
Joined: 4/4/2007
Status: offline
~fr~

Between them, AT&T, Verizon and Sprint have over 80% of the market for voice and data carrier services in the USA.

Communication requires at least two endpoints. Naïvely, using the above figure, we get 64% of communication having both endpoints in this triad, and 32% of communication having one endpoint in this triad, and only 4% of communication having no endpoints in this triad. Hence, compromising that triad gives access to 96% of all communication, while most of the remaining 4% can be inferred by social model-aware traffic analysis, which doesn't yield access to all the content, but does yield- to a reasonable degree of accuracy (i.e. far better than policework)- the interesting part of the information: signalling, connectivity and positioning. Which, of course, is the bulk of the metadata anyway.

As for feasability, if we make a reasonable assumption of redundancy, and light compression, at current disk prices...

... the monthly cost of storage for all global internet and mobile traffic is comparable to the cost of Kim Kardashian's toilet.

It's trivial to do, but not cost effective for intelligence purposes. Though, if I were in charge of intelligence, I'm not sure I'd see any reason not to store everything (apart from the legality and ethics of it), since you never know when it might come in handy to go back and have a look at some older data, and the storage itself costs next to nothing, while major compression is a modest cost. The major cost, obviously, is the pipework, but thanks to modern network engineering practices, this isn't a problem, either, so long as you're willing to live with the possibility of losing data whenever there's a problem (i.e. you use the redundancy and headroom in the network when you can, and gracefully dial down to what you need the rest of the time).

Which is not to say they're doing that, just that feasability isn't the issue.

Of course, it's pretty easy to avoid if you're out to pull off another 9/11, but it'll get some of the low hanging fruit and encourage the rest to improve, weeding out the weakest and encouraging the mediocre to become good at what they do. Not that I'd like them to become good at what they do, but someone clearly must. AQ praises the A-man for the gift of competent agencies under incompetent leadership, no doubt.

Bonus points for driving down the cost of hardware and motivating cypherpunks to keep developing new solutions to old problems.

IWYW,
— Aswad.


_____________________________

"If God saw what any of us did that night, he didn't seem to mind.
From then on I knew: God doesn't make the world this way.
We do.
" -- Rorschack, Watchmen.


(in reply to JeffBC)
Profile   Post #: 29
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 1:39:19 PM   
DomKen


Posts: 19457
Joined: 7/4/2004
From: Chicago, IL
Status: offline
Yes, feasibility is the issue.

To be effective you'd have to use lossless compression which is relatively effective at text but is next to completely useless for video, still images and audio which makes up most of the net traffic.

The reason to store everything would be things like this
http://en.wikipedia.org/wiki/Steganography#Digital

So no the storage required is not cheap. It is both terrifically expensive and physically does not exist. We'd know if the NSA were diverting a significant chunk of worldwide data storage manufacturing which is at the minimum what they'd have to be doing to just try much less succeed at this.

(in reply to Aswad)
Profile   Post #: 30
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 7:09:49 PM   
littlewonder


Posts: 15659
Status: offline
quote:


That is utter nonsense. You are making wild ass claims not backed up by anything. You claim to know network engineering but you seem to think the NSA could split the entire net's packet stream and no one would notice.

Note that the Boston bomber went to at least one jihadi website forum and read the AQ magazine on how to make the pressure cooker bomb. If the NSA was really monitoring the entire net wouldn't a person in the US reading that article be a fairly major red flag?

Furthermore details of Yahoo's attempt to resist the NSA came out today and the NSA specifically was seeking only certain information on specific foreign users.
http://www.nytimes.com/2013/06/14/technology/secret-court-ruling-put-tech-companies-in-data-bind.html


I hate to say it but Ken is right. The NSA would be completely overwhelmed with packets and it would have definitely set off bells with hackers and others in the networking business if that was actually happening. It would have been extremely easy. I mean the idea of reading all of those packets from the net? That's just funny. I keep picturing the computers trying so hard to keep up that they explode.

_____________________________

Nothing has changed
Everything has changed

(in reply to DomKen)
Profile   Post #: 31
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 7:25:00 PM   
njlauren


Posts: 1577
Joined: 10/1/2011
Status: offline

quote:

ORIGINAL: DomKen

I'm sure they have hooked into most ISP routers and I really don't like it.

My point in all this is they are not and will not soon have the capability to actually monitor all net activity.

It does concern me that they could monitor specific people's activity or monitor a specific website to determine who uses it.

I totally agree with you, Ken, and what is worse, that capability is only going to increase, as computer power increases and more importantly, as the net becomes more and more integral in people's lives. I don't even think the NSA would want to monitor all net activity, no bang for the buck, the thing most people forget is that you don't need to get at everyone, you can blackmail key people, judges, congressman, you name it, people in swing positions, positions of power, and get the outcome you want. Not to mention they might not even have to blackmail anyone, they can simply make it well known everyone is vulnerable, which is a well known tactic of police states. The USSR didn't have to monitor all its citizens, it simply had to make public displays of rounding people up, sending people to work camps, show trials, you name it, to keep them in line.

(in reply to DomKen)
Profile   Post #: 32
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 7:30:51 PM   
njlauren


Posts: 1577
Joined: 10/1/2011
Status: offline
Ken is correct, even with lossy compression if they could even trap 65% of the traffic over the US backbone, storing that would take huge investment in drives. The other thing to keep in mind is that a lot of traffic doesn't originate and terminate in the US, and traffic from Asia to Europe can go through the US, it all depends on how links in the internet are functioning, load, and so forth.

The Chinese pulled an experiment last year that shook some people up, they played tricks with network routers and ended up with I believe 25% of all internet traffic going through their systems, and yes, it is possible to do that. Among other things, in a cyberwar situation the chinese could use that kind of thing to shut off the internet in other countries, or compromise it, and if they can force traffic through their servers, they can monitor it and block it, too.....given how utterly insane the Chinese government is, it wouldn't surprise me if they are building a strategy of cyber warfare to use in blackmailing other countries into allowing them to censor the net, it wouldn't surprise me one bit.

(in reply to njlauren)
Profile   Post #: 33
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 9:01:47 PM   
DomKen


Posts: 19457
Joined: 7/4/2004
From: Chicago, IL
Status: offline
quote:

ORIGINAL: njlauren


quote:

ORIGINAL: DomKen

I'm sure they have hooked into most ISP routers and I really don't like it.

My point in all this is they are not and will not soon have the capability to actually monitor all net activity.

It does concern me that they could monitor specific people's activity or monitor a specific website to determine who uses it.

I totally agree with you, Ken, and what is worse, that capability is only going to increase, as computer power increases and more importantly, as the net becomes more and more integral in people's lives. I don't even think the NSA would want to monitor all net activity, no bang for the buck, the thing most people forget is that you don't need to get at everyone, you can blackmail key people, judges, congressman, you name it, people in swing positions, positions of power, and get the outcome you want. Not to mention they might not even have to blackmail anyone, they can simply make it well known everyone is vulnerable, which is a well known tactic of police states. The USSR didn't have to monitor all its citizens, it simply had to make public displays of rounding people up, sending people to work camps, show trials, you name it, to keep them in line.

While I think blackmail is a possibility I think it is mostly a decreasing possibility. The most likely blackmail material you could get about someone from their net usage is they look at porn. The fact is pretty much everyone who uses the net looks at some porn so it will become an increasingly ineffective blackmail material.
http://www.youtube.com/watch?v=GQxLs542bD8

< Message edited by DomKen -- 6/15/2013 9:02:35 PM >

(in reply to njlauren)
Profile   Post #: 34
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 9:15:54 PM   
TheHeretic


Posts: 19100
Joined: 3/25/2007
From: California, USA
Status: offline
FR

So on the one hand, people are talking about NSA data collection, and that is a good thing. On the other hand, a lot of them are talking total ignorance, and infantile mush.

Yes, they can. Yes, they do. Yes, they are building the next level in Utah, as we sit here scrolling through the nonsense.

The actual question regarding capability is how fast are they advancing in realtime electronic analysis, and the question for those who value individual liberty is how we draw bright lines around how such information might ever be used.


_____________________________

If you lose one sense, your other senses are enhanced.
That's why people with no sense of humor have such an inflated sense of self-importance.


(in reply to njlauren)
Profile   Post #: 35
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 9:22:52 PM   
TheHeretic


Posts: 19100
Joined: 3/25/2007
From: California, USA
Status: offline
I came upon this fun piece a few days ago. It may be usefully instructive, for those who'd like to engage, but haven't a clue beyond the universe of the afternoon news/chat show, filtered through their own bigotries.

Using Metadata to Find Paul Revere

quote:

I have been asked by my superiors to give a brief demonstration of the surprising effectiveness of even the simplest techniques of the newfangled Social Networke Analysis in the pursuit of those who would seek to undermine the liberty enjoyed by His Majesty’s subjects. This is in connection with the discussion of the role of “metadata” in certain recent events and the assurances of various respectable parties that the government was merely “sifting through this so-called metadata” and that the “information acquired does not include the content of any communications”. I will show how we can use this metadata to find key persons involved in terrorist groups operating within the Colonies at the present time. I shall also endeavour to show how these methods work in what might be called a relational manner.


_____________________________

If you lose one sense, your other senses are enhanced.
That's why people with no sense of humor have such an inflated sense of self-importance.


(in reply to TheHeretic)
Profile   Post #: 36
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 9:58:26 PM   
Aswad


Posts: 9374
Joined: 4/4/2007
Status: offline
quote:

ORIGINAL: njlauren

Ken is correct, even with lossy compression if they could even trap 65% of the traffic over the US backbone, storing that would take huge investment in drives.


Nonsense. The amount of traffic would be less than one exabyte per month. The redundancy in that traffic is immense, even above and beyond what you see in normal compressible data, due to the sheer number of users that are using the exact same data. Compression ratios upward of a thousand should be quite feasible. The drives required to cover that would, as noted, cost less than Kardashian's toilet. Multiply by twelve months per year, and you have a decent salary.

The problem isn't storing it. Or even getting it there. It just ain't cost effective, s'all. Hence, metadata instead.

Metadata gives you 90% of what you want, for a fraction of the effort.

That's what Norway and Sweden use for most online intelligence work, in addition to social data (e.g. mail, cell, phone, facebook, skype, forums and so forth) that are used to establish a baseline for normal activity and to flag any abnormal activity for review, and obviously to construct maps of the social networks of the population so that it's easier to keep a current picture of who's affiliated with what, and to flag new groups as they form so they can be reviewed manually for an idea of whether or not they're potentially interesting.

As for the people mentioning the possibility of giving these folks more stuff to process, the best thing one can do toward that end is to use more encrypted traffic to establish background noise. Currently, encrypted traffic sticks out like a sore thumb, because few people use it. As an example, I always encourage people to chat using OTR with Pidgin (on PC) or Adium (on Mac), as OTR replicates the properties of face to face communication in terms of privacy, and the client can run over your existing chat services, though it doesn't do video or voice.

IWYW,
— Aswad.


_____________________________

"If God saw what any of us did that night, he didn't seem to mind.
From then on I knew: God doesn't make the world this way.
We do.
" -- Rorschack, Watchmen.


(in reply to njlauren)
Profile   Post #: 37
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 10:08:59 PM   
DomKen


Posts: 19457
Joined: 7/4/2004
From: Chicago, IL
Status: offline
Again you cannot use lossy compression because the bad guys are known to embed messages in images so in order to get those messages you must use lossless compression which basically doesn't do anything to video, images or audio.

BTW a single Exabyte of data storage, assuming it is all stored on the relatively new and still expensive 10 terabyte drives, requires 100,000 hard drives which each cost about $1000. Now the NSA has a huge budget but can anyone really see them spending 1.2 billion dollars a year on hard drives? Can anyone show me a manufacturer with that much capacity?

(in reply to Aswad)
Profile   Post #: 38
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 10:31:56 PM   
Aswad


Posts: 9374
Joined: 4/4/2007
Status: offline
quote:

ORIGINAL: TheHeretic

I came upon this fun piece a few days ago. It may be usefully instructive, for those who'd like to engage, but haven't a clue beyond the universe of the afternoon news/chat show, filtered through their own bigotries.

Using Metadata to Find Paul Revere


Great introduction to simple social network analysis.

As I noted, SNA is the most heavily used method where I live, and one that relies only on the metadata, which are easily acquired, easily stored and easily analysed. The interesting connections in any such network (e.g. Paul Revere) can then, of course, be studied closer with a secret warrant.

The percentage of the population flagged by the Norwegian system as persons of potential interest (and thus flagged for the intelligence service to watch them more closely) is similar to the percentage of the US population for which PRISM partner Microsoft has approved requests for further information, according to their public statement.

In Norway, about one thousand individuals are scheduled for a "preventative interview" with the domestic intelligence service cum secret police, since the practice started in the aftermath of the Oslo/Utøya attacks; I note for reference that we have 5 million citizens in total, so this is one in five thousand to be "discouraged" in person.

I've no idea how the USA handles persons of potential interest. Hopefully in a civilized manner.

Anyway, yes, a great introduction, and your recommendation is enthusiastically seconded. For anyone that doesn't have any idea how or why metadata matters, this is an intuitively understandable introduction, on par with Malte Spitz' demo with regard to location information (which is part of the metadata collected, presumably), but covering a more relevant aspect of such analyses.

IWYW,
— Aswad.


_____________________________

"If God saw what any of us did that night, he didn't seem to mind.
From then on I knew: God doesn't make the world this way.
We do.
" -- Rorschack, Watchmen.


(in reply to TheHeretic)
Profile   Post #: 39
RE: Is the Govt. "tapping" the Internet? - 6/15/2013 10:41:17 PM   
TheHeretic


Posts: 19100
Joined: 3/25/2007
From: California, USA
Status: offline

quote:

ORIGINAL: Aswad

I've no idea how the USA handles persons of potential interest. Hopefully in a civilized manner.




They very helpfully assist them in building and planting "bombs" that don't go boom, when the button is pushed. See the attempted Christmas tree bombing in Portland Oregon, or the bridgebusting Occupists/anarchists in Ohio.

_____________________________

If you lose one sense, your other senses are enhanced.
That's why people with no sense of humor have such an inflated sense of self-importance.


(in reply to Aswad)
Profile   Post #: 40
Page:   <<   < prev  1 [2] 3 4 5   next >   >>
All Forums >> [Community Discussions] >> Dungeon of Political and Religious Discussion >> RE: Is the Govt. "tapping" the Internet? Page: <<   < prev  1 [2] 3 4 5   next >   >>
Jump to:





New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts




Collarchat.com © 2025
Terms of Service Privacy Policy Spam Policy

0.109