| Author |
Message |
nntp chip
Guest
|
 Posted:
Wed Jan 12, 2005 3:31 pm Post subject:
Extract data from packet dump (pcap) |
|
|
Hello all,
I have some data recorded with tcpdump that needs to be reassembled. The
idea is to filter out what is relevant and then glue together just the
data-part of every packet into a file.
Any ideas?
TIA
--- |
|
| Back to top |
|
 |
Mirko Parthey
Guest
|
| Posted:
Wed Jan 12, 2005 3:55 pm Post subject:
Re: Extract data from packet dump (pcap) |
|
|
On Wed, 12 Jan 2005 15:58:21 +0100, nntp chip wrote:
| Quote: | :I have some data recorded with tcpdump that needs to be reassembled.
The :idea is to filter out what is relevant and then glue together just
the :data-part of every packet into a file.
My main reason for reassembling data is to find bad behaviour in our
networks. Most things can be filtered and read in ethereal or ngrep. But when
it comes to streaming media and transferred files it gets harder to tell
when something bad is happening. That data can be any of TCP or UDP.
Since I cannot do any programming myself I am searching for existing
tools that can read a pcap-file, do all the needed magic and then write
the payload to a file/stdout.
|
http://users.tpg.com.au/adsln4yb/chaosreader.html
I haven't tried it myself yet, but from the description it looks like it
might solve your problem.
Mirko |
|
| Back to top |
|
|
Walter Roberson
Guest
|
| Posted:
Wed Jan 12, 2005 4:30 pm Post subject:
Re: Extract data from packet dump (pcap) |
|
|
In article <pan.2005.01.12.10.31.50.964778@privacy.net>,
nntp chip <me@privacy.net> wrote:
:I have some data recorded with tcpdump that needs to be reassembled. The
:idea is to filter out what is relevant and then glue together just the
:data-part of every packet into a file.
With or without the possibility that the capture includes retransmissions?
If out of order data is detected, should the data be buffered or be
discarded? I've recently read that linux puts packet fragments out
in reverse order (last first), so some amount of packet reassembly
may have to be done. I guess when I framing the question I was thinking
of TCP ordering, but I notice now that you do not indicate TCP specifically.
If the data does -not- consist of nice unfragmented in-order post-
error recovery TCP packets, then I'd suggest grabbing the Linux
or FreeBSD IP static and hacking it to suit your needs. No point in
re-writing all the retransmission logic if you don't need to.
--
Oh, to be a Blobel! |
|
| Back to top |
|
|
nntp chip
Guest
|
| Posted:
Wed Jan 12, 2005 7:58 pm Post subject:
Re: Extract data from packet dump (pcap) |
|
|
On Wed, 12 Jan 2005 11:30:54 +0000, Walter Roberson wrote:
| Quote: | In article <pan.2005.01.12.10.31.50.964778@privacy.net>, nntp chip
me@privacy.net> wrote:
:I have some data recorded with tcpdump that needs to be reassembled.
The :idea is to filter out what is relevant and then glue together just
the :data-part of every packet into a file.
With or without the possibility that the capture includes
retransmissions? If out of order data is detected, should the data be
buffered or be discarded? I've recently read that linux puts packet
fragments out in reverse order (last first), so some amount of packet
reassembly may have to be done. I guess when I framing the question I
was thinking of TCP ordering, but I notice now that you do not indicate
TCP specifically.
If the data does -not- consist of nice unfragmented in-order post- error
recovery TCP packets, then I'd suggest grabbing the Linux or FreeBSD IP
static and hacking it to suit your needs. No point in re-writing all the
retransmission logic if you don't need to.
|
Just getting the raw data would sometimes be good enough.
My main reason for reassembling data is to find bad behaviour in our
networks. Most things can be filtered and read in ethereal or ngrep. But when
it comes to streaming media and transferred files it gets harder to tell
when something bad is happening. That data can be any of TCP or UDP.
Since I cannot do any programming myself I am searching for existing
tools that can read a pcap-file, do all the needed magic and then write
the payload to a file/stdout.
--- |
|
| Back to top |
|
|
Walter Roberson
Guest
|
| Posted:
Thu Jan 13, 2005 1:55 am Post subject:
Re: Extract data from packet dump (pcap) |
|
|
In article <pan.2005.01.12.14.58.21.31286@privacy.net>,
nntp chip <me@privacy.net> wrote:
:My main reason for reassembling data is to find bad behaviour in our
:networks. Most things can be filtered and read in ethereal or ngrep. But when
:it comes to streaming media and transferred files it gets harder to tell
:when something bad is happening. That data can be any of TCP or UDP.
Urrr, what is "bad" in this context? 'bad' in a network sense
usually requires looking at the exact packets, not at the reassembled
stream, as one is looking for oddities like packets that have
an ACK without an initial SYN, or packets for which the checksum
is wrong, or packets that have ip source routing specified.
If you are trying to detect people using P2P and so on, then you
should be using an IDS such as 'snort' that wants the original
packets.
If, on the other hand, what you are looking for is matters such
as people sending trade secrets to competitors, or people
sending personal email through a company machine, then you get into
a moras of legal issues. In Canada and the USA, you can often
get around those issues by having a written security policy that
is literally signed off on by everyone... which works for -new-
employees but can be a problem if an existing employee refuses to
sign (unless you are an "at will" employer for US legal purposes;
in that case, you just tell them outright that if they don't sign they
will be fired, but if you aren't "at will" then you'll find that
refusing to agree to an impose -change- in employment conditions is not
considered adequate grounds for justifiable termination.)
In the EU, the EU privacy database policies are, I gather, interpreted
different ways in different places. About a year ago or so, someone
from, ummm Finland I think it was, posted indicating that in his
country recording the content of employee data packets was a
violation of employee privacy rights under the national implimentation
of the EU database privacy directives. I do not have information
about whether any particular national implimentation has exceptions;
the implication of the posting was that it at least could not be
-routinely- done the way it often is in the USA.
--
Preposterous!! Where would all the calculators go?! |
|
| Back to top |
|
|
nntp chip
Guest
|
| Posted:
Fri Jan 14, 2005 6:42 pm Post subject:
Re: Extract data from packet dump (pcap) |
|
|
On Wed, 12 Jan 2005 15:55:48 +0000, Mirko Parthey wrote:
| Quote: | On Wed, 12 Jan 2005 15:58:21 +0100, nntp chip wrote:
:I have some data recorded with tcpdump that needs to be reassembled.
The :idea is to filter out what is relevant and then glue together
just the :data-part of every packet into a file.
My main reason for reassembling data is to find bad behaviour in our
networks. Most things can be filtered and read in ethereal or ngrep.
But when it comes to streaming media and transferred files it gets
harder to tell when something bad is happening. That data can be any
of TCP or UDP.
Since I cannot do any programming myself I am searching for existing
tools that can read a pcap-file, do all the needed magic and then
write the payload to a file/stdout.
http://users.tpg.com.au/adsln4yb/chaosreader.html
I haven't tried it myself yet, but from the description it looks like it
might solve your problem.
|
Sweet... Thank you very much.
Made some trial runs with known pcap-data that came out very nice. This
sure will be of good use around here.
--- |
|
| Back to top |
|
|
nntp chip
Guest
|
| Posted:
Fri Jan 14, 2005 8:07 pm Post subject:
Re: Extract data from packet dump (pcap) |
|
|
On Wed, 12 Jan 2005 20:55:00 +0000, Walter Roberson wrote:
| Quote: | In article <pan.2005.01.12.14.58.21.31286@privacy.net>, nntp chip
me@privacy.net> wrote:
:My main reason for reassembling data is to find bad behaviour in our
:networks. Most things can be filtered and read in ethereal or ngrep.
But when :it comes to streaming media and transferred files it gets
harder to tell :when something bad is happening. That data can be any of
TCP or UDP.
Urrr, what is "bad" in this context? 'bad' in a network sense usually
requires looking at the exact packets, not at the reassembled stream, as
one is looking for oddities like packets that have an ACK without an
initial SYN, or packets for which the checksum is wrong, or packets that
have ip source routing specified.
|
That is what ethereal and similar tools are good for. When strange and
unexplained things happen I usually fire up at least one tcpdump first. If
possible more than one to be able to view problems from different angles.
Those dump-files are nice to have if things need to be analyzed in greater
detail. Should it turn out they are not needed I simply erase them again.
| Quote: | If you are trying to detect people using P2P and so on, then you should
be using an IDS such as 'snort' that wants the original packets.
|
I should really take the time needed to setup IDS again. The old box is no
more :( Until then I watch for excessive traffic and unusual ports and
such things. It's not a big problem... yet.
| Quote: | If, on the other hand, what you are looking for is matters such as
people sending trade secrets to competitors, or people sending personal
email through a company machine, then you get into a moras of legal
issues. In Canada and the USA, you can often get around those issues by
having a written security policy that is literally signed off on by
everyone... which works for -new- employees but can be a problem if an
existing employee refuses to sign (unless you are an "at will" employer
for US legal purposes; in that case, you just tell them outright that if
they don't sign they will be fired, but if you aren't "at will" then
you'll find that refusing to agree to an impose -change- in employment
conditions is not considered adequate grounds for justifiable
termination.)
|
I check the networks of a municipality in Sweden. There are not really any
trade secrets since anyone can ask for public records. If you handle
records that are not public you must sign a contract of confidentiality. I
have done that.
The 'security policy' they have is simply telling employees that computers
and phones are for work only, but using common sense they are allowed to
call or surf for childcare, medical care, banking arrangements, car repair
shop and such things that are only open during the day. They are also told
that we can check things and will do just that if needed.
I know it would be illegal for me to sniff traffic for my own personal
use.
Hunting down people that surf pr0n, use p2p-applications or spend lots of
time in chatrooms and so on should be ok. My boss asked me to watch for
things like that, so if someone doesn't like that I guess he will have to
answer for it.
If anyone believe that I am totally wrong I would like to know what is
right and where I can read up on that.
| Quote: | In the EU, the EU privacy database policies are, I gather, interpreted
different ways in different places. About a year ago or so, someone
from, ummm Finland I think it was, posted indicating that in his country
recording the content of employee data packets was a violation of
employee privacy rights under the national implimentation of the EU
database privacy directives. I do not have information about whether any
particular national implimentation has exceptions; the implication of
the posting was that it at least could not be -routinely- done the way
it often is in the USA.
|
Can you still get in trouble in USA/Canada for troubleshooting networks or
is it looking at transferred data that is forbidden?
--- |
|
| Back to top |
|
|
James Knott
Guest
|
| Posted:
Sat Jan 15, 2005 4:06 am Post subject:
Re: Extract data from packet dump (pcap) |
|
|
nntp chip wrote:
| Quote: | If the data does -not- consist of nice unfragmented in-order post- error
recovery TCP packets, then I'd suggest grabbing the Linux or FreeBSD IP
static and hacking it to suit your needs. No point in re-writing all the
retransmission logic if you don't need to.
Just getting the raw data would sometimes be good enough.
My main reason for reassembling data is to find bad behaviour in our
networks. Most things can be filtered and read in ethereal or ngrep. But
when it comes to streaming media and transferred files it gets harder to
tell when something bad is happening. That data can be any of TCP or UDP.
Since I cannot do any programming myself I am searching for existing
tools that can read a pcap-file, do all the needed magic and then write
the payload to a file/stdout.
|
Have you looked at what ethereal can do? It can read various dump formats,
filter the data, then export to various formats and also print to text
file, etc. |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in