Joined: 28 Feb 2006
|Posted: Sat Mar 24, 2007 7:24 am Post subject: Using netcat and tar to quickly transfer files between boxes
Netcat or nc in short can be aptly described as one of those two letter command-line tools that have all of legendary UNIX magic and power.
nc however is a new program and does not share the age of well known programs like cat or dd. However its power and versatility make one think why no one came up with this before.
Past is past. Let us worry about the future now. The future is VoIP, multimedia streaming, video on demand...but we are digressing. We are discussing netcat here.
Interestingly netcat is going to help us in future as the best way to debug future network protocols. Peer to peer networking is often touted as the wave of the future Internet and I shall talk about an interesting anecdote of how invaluable netcat proved to me whilst testing my peer topeer NAT traversal library.
It is not very likely you have used it. So let us start with few simple exams before we get on to nontrivial ones.
$nc -l -p 1234
will set up a TCP server on your machine. You can fire up
$nc localhost 1234
on another terminal which will set up a TCP client. You can now start talking between the two. Whatever you type here appears there and vice versa. Simple, isn't it?
on another terminal which will set up a TCP client. You can now start. Moreover it has power. The sort of power that only deceptively simple tools can possess. Let me demonstrate.
$nc -l -u -p 1234
will setup a UDP server. And you can talk to it with
$nc -u localhost 1234
Now, you will argue I am sure that telnet can be used as TCP clients.But it can hardly compare with netcat in its power and versatility as a network tool. Nowadays of course nobody uses telnet. We all use ssh.
Anyway netcat is used along with ssh for HTTP proxy authentication. And netcat can serve as simple inetd replacement, a simple command-line browser and so on. My God! You mean to say that netcat is so powerful?
It just depends on the way you look at it. netcat has a very clear goal and agenda. Of that of doing one thing and doing it well. And you use itas a duct tape for connecting with other similar tools to achieve your goal.
You could use netcat as a simple UDP or TCP based server and spawn a shell command every time a client connects. netcat can be used a simple chat application or even for sniffing network packets. The possibilities are endless.
The time has come to tell you how I used netcat for testing my p2p NAT library.
I used an approach called "UDP hole punching" for making inbound connections to machines behind NAT. Some say "protected by" but I shall avoid using that term.
I had to figure out if my NAT traversal library was doing its job. So once my library created the appropriate NAT mapping, I had to test application traffic. I used netcat for that.
In fact you can transfer files very simply even binary ones.
$nc -l -p 5000 > /tmp/network-file
and connect to this host across the Internet like this.
$nc server.com 5000 < network-file
You can even develop a simple VoIP application using netcat as it simply sets up a network pipe which you can send and receive data transparently.
For that you need to use a nice like G.729 using speex and use netcat's UDP serving capability. Voice traffic is delay sensitive and not loss sensitive and hence UDP is what we care. If you are lucky you could even use shell commands like this.
$ mkfifo /tmp/inputvoice $ cat /dev/dsp > /tmp/inputvoice&$ nc -u 1700 < /tmp/inputvoice > /dev/dsp
On the other side, you do something similar and
$nc -u 1700 < /tmp/inputvoice >/dev/dsp
Of course, the above example is merely an example and not a practical one.
nc can be used for taking remote backups and you can combine it with ssh and cron to do really cool things.
Think of nc as extending the UNIX pipe concept across the network and a whole new world of possibilities open up.
Did I tell you that netcat can do port scanning and network exploration? That is what the man page is for!
Using netcat and tar to quickly transfer files between machines, aka tar pipe
So you have gigs of data to transfer between two machines over ethernet. A nice quick and dirty method is to use netcat and tar. This is by no means secure, but if you haven't got the time or desire to setup NFS, FTPd, Samba. Or wait hours for scp to do it's job then this can save you a lot of time.
Linux System using tar and netcat
On the receiving end do:
# netcat -l -p 7000 | tar x
And on the sending end do:
# tar cf - * | netcat otherhost 7000
Depending on your hardware you might want to use compression with tar. You'll probably find your hard disk drive is a bottleneck on a fast network. Compression is useless if you can't saturate your network and will probably just slow things down. tar cf - * is copy everything in the current working directory. And your files will be untared at the other end from where you started the listening netcat. It won't be obivous that the operation is finished so you'll need to check network activity or top. This is indeed quick and dirty.
OpenBSD using tar and nc
OpenBSD usually offers a much tighter system rather than the mess you have with Linux. OpenBSD has it's own verison of netcat just refered as nc. The tar command is less sloppy than what you find in Linux. So if you're using two OpenBSD machines then one would do the following.
On the receiving end do:
# nc -l 7000 | tar -xpf -
And on the sending end do:
# tar -cf - * | nc otherhost 7000