[GLLUG] TCP/IP protocol efficiency

[Jeremy Bowers]

Mike Szumlinski wrote:
> It is single frames of film datacined from a $500k Thompson Bones.  It 
> actually is going onto a 4Gb fiber shared storage solution that 
> Windows can't read because it is formatted XFS.  We tried ext3, but 
> the FS just couldn't handle the data speed (we sustain about 
> 320-350MB/sec with the datacine).  Basically once the files are 
> scanned in from film, they are fed to an NLE that is doing online 2K 
> film editing...so we are simply trying to move data from an XFS volume 
> that is really really fast to an NTFS volume that is really really 
> fast.  The big problem is that we can't read from XFS in a supported 
> way on the Windows side and we can't write to NTFS in a supported way 
> on the Linux side...so enter network layer to make things slower.
Oh, I had the impression the data was going cross country, but now I 
realize you meant you were admining cross-country.

You might consider something like:

http://compsoc.dur.ac.uk/~djw/tarpipe.html

since encryption should really not be an issue. If the port you use 
netcat with isn't publically accessible, there aren't any *signficant* 
security concerns, either. (By "significant" I mean that in order to get 
to the netcat tunnel, they've already compromised a machine in your 
network so you've got bigger problems.) You could lock it down more 
tightly if you need to with standard port securing tools.

Can't beat the speed or ease of that, most likely. Cygwin should have 
all the necessary tools for the Windows side. Since you're working with 
pipes, you can do pipe things like add "gzip" in there to see if it 
helps, or use "time" to benchmark things easily.