Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center...
-
Upload
avis-banks -
Category
Documents
-
view
214 -
download
0
Transcript of Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center...
![Page 1: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/1.jpg)
Hepix LAL April 2001
An alternative to ftp : bbftpAn alternative to ftp : bbftp
Gilles FarracheIn2p3 Computing Center
![Page 2: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/2.jpg)
2Hepix LAL April 2001 Gilles Farrache
HistoryHistory
In 1999, the members of the French Babar collaboration decided to import from SLAC to Lyon a big part of the SLAC objectivity database
First approach : via tapes Needs a lot of manpower Problems with cartridge compatibility Delay between data production at SLAC and their availability in
Lyon quite long
![Page 3: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/3.jpg)
3Hepix LAL April 2001 Gilles Farrache
HistoryHistory
Second approach : via the network using standard ftp
Security problems :Username and password flying unencryptedUsername and password present in .netrc file
Performances problems (Maximum bandwidth available at this time between
SLAC and Lyon was 6 Mbps) : Throughput performance on one ftp session : 80 Kbytes/s~Seven hours for one two-gigabytes file
![Page 4: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/4.jpg)
4Hepix LAL April 2001 Gilles Farrache
HistoryHistory
First idea to improve ftp performances : Start n concurrent ftp
• Total throughput reaches 3.5 Mbps But you still need no network outage for seven hours
Second idea to improve ftp performances : Cut the file in n parts Start n concurrent ftp (one for each part) Rebuild the file But you double the disk space usage at each endpoint
![Page 5: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/5.jpg)
5Hepix LAL April 2001 Gilles Farrache
GoalsGoals
Improve performances on single file transfer
Optimize disk space usage
Improve security
![Page 6: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/6.jpg)
6Hepix LAL April 2001 Gilles Farrache
Performance improvementsPerformance improvements
Use several TCP streams to transfer a file
Use large TCP windows (RFC 1323)
Compress the data on the fly (using gzip algorithm)
![Page 7: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/7.jpg)
7Hepix LAL April 2001 Gilles Farrache
Performance improvementsPerformance improvements
The client opens a control connection to the server
Exchange control information
Each side starts as many processes as requested, each process will send a slice of the file.
![Page 8: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/8.jpg)
8Hepix LAL April 2001 Gilles Farrache
Performance improvementsPerformance improvements
100 MB file
0
10
20
30
40
50
60
70
90
100
80
0
10
20
30
40
50
60
70
90
100
80
compress uncompress
bbftp client process bbftp server process
100 MB file
![Page 9: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/9.jpg)
9Hepix LAL April 2001 Gilles Farrache
Performance improvementsPerformance improvements
Compression plus parallel streams lead to throughput between 7 and 20 Mbps on the 6 Mbps link (often 12 Mbps with a standard Babar file)
But with higher speed links (34 Mbps), compression often reduces performance due to lack of CPU (perhaps we have to use another algorithm)
![Page 10: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/10.jpg)
10Hepix LAL April 2001 Gilles Farrache
Security improvementsSecurity improvements
Standard bbftp connection method Use RSA encryption methods to transfer username and password
SSH bbftp connection method (Tim Adye’s idea) Use ssh to remotely start the server
![Page 11: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/11.jpg)
11Hepix LAL April 2001 Gilles Farrache
Security improvementsSecurity improvements
Standard bbftp connection method
client server
connection
RSA key pair generationsend public key
Crypt user name and password using the public key
send encrypted data
Decrypt user name and password using the private key
OK or BAD
![Page 12: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/12.jpg)
12Hepix LAL April 2001 Gilles Farrache
Security improvementsSecurity improvements
Standard bbftp connection method without password
Start a bbftpcd client which will keep the username and password in memory
Start a bbftpc client which will talk with bbftpcd and will give it the command to execute
Limitation : bbftpcd and bbftpc have to run on the same computer
![Page 13: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/13.jpg)
13Hepix LAL April 2001 Gilles Farrache
Security improvementsSecurity improvements
SSH bbftp connection method
client
SSHD start the bbftpd process
Start a ssh client like ssh -x -l username “bbftpd -s” host
SSH client SSH daemon bbftpd process
Use the encrypted connection as control connection
![Page 14: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/14.jpg)
14Hepix LAL April 2001 Gilles Farrache
Security improvementsSecurity improvements
SSH bbftp connection method without password
If the sshd server side allows the RSA authentication, a connection without any password is possible
![Page 15: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/15.jpg)
15Hepix LAL April 2001 Gilles Farrache
BBFTP state of the artBBFTP state of the art
Current version 2.0.0 patch03 Possibility to play with :
• TCP window size• Read (Write) buffer size• Compression• Automatic directory creation• File mode• File access time • Number of streams• Remote and local umask• .bbftprc file• RFIO
![Page 16: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/16.jpg)
16Hepix LAL April 2001 Gilles Farrache
BBFTP state of the artBBFTP state of the art
Current commands :• cd• get• put• lcd• mkdir• mget• mput
![Page 17: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/17.jpg)
17Hepix LAL April 2001 Gilles Farrache
BBFTP state of the artBBFTP state of the art
Control file example :• 1 setnbstream 20 • 2 setremoteumask 022 • 3 setoption nocreatedir • 4 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1 • 5 setoption createdir • 6 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1 • 7 setnbstream 5 • 8 setrecvwinsize 1024 • 9 setoption gzip • 10 put /home/babar/jon/f2 /home/babar/bbrdist/newfiles/f2
![Page 18: Hepix LAL April 2001 An alternative to ftp : bbftp Gilles Farrache In2p3 Computing Center farrache@cc.in2p3.fr.](https://reader036.fdocuments.net/reader036/viewer/2022082713/5697bf7b1a28abf838c8364e/html5/thumbnails/18.jpg)
18Hepix LAL April 2001 Gilles Farrache
BBFTP state of the artBBFTP state of the art
BBFTP home page : http://ccweb.in2p3.fr/bbftp/ BBFTP discussion list : [email protected]
Archives on : http://ccmail.in2p3.fr/archives/bbftp-l.html Current experiments or sites using it :
PHENIX
ALICE (project)