User-space file system for Conduant StreamStor
A read-only filesystem for the Conduant StreamStor disk recorders. These recorders are used in Mark5, one of the several types of astronomic and geodetic VLBI data storage computers.
My file system layer provides normal file access to the data stored on the recorders diskpack(s). It works in default kernel 2.4 and kernel 2.6. On a Mark5's the file system makes any 'disk2file' step obsolete, as you no longer have to copy scans out onto normal disk or RAID before you can access them. In addition scans become accessible to normal software, for FTP, Tsunami UDP, NFS or cluster export, data processing such as correlation, and so on.
Of the Haystack recorders both Mark5A and Mark5B are supported. Haystack is working on another fuseMk5 for Mark5C support.
Source code access
Currently no binaries are distributed, because the Conduant StreamStor library licensing scheme is not clear on binary redistribution. For this reason you have to compile fuseMk5A from the sources. Source tarballs can be downloaded from the file releases section on the project overview page. You can also view current devel code through ViewVC browse cvs. You can also get the latest developemt version through anon CVS checkout:
cvs -d:pserver:anonymous@fusemk5a.cvs.sourceforge.net:/cvsroot/fusemk5a login cvs -z3 -d:pserver:anonymous@fusemk5a.cvs.sourceforge.net:/cvsroot/fusemk5a co -P fuseMk5A
Licensing is according to GNU LGPL with an added condition that if you used fuseMk5A you should give credit in the first three related presentations or publications. For version differences and new features please see the ChangeLog.
Current version: 1.0.4 was released 08/2009.
Compiling the source
Background info: FUSE itself is a generic library and kernel module (for more info, visit the fuse.sf.net site). File system providers like fuseMk5A, sshfs, obexfs etc use FUSE to provide the actual file system. So you need to install both the FUSE kernel modules, FUSE helper applications, and finally fuseMk5A.
Requirements: you need the Conduant StreamStor card and SDK 6.x to 9.x. The SDK licensing conditions regarding redistribution are unclear. For this reason no .deb/.rpm package with a precompiled fuseMk5A is provided on this page, sorry. The SDK is already found on your Mark5 so you have the required files to compile fuseMk5A. You have to use old venerable g++ v3.3 to compile because the SDK libraries do not link in new gcc v4.x
Compiling: The first step is to install FUSE, then get fuseMkA sources and compile them. Installing FUSE on a Haystack Mark5 has several paths. FUSE is already included in e.g. Ubuntu, Debian Etch or other distros with kernels 2.6.8 and newer. In old RedHat or Debian with kernel 2.4 you have to install FUSE manually. For Sarge with kernel 2.4 please follow the SargePreparations. In case of RedHat please follow RedhatPreparations. All non-Mark5 owners of a StreamStor card should use apt, YaST2 or other to install everything fuse'ish ('aptitude search fuse') that comes with their favorite Linux distro. Once you have FUSE you can compile the actual fuseMk5A file system as described in the readme.
Questions and issues you should send to the public forum so that everyone can benefit.
Features and usage
Mounts Mark5A, Mark5B diskpack contents into a directory. Scans can be accessed as normal Linux files. Mount options are --bank=[0|1] to select either the diskpack in Bank A or in Bank B (default is 0 for Bank A), --verbose to run fuseMk5A in the foreground with debug info output (this is recommended to workaround a FUSE 'unmount' bug), --cachesize=[bytes] to specify a read cache size in bytes (default 16MB). Supports 16-disk module VSNs in addition to the more common 8-disk module VSNs beginning with version 1.0.2.
oper@Mark5-637% ./fuseMk5A --verbose /mnt/diskpack
Bank selected : 0
Prefetch cache : 15990784
Opening diskpack...
Selected bank 0. It has label 'FGI-0005/4000/1024Recorded'
.. in another console window:
oper@Mark5-637% ll /mnt/diskpack/euro89_cta*
-r--r--r-- 1 oper users 960411032 Dec 31 1969 /mnt/diskpack/euro89_cta26_246-2239
-r--r--r-- 1 oper users 2304363960 Dec 31 1969 /mnt/diskpack/euro89_cta26_246-2357
-r--r--r-- 1 oper users 2207595680 Dec 31 1969 /mnt/diskpack/euro89_cta26_247-0045
-r--r--r-- 1 oper users 2112310824 Dec 31 1969 /mnt/diskpack/euro89_cta26_247-0228
-r--r--r-- 1 oper users 2111945200 Dec 31 1969 /mnt/diskpack/euro89_cta26_247-0615
...
oper@Mark5-637% /usr/bin/time cat /mnt/diskpack/euro89_cta26_246-2239 > /dev/null
0.20user 0.63system 0:09.79elapsed 8%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (108major+12minor)pagefaults 0swaps
oper@Mark5-637% /usr/bin/time md5sum /mnt/diskpack/euro94_mh_191-1616
6c6b9e39759d5b83525d50f3c9900aba euro94_mh_191-1616
1.00user 0.48system 0:04.97elapsed 29%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (1major+198minor)pagefaults 0swaps
oper@Mark5-637% scp /mnt/diskpack/euro94_mh_191-1616 oper@abidal.metsahovi.fi:/raid/
...
oper@Mark5-637% sudo umount /mnt/diskpack
To export the diskpack over NFS, please read NfsExport.
Typical transfer rate examples
With the current version read-out speeds are above 800 Mbps on an antiquated 1997 Mark5A (Redhat 9, kernel 2.4). Performance is about the same on the Mark5B. Better rates on newer motherboards. At Metsähovi we get ~1.6 Gbps on a 5A. The 1.6 Gbps seems to be about the maximum the StreamStor PCI-X card can do. For really slow computers restricted by memory copy speed, fuseMk5A throughput could be increased by improving the caching code and make it threaded.
About NFS, Bonn MPIfR reports 600 Mbps network read rates from a userspace-NFS -mounted fuseMk5A diskpack over 1G LAN. At Metsähovi we saw around 1.5 Gbps over 10GbE LAN.
Known limitations and issues
- Read-only access, can't write to the diskpack nor erase scans. Haystack and Conduant are possibly going to provide write support in the future. Others can of course volunteer.
- The StreamStor design allows only one application to use a diskpack bank at any time. This means you should stop 'Mark5A' before mounting the diskpack, and umount the diskpack before you start 'Mark5A'. The Mark5A program also resets the card so that it is unlikely it would be possible to read from one fuseMk5'ed bank while 'Mark5A' writes to the other.
- Only one file can be open and be read at a time in v1.0.2 and earlier. Beginning with v1.0.3, multiple files can be open and read, this is good with NFS. But the StreamStor has very poor seek performance, nowhere close to software RAID, so don't say you weren't warned... Try to avoid reading more than two files in parallel.
- You can export the fuseMk5A file system with, for example, sshfs or NFS. You should export not as root but
as the user who mounted the diskpack. In /etc/fstab or in the nfs mount options use rsize=16384 or larger for best
performance. Check that
cat /proc/sys/net/core/rmem_default is a few hundred kByte. - NFS clients are sometimes prone to freeze with no timeouts in situations when the NFS server disappears but the server computer still runs. E.g. with an init.d -terminated nfs server. The problem happens not just with fuse. Resolution unknown. Maybe 'nolock' option, maybe not.
- If you have StreamStor SDK 9.x you have to add -lwdapi910 to the LDFLAGS, or add -lwdapi801 in case of SDK 7.x.
- The fuse-utils 2.5.x/2.6.x contains a bug when you unmount any fuse
file system (sshfs, obexfs, fuseMk5a) are mounted into the background. This happens with
earlier fuseMk5A versions started without '--verbose' or FUSE file systems mounted from /etc/fstab.
Unmounting the file system appears successful, but in reality the file system provider
process may still be running ('ps axuf|grep fuseMk5A'). Unfortunately a
'kill -9 <fusemk5a pid>' is not really effective. For this reason, unless you can upgrade to fuse-utils 2.7.1, you should start all fuse filesystems (sshfs and others) into the foreground.
