User-space file system for Conduant StreamStor
A read-only filesystem for the Conduant StreamStor disk recorders. These recorders are used in Mark5, one of the several types of astronomic and geodetic VLBI data storage computers.
This file system provides normal file access to the data stored on the recorder diskpack(s). It works in normal kernel 2.4 and kernel 2.6 on normal Linux distros. On a Mark5's the file system makes any 'disk2file' step obsolete, as you no longer have to copy scans out onto disk before you can access them. In addition scans become accessible to normal software, for FTP, Tsunami UDP, Grid-FTP, UDT, NFS or cluster export, data processing such as correlation, Matlab, and so on.
Of the Haystack recorders both Mark5A and Mark5B are supported. Haystack has another fuseMk5 that provides Mark5C support, it comes preinstalled on the Mark5C.
The file system is implemented via FUSE. FUSE is part of every usual Linux distro. It is a generic library and kernel module that makes writing and debugging file systems very easy. It is used by many other user-space file systems such as GridFS, sshfs, ZFS for Linux and iFuse.
Source code access
Currently no binaries are distributed, because the Conduant StreamStor library licensing scheme is not clear on binary redistribution. For this reason you have to compile fuseMk5 from the sources. Source tarballs can be downloaded from the file releases section on the project overview page. You can also view current devel code through ViewVC browse cvs or download the latest development version via anonymous CVS checkout:
cvs -d:pserver:anonymous@fusemk5a.cvs.sourceforge.net:/cvsroot/fusemk5a login # note: blank password cvs -z3 -d:pserver:anonymous@fusemk5a.cvs.sourceforge.net:/cvsroot/fusemk5a co -P fuseMk5A
Licensing is according to GNU LGPL. For version differences and new features please see the ChangeLog.
Current version: 1.0.5 was released 10/2010. Threaded version 1.0.6beta released 01/2011.
Compiling the source
Prerequisites:
1) Conduant StreamStor card, installed SDK with any version 7.xx/8.xx/9.xx
2) old g++ v3.3 compiler because the SDK libraries do not link with gcc v4.x
3) FUSE module and tools (aptitude install libfuse2 libfuse-dev fuse-utils)
FUSE for ancient Linux distros:
If you are unfortunate to have an extremely old RedHat or Debian Sarge
system, see README-SargePreparations.txt
or README-RedhatPreparations.txt.
It involves some more steps than just aptitude install, but in general, even here installing FUSE is relatively easy and fast to accomlish.
Compiling:
Once you have installed FUSE and have the StreamStor SDK, you can compile the actual fuseMk5 file
system as described in the README. Basically
before make, sudo make install you need to edit the Makefile and point it to your correct StreamStor SDK libraries.
Questions and issues you should send to the public forum so that everyone can benefit.
Features and usage
Mounts Mark5A, Mark5B diskpack contents into a directory. Scans can be accessed as normal Linux files. Mount options are --bank=[0|1] to select either the diskpack in Bank A or in Bank B (default is 0 for Bank A), --noreset to skip a streamstor card reset, --cachesize=[bytes] to specify a read cache size in bytes (default 16MB).
oper@Mark5-637% ./fuseMk5A --noreset /mnt/diskpack
fuseMk5A version 1.0.6
Bank selected : 0
Prefetch cache : 15990784
Checking that Mark5 software not running (TCP port 2620)...
Opening diskpack...
Selected bank 0. It has label 'FGI-0005/4000/1024Recorded'
.. in another console window:
oper@Mark5-637% ll /mnt/diskpack/euro89_cta*
-r--r--r-- 1 oper users 960411032 Dec 31 1969 /mnt/diskpack/euro89_cta26_246-2239
-r--r--r-- 1 oper users 2304363960 Dec 31 1969 /mnt/diskpack/euro89_cta26_246-2357
-r--r--r-- 1 oper users 2207595680 Dec 31 1969 /mnt/diskpack/euro89_cta26_247-0045
-r--r--r-- 1 oper users 2112310824 Dec 31 1969 /mnt/diskpack/euro89_cta26_247-0228
-r--r--r-- 1 oper users 2111945200 Dec 31 1969 /mnt/diskpack/euro89_cta26_247-0615
...
oper@Mark5-637% /usr/bin/time cat /mnt/diskpack/euro89_cta26_246-2239 > /dev/null
0.20user 0.63system 0:09.79elapsed 8%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (108major+12minor)pagefaults 0swaps
oper@Mark5-637% /usr/bin/time md5sum /mnt/diskpack/euro94_mh_191-1616
6c6b9e39759d5b83525d50f3c9900aba euro94_mh_191-1616
1.00user 0.48system 0:04.97elapsed 29%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (1major+198minor)pagefaults 0swaps
oper@Mark5-637% scp /mnt/diskpack/euro94_mh_191-1616 oper@abidal.metsahovi.fi:/raid/
...
oper@Mark5-637% fusermount -u /mnt/diskpack
To export the diskpack over NFS, please read NfsExport.
Typical transfer rate examples
With the threaded version 1.0.6beta, read-out speed reported by 'dd if=... of=/dev/null' on a Mark5B+ is around 160MB/s (1.3 Gbps), while the speed reported on the FUSE side is 340MB/s (2.7 Gbps). With the non-threaded version 1.0.5 the 'dd' speed is 130MB/s (1 Gbps). On an old 1997 Mark5A system with RedHat 9 kernel 2.4 the 'dd' speed of v1.0.3 is around 100MB/s (0.8 Gbps) whereas with a 2005 gaming motherboard setup it reached 170 MB/s (1.4 Gbps).
The included program 'maxspeed' may be used to check the raw dispack read througput. This throughput can not be attained by the fuseMk5 file system (which itself is very lightweight) because there are several FUSE/kernel/user interaction layers with associated context switches and some additional memcpy's.
With a userspace NFS server that exports the mounted diskpack, read rates of 600-1000 Mbps are possible over 1G LAN.
Known limitations and issues
- Read-only access, can't write to the diskpack nor erase scans. Haystack and Conduant are possibly going to provide write support in the future. Others can of course volunteer.
- Files do not have real timestamps because StreamStor does not use timestamps.
- StreamStor SDK versioning is somewhat confusing. Makefile LDFLAGS must be updated manually. For SDK9.x Beta, use -lwdapi921, for 9.x use -lwdapi910, for 7.x/8.x use -lwdapi801. Best to figure this out using 'locate libwdapi'...
- The StreamStor design allows only one application to use a diskpack bank at any time. This means you should stop 'Mark5A' before mounting the diskpack, and umount the diskpack before you start 'Mark5A'. The Mark5A program also resets the card so that it is unlikely it would be possible to read from one fuseMk5'ed bank while 'Mark5A' writes to the other.
- Beginning with v1.0.3, multiple files can be opened and read, which is good when using NFS. But the StreamStor has very poor seek performance, nowhere close to software RAID, so don't say you weren't warned... Try to avoid reading from more than two files simultaneously.
- You can export the fuseMk5A file system with sshfs or userspace NFS. You should export not as root but as the
user who mounted the diskpack. In /etc/fstab or in the nfs mount options use rsize=16384 or larger for best
performance. Verify that
cat /proc/sys/net/core/rmem_default is a few hundred kByte. - NFS clients are sometimes prone to freeze with no timeouts in situations when the NFS server disappears but the server computer still runs. E.g. with an /etc/init.d/nfs stop -terminated nfs server. The problem happens not just with fuse. Resolution unknown. Maybe 'nolock' option, maybe not.
- The fuse-utils 2.5.x/2.6.x contains a bug when you unmount any fuse file system (sshfs, obexfs, fuseMk5a) that
was mounted into the background. This happens with earlier fuseMk5A versions (<=1.0.3) started without '--verbose' or
with any (sshfs, ... etc) FUSE file systems mounted from /etc/fstab. Unmounting the file system appears successfull,
but in reality the file system provider process may still be running ('ps axuf|grep fuseMk5A'). Unfortunately a
'kill -9 <fusemk5a pid>' is not really effective. For this reason, unless you can upgrade to fuse-utils 2.7.1, you should start all fuse filesystems (sshfs and others) into the foreground.
