--- Read this first
This version 2 of the libnfs API. It is not compatible with the earlier API.
All rpc_<protocol>_<function>_async() functions have been changed to
rpc_<protocol>_<dunction>_task().
The _task() functions return a struct rpc_pdu * which can later be used
to cancel an inflight command.

nfs_[]read and nfs_[p]write functions have new signatures.
Pay attention to any compiler warnings when compiling against this new API.

The symbol LIBNFS_API_V2 can be used to identify that the library uses the new
API.

This version of the library supports zero-copy read and write for NFS v3/4
---


LIBNFS is a client library for accessing NFS shares over a network.

LIBNFS offers three different APIs, for different use :
1, RAW : A fully async low level RPC library for NFS protocols
This API is described in include/libnfs-raw.h
it offers a fully async interface to raw XDR encoded blobs.
This API provides very flexible and precise control of the RPC issued.

examples/nfsclient-raw.c provides examples on how to use the raw API

2, NFS ASYNC : A fully asynchronous library for high level vfs functions
This API is described by the *_async() functions in include/libnfs.h.
This API provides a fully async access to posix vfs like functions such as
stat(), read(), ...

examples/nfsclient-async.c provides examples on how to use this API


3, NFS SYNC : A synchronous library for high level vfs functions
This API is described by the *_sync() functions in include/libnfs.h.
This API provides access to posix vfs like functions such as
stat(), read(), ...

examples/nfsclient-sync.c provides examples on how to use this API

NFSv4:
======
NFSv3 is the default but NFSv4 can be selected either by using the URL argument
version=4 or programatically calling nfs_set_version(nfs, NFS_V4) before
connecting to the server/share.

NFSv4.2 is selected with version=4.2 or nfs_set_version(nfs, NFS_V4_2). It
adds sparse file support: nfs_fallocate() for reserving, punching and zeroing
ranges, lseek() with SEEK_HOLE and SEEK_DATA, and the READ_PLUS and
WRITE_SAME operations. NFSv4.2 support is built by default and can be left
out with --disable-nfs4-2 (autoconf) or -DENABLE_NFS4_2=OFF (cmake).

NFSv4.1 is selected with version=4.1 or nfs_set_version(nfs, NFS_V4_1). It uses
the same sessions as NFSv4.2 but not the sparse file operations:
nfs_fallocate() and lseek() with SEEK_HOLE or SEEK_DATA fail with -ENOTSUP. It
is built with NFSv4.2, so the options above remove both and
nfs_set_version(nfs, NFS_V4_1) then fails at run time as well.

Applications compile against it behind LIBNFS_FEATURE_NFS4_2, which libnfs.h
defines wherever the dialect exists at all. As with the other
LIBNFS_FEATURE_ macros that says the API is there rather than that this
particular build has it, so a build that dropped the dialect still reports
that only at run time, by failing nfs_set_version(nfs, NFS_V4_2).

CONCURRENCY AND NFSv4.1/4.2:
----------------------------
This matters if you drive libnfs hard, and it is the one place where the
NFSv4.1 and NFSv4.2 dialect behaves differently from NFSv3 and NFSv4.0.

From NFSv4 minor version 1 onwards every request travels in a session slot,
and a slot carries one request at a time because the server's reply cache is
keyed on the (slot, sequence id) pair. The session therefore has a fixed
ceiling, set by the server, on how many requests can be outstanding at the
server at once. libnfs asks for 64 slots and uses whatever the server grants.

This is a limit on requests in flight, not on requests you may submit. libnfs
claims a slot as a request goes out on the wire rather than when you hand it
over, so queueing tens of thousands of async calls behaves exactly as it does
on NFSv3 and NFSv4.0; they are simply released onto the connection as slots
free up, and nothing is rejected or blocked at submission time.

What it does mean is that NFSv4.1 and NFSv4.2 cannot keep more than the
granted number of requests at the server simultaneously, where NFSv4.0 has no
such bound. If your throughput comes from very deep request pipelining rather
than from large transfers, measure both dialects before choosing; NFSv4.0 may
well be faster for that shape of workload.

NFSv3 and NFSv4.0 have no sessions and none of this applies to them.

NFSv4.1/4.2 BACK CHANNEL:
-------------------------
From NFSv4 minor version 1 onwards the server can send requests to the
client, callbacks, over the client's own connection. Delegations depend on
this: a server only grants one when it can recall it with CB_RECALL.

libnfs supports a back channel through the raw API only, and it is off by
default. Enable it before the session is created, i.e. before nfs_mount():

    rpc_nfs4_enable_backchannel(nfs_get_rpc_context(nfs), my_cb, my_data);

libnfs answers CB_NULL and CB_SEQUENCE, including replays, itself. my_cb is
called for CB_GETATTR, CB_RECALL, CB_PUSH_DELEG, CB_RECALL_ANY and
CB_RECALLABLE_OBJ_AVAIL, with the result preset to NFS4ERR_NOTSUPP, and sets
the result. Other operations get NFS4ERR_NOTSUPP. rpc_nfs4_backchannel_is_up()
says whether the server accepted the back channel.

To ask for a delegation, send OPEN with the OPEN4_SHARE_ACCESS_WANT_* flags,
or WANT_DELEGATION, through rpc_nfs4_compound_task(). A server that has none
to give may push one later with CB_PUSH_DELEG, if the OPEN had
OPEN4_SHARE_ACCESS_WANT_PUSH_DELEG_WHEN_UNCONTENDED, or say with
CB_RECALLABLE_OBJ_AVAIL that it is worth asking again, if it had
OPEN4_SHARE_ACCESS_WANT_SIGNAL_DELEG_WHEN_RESRC_AVAIL.

Restrictions:
- NFSv4.1 and NFSv4.2 only. NFSv4.0 callbacks would need the client to accept
  connections from the server.
- TCP and TLS only, not RDMA.
- The back channel uses AUTH_NONE, also on a krb5 mount.
- One callback at a time (one back channel slot).
- The callback must set the result before it returns and must not block.
  Do follow-up work, such as DELEGRETURN after CB_RECALL, afterwards.
  Its arguments are only valid during the call.
- With a back channel the high-level API asks for no delegations on OPEN,
  as it cannot handle them. Delegations are only obtained with raw calls.
- The application has to service the context all the time, not only while
  its own requests are in flight. The server expects an answer to a callback
  within the lease period and otherwise revokes the delegation. The sync API
  only reads the socket while one of its calls runs, so on its own it is not
  enough. Use one of the two ways below.

1, An event loop (async API)
Keep the context in the event loop permanently. Get the fd and the events
again each time round, they change on reconnect, and wake up at least every
100ms or so to call nfs_service() even without events, as the lease
keepalive and the timeouts run from there.

    static void my_cb(struct rpc_context *rpc, struct nfs_cb_argop4 *arg,
                      struct nfs_cb_resop4 *res, void *private_data)
    {
            struct my_state *s = private_data;

            if (arg->argop == OP_CB_RECALL) {
                    s->stateid = arg->nfs_cb_argop4_u.opcbrecall.stateid;
                    s->recall_pending = 1;
                    res->nfs_cb_resop4_u.opcbrecall.status = NFS4_OK;
            }
    }

    rpc_nfs4_enable_backchannel(nfs_get_rpc_context(nfs), my_cb, &state);
    nfs_mount_async(nfs, server, export, mount_cb, &state);
    for (;;) {
            struct pollfd pfd = { nfs_get_fd(nfs), nfs_which_events(nfs), 0 };

            if (poll(&pfd, 1, 100) < 0 || nfs_service(nfs, pfd.revents) < 0) {
                    break;
            }
            if (state.recall_pending) {
                    state.recall_pending = 0;
                    /* your code: PUTFH + DELEGRETURN with rpc_nfs4_compound_task() */
                    send_delegreturn(nfs_get_rpc_context(nfs), &state);
            }
    }

2, The service thread (sync API)
nfs_mt_service_thread_start() services the context from a thread of its own,
see README.multithreading. The callback then runs on that thread. It must not
call any *_sync() function, which would wait for the very thread it runs on.
It may queue raw *_task() calls.

    static void my_cb(struct rpc_context *rpc, struct nfs_cb_argop4 *arg,
                      struct nfs_cb_resop4 *res, void *private_data)
    {
            if (arg->argop == OP_CB_RECALL) {
                    res->nfs_cb_resop4_u.opcbrecall.status = NFS4_OK;
                    send_delegreturn(rpc, &arg->nfs_cb_argop4_u.opcbrecall.stateid);
            }
    }

    nfs = nfs_init_context();
    nfs_set_version(nfs, NFS_V4_2);
    rpc_nfs4_enable_backchannel(nfs_get_rpc_context(nfs), my_cb, NULL);
    if (nfs_mount(nfs, server, export) || nfs_mt_service_thread_start(nfs)) {
            ...
    }
    /* use the sync API from any thread */
    nfs_mt_service_thread_stop(nfs);
    nfs_umount(nfs);
    nfs_destroy_context(nfs);

SERVER SUPPORT:
===============
Libnfs supports building RPC servers.
Examples/portmapper-server.c is a small "portmapper" example written using
libnfs.

URL-FORMAT:
===========
Libnfs uses RFC2224 style URLs extended with some minor libnfs extensions.
The basic syntax of these URLs is :

nfs://[<username>@]<server|ipv4|ipv6>[:<port>]/path[?arg=val[&arg=val]*]

Special characters in 'path' are escaped using %-hex-hex syntax.

For example '?' must be escaped if it occurs in a path as '?' is also used to
separate the path from the optional list of url arguments.

Example:
nfs://127.0.0.1/my?path/?version=4
must be escaped as
nfs://127.0.0.1/my%3Fpath/?version=4

Arguments supported by libnfs are :
 tcp-syncnt=<int>  : Number of SYNs to send during the session establish
                     before failing setting up the tcp connection to the
                     server.
 uid=<int>         : UID value to use when talking to the server.
                     default it 65534 on Windows and getuid() on unixen.
 gid=<int>         : GID value to use when talking to the server.
                     default it 65534 on Windows and getgid() on unixen.
 debug=<int>       : Debug level used by libnfs. Default is 0 which is quiet.
                     Higher values increase verbosity.
 timeo=<int>       : The time in deciseconds (tenths of a second) libnfs client
                     will wait for a response before it retries an RPC request.
                     Default value of 'timeo' is 600, i.e., 60 seconds.
                     Values less than 100, i.e., 10 seconds, are not allowed.
 retrans=<int>     : After 'retrans' failed retries libnfs will generate a
                     "server not responding" message and then attempt further
                     recovery action. If no successful RPC response has been
                     received over the connection for the last 'timeo' period,
                     then the connection would be terminated and all queued RPCs
                     will be retried over the new connection. If other RPC
                     responses are being received then it means connection is
                     fine and this is likely a problem with this specific RPC,
                     in that case it'll simply keep retrying the RPC for ever at
                     'timeo' interval. This mimics the 'hard' mount behaviour of
                     NFS clients.
                     If 'retrans' is 0, then RPC is not retried on timeout but
                     instead failed with RPC_STATUS_TIMEOUT. This will roughly
                     mimic the 'soft' mount behaviour of NFS clients.
                     Default value of 'retrans' is 2.
 sec=<krb5|krb5i|krb5p>
                   : Specify the security mode.
 xprtsec=<none|tls|mtls>
                   : Specify the transport security mode.
                     none : No TLS security. This is the default.
                     tls  : TLS with server authentication only.
                     mtls : Mutual TLS, both server and client authentication.

                     See CERTIFICATES for details about specifying certificates/keys
                     that libnfs must use when using "tls" or "mtls" security.
 proto=<tcp|rdma>  : Transport for NFS. rdma uses RPC-over-RDMA on port 20049
                     unless nfsport is given. Default is tcp. See RDMA.
 auto-traverse-mounts=<0|1>
                   : Should libnfs try to traverse across nested mounts
                     automatically or not. Default is 1 == enabled.
 dircache=<0|1>    : Disable/enable directory caching. Enabled by default.
 readonly	   : Set the mount to readonly.
 autoreconnect=<-1|0|>=1>
                   : Control the auto-reconnect behaviour to the NFS session.
                    -1 : Try to reconnect forever on session failures.
                         Just like normal NFS clients do.
                     0 : Disable auto-reconnect completely and immediately
                         return a failure to the application.
                   >=1 : Retry to connect back to the server this many
                         times before failing and returing an error back
                         to the application.
                     Transient reconnect failures (for example, an unavailable
                     network) retry after one second while preserving queued
                     requests. Requests retain their configured timeout/retry
                     policy (default RPC timeout: 60 seconds), rather than
                     failing immediately. Keep servicing RPC timers during the
                     outage. This does not recover expired NFSv4 lease/open state.
 if=<interface>    : Interface name (e.g., eth1) to bind; requires `root`
 version=<3|4|4.1|4.2>
                   : NFS Version. Default is 3. 4 is NFSv4.0, 4.1 is NFSv4.1
                     and 4.2 is NFSv4.2; see the NFSv4 section above for what
                     4.2 adds and for how the session slots of 4.1 and 4.2
                     interact with very high request concurrency.
 nfsport=<port>    : Use this port for NFS instead of using the portmapper.
 mountport=<port>  : Use this port for the MOUNT protocol instead of
                     using portmapper. This argument is ignored for NFSv4
                     as it does not use the MOUNT protocol.
 rsize=<int>       : This is the maximum number of bytes libnfs client will
                     ask in a single READ request. The actual value is the
                     minimum of this and the 'rtmax' value shared by the
                     server in the FSINFO response. The largest rsize value
                     supported is 4,194,304 bytes, i.e., 4MiB. The smallest
                     rsize value supported is 8192 bytes, i.e., 8KiB. The
                     provided value must be a multiple of 4096, else it's
                     rounded down to the nearest 4096 bytes.
                     Default value used when rsize option is not specified,
                     is 1,048,576 bytes, i.e., 1MiB.
 wsize=<int>       : This is the maximum number of bytes libnfs client will
                     send in a single WRITE request. The actual value is the
                     minimum of this and the 'wtmax' value shared by the
                     server in the FSINFO response. The largest wsize value
                     supported is 4,194,304 bytes, i.e., 4MiB. The smallest
                     wsize value supported is 8192 bytes, i.e., 8KiB. The
                     provided value must be a multiple of 4096, else it's
                     rounded down to the nearest 4096 bytes.
                     Default value used when wsize option is not specified,
                     is 1,048,576 bytes, i.e., 1MiB.
 readdir-buffer=<count> | readdir-buffer=<dircount>,<maxcount>
                   : Set the buffer size for READDIRPLUS, where dircount is
                     the maximum amount of bytes the server should use to
                     retrieve the entry names and maxcount is the maximum
                     size of the response buffer (including attributes).
                     If only one <count> is given it will be used for both.
                     The provided value(s) must be multiple of 4096, else
                     they are rounded down to the nearest 4096 bytes.
                     The actual value is the minimum of this and the 'dtpref'
                     value shared by the server in the FSINFO response.
                     Default is 8192 for both.

Auto_traverse_mounts
====================
Normally in NFSv3 if a server has nested exports, for example if it would
export both /data and /data/tmp then a client would need to mount
both these exports as well.
The reason is because the NFSv3 protocol does not allow a client request
to return data for an object in a different filesystem/mount.
(legacy, but it is what it is. One reason for this restriction is to
guarantee that inodes are unique across the mounted system.)

This option, when enabled, will make libnfs perform all these mounts
internally for you. This means that one libnfs mount may now have files
with duplicate inode values so if you cache files based on inode
make sure you cache files based on BOTH st.st_ino and st.st_dev.


ROOT vs NON-ROOT
================
When running as root, libnfs tries to allocate a system port for its connection
to the NFS server. When running as non-root it will use a normal
ephemeral port.
Many NFS servers default to a mode where they do not allow non-system
ports from connecting.
These servers require you use the "insecure" export option in /etc/exports
in order to allow libnfs clients to be able to connect.

On Linux we can get around this restriction by setting the NET_BIND_SERVICE
capability for the application binary.

This is set up by running
    sudo setcap 'cap_net_bind_service=+ep' /path/to/executable
This capability allows the binary to use systems ports like this even when
not running as root. Thus if you set this capability for your application
you no longer need to edit the export on the NFS server to set "insecure".


I do not know what equivalent "capability" support is available on other
platforms. Please drop me an email if your os of choice has something similar
and I can add it to the README.


BUILDING WITH TLS
=================
RPC-with-TLS support is only built if the build host has everything we need for
it, that is:

- GnuTLS >= 3.4.6, headers and library.
- Linux kernel TLS (kTLS), i.e. <linux/tls.h> and the kTLS socket options
  TCP_ULP, SOL_TLS and TLS_TX in the kernel headers we build against.

Both build systems probe for this at configure time and simply build without
TLS support if anything is missing, so libnfs builds fine on a host that has no
gnutls or a kernel without kTLS.

With autoconf:
  ./configure               # build TLS support only if available (default)
  ./configure --enable-tls  # fail the configure if TLS support is not available
  ./configure --disable-tls # never build TLS support

With cmake:
  cmake -DENABLE_TLS=AUTO ..  # build TLS support only if available (default)
  cmake -DENABLE_TLS=ON ..    # fail the configure if TLS support is not available
  cmake -DENABLE_TLS=OFF ..   # never build TLS support


BUILDING WITH KERBEROS
======================
sec=krb5 support needs the gssapi_krb5 headers and library, e.g. from MIT
Kerberos.

With autoconf it is built by default and --without-libkrb5 disables it.

With cmake:
  cmake -DENABLE_KRB5=AUTO ..  # build Kerberos support only if available (default)
  cmake -DENABLE_KRB5=ON ..    # fail the configure if Kerberos is not available
  cmake -DENABLE_KRB5=OFF ..   # never build Kerberos support


RDMA
====
NFS over RDMA (RPC-over-RDMA version 1, RFC 8166) is built on Linux when the
librdmacm and libibverbs development files from rdma-core are found, see
--enable-rdma and -DENABLE_RDMA. It is selected with proto=rdma or
nfs_set_transport(). MOUNT and PORTMAP still use TCP, so an NFSv3 mount needs
both.

READ data is placed directly in the caller's buffer and WRITE data is read
directly from it, which pins that memory while the call is in flight. Processes
without CAP_IPC_LOCK are limited by RLIMIT_MEMLOCK, which may need raising for
many large requests in parallel.

Software iWARP (siw) or RoCE (rxe) is enough for testing:
  rdma link add siw0 type siw netdev eth0

CERTIFICATES
============
libnfs reads CA certs from the following places, so make sure your server's CA
cert is present in at least one of these:

- default system trust store (/etc/ssl/certs/ca-certificates.crt on most systems).
- all PEM format certs in the directory specified by env variable LIBNFS_TLS_TRUSTED_CA_DIR.
- cert(s) present in the PEM file specified by env variable LIBNFS_TLS_TRUSTED_CA_PEM.

If you want to use mutual TLS (xprtsec=mtls) then you will need a client certificate
too. You need to provide the client's public certificate and the private key using
the following environment variables.

- LIBNFS_TLS_CLIENT_CERT_PEM - this is the public certificate.
- LIBNFS_TLS_CLIENT_KEY_PEM  - this is the private key.


DOCUMENTATION
=============
libnfs sources ship with prebuilt manpage(s) in the doc directory.
If you change the manpage sources you need to manually regenerate the new
manpages by running
  cd doc
  make doc


PLATFORM support
=================
This is a truly multiplatform library.

Linux:  - tested with Ubuntu 10.04 - should work with others as well
Cygwin: - tested under 64bit win2k8.
MacOSX: - tested with SDK 10.4 (under Snow Leopard) - should also work with later SDKs and 64Bit
iOS:    - tested with iOS SDK 4.2 - running on iOS 4.3.x
FreeBSD:- tested with 8.2
Solaris
Windows:- tested on Windows 7 64 and Windows XP 32 using Visual Studio 10
        - tested on Windows 7 64 using MingW on Linux to cross-compile (Debian and Ubuntu tested)
Android:- tested with NDK r10e - running on Android 4.4 (should work starting from 2.3.3)
AROS: - Build with 'make -f aros/Makefile.AROS'
Playstation 2: - Build and install with 'cd ps2_ee; make -f Makefile.PS2_EE install'
PlayStation 3: - Build and install the library with 'make -f ps3_ppu/Makefile.PS3_PPU install'


LD_PRELOAD
==========
examples/ld_nfs.c contains a LD_PRELOADable module that can be used to make
several standard utilities nfs aware.
It is still very incomplete but can be used for basic things such as cat and cp.
Patches to add more coverage is welcome.

Compile with :
gcc -fPIC -shared -o ld_nfs.so examples/ld_nfs.c -ldl -lnfs

Then try things like
LD_NFS_DEBUG=9 LD_PRELOAD=./ld_nfs.so cat nfs://127.0.0.1//data/tmp/foo123

LD_NFS_DEBUG=9 LD_PRELOAD=./ld_nfs.so cp nfs://127.0.0.1//data/tmp/foo123 nfs://127.0.0.1//data/tmp/foo123.copy

LD_NFS_UID and LD_NFS_GID can be used to fake the uid and the gid in the nfs context.
This can be useful on "insecure" enabled NFS share to make the server trust you as a root.
You can try to run as a normal user things like :
LD_NFS_DEBUG=9 LD_NFS_UID=0 LD_NFS_GID=0 LD_PRELOAD=./ld_nfs.so chown root:root nfs://127.0.0.1//data/tmp/foo123

This is just a toy preload module. Don't open bugs if it does not work. Send
patches to make it better instead.


RELEASE TARBALLS
================
Release tarballs are available at
https://sites.google.com/site/libnfstarballs/li



MAILING LIST
============
A libnfs mailing list is available at http://groups.google.com/group/libnfs
Announcements of new versions of libnfs will be posted to this list.


NFSv4.0 client-state loss
------------------------

High-level NFSv4.0 operations treat NFS4ERR_STALE_CLIENTID and
NFS4ERR_EXPIRED as terminal client-state loss.
The operation callback receives its normal errno; rpc_service()/nfs_service()
then fails outstanding requests and returns -1 with a client-state diagnostic,
including when RENEW detects it. This also releases callers waiting on the
built-in service thread. Stop that thread, if used, destroy the context and
mount with a fresh one. Transport autoreconnect cannot repair this state.
Ordinary pathname errors remain per-operation errors.
