nds2-client - User  0.15.3
 All Classes Namespaces Functions Variables Typedefs Enumerations Enumerator Groups Pages
SWIG Bindings for NDS Client Library

Table of Contents

Abstract

NDS Developers
LIGO Laboratory, California Institute of Technology, LIGO Scientific Collaboration
nds-discuss@ligo.org
This document explains how to use the NDS (Network Data Server) client interface in Python, Octave, and MATLAB programs. NDS is a TCP/IP protocol for retrieving online or archived data from thousands of instrument channels at LIGO (Laser Interferometer Gravitational-Wave Observatory) sites and data analysis clusters. Language modules for Python, Octave, and MATLAB (via Java) are provided through SWIG.
This document is for the NDS2 client 0.15.3.

Introduction

This document explains how to use the NDS (Network Data Server) client interface in Python, Octave, and MATLAB programs. NDS is a TCP/IP protocol for retrieving online or archived data from thousands of instrument channels at LIGO (Laser Interferometer Gravitational-Wave Observatory) sites and data analysis clusters. Language modules for Python, Octave, and MATLAB (via Java) are provided through SWIG.

Features

The SWIG language interface provides some special features:

Uniform API across multiple languages.

The NDS programming interface is the same in C++, Python, Octave, and MATLAB/Java, up to syntax and coding convention differences in each of the languages.

Works in all (modern) versions of MATLAB.

The MATLAB/Java interface is compatible with all versions of MATLAB going back to the early 2000s.

Supports NDS2 epochs.

NDS2 servers have the concept of epochs of work, these are ranges of start/stop times that requests may be constrained to. This allows requests for data availability and channel lists to be limited to the channels available during the specified epoch, as opposed to the entire history of NDS2.

Server protocols are cached.

When you connect to a new server, the protocol version of the server is saved on disk to speed up the process of connecting to that server in the future.

Users can specify how to handle data that resides on tape.

The NDS2 servers have access to large stores of data that are backed by archival quality tape systems. This means that data that has not been recently accessed may require retrieval from tape. This is very slow operation which can take from minutes to an hour depending on the load on the storage system. To help users gauge the length of time a request will take the system can raise a new error when data is seen to reside on tape storage.

Iterate and Fetch requests that work with gaps.

The old interface will return a data not found error if there is a gap in the data. The new interface allows the application to select the response to gaps. This can be aborting (as in the old code) or filling in the gap with some value.

Channel availability information is exposed.
A new api is introduced which allows for the retreival of availablility information from NDS2 servers on a per channel basis.

Known issues

The following known issues exist:

One transfer at a time with any given connection.

The SWIG interface permits reusing an NDS connection for multiple requests. However, while one request is in progress over a given connection, the server will refuse most other requests. As a consequence, some care must be taken when using the online data retrieval functions iterate() and next(). For example, consider the following MATLAB transcript in which iterate() is called twice on the same connection object:

conn = nds2.connection('nds.ligo-la.caltech.edu');
conn.iterate(1037022904, 1037023024, {'L1:PSL-ISS_PDA_OUT_DQ'});
conn.iterate(1037022904, 1037023024, {'L1:PSL-ISS_PDA_OUT_DQ'});
Error using nds2.connection/iterate
Java exception occurred:
java.lang.RuntimeException: Another transfer is already in progress. Complete
the transfer or retry on a new connection object.
at nds2.nds2JNI.connection_iterate__SWIG_0(Native Method)
at nds2.connection.iterate(connection.java:87)

The above example could be fixed by (a) creating a second connection object for the second call to iterate() or (b) completing the first request by consuming all of the data with next().

Note that the find_channels() and fetch() methods always complete their transfers before returning, so it is safe to issue them repeatedly in any combination.

Windows installer by default does not add NDS libraries to the system PATH.

The Windows installer by default has the option "Do not add nds2-client to the system PATH" checked. Be sure to choose one of the other two options, "Add nds2-client to the system PATH for all users" or "Add nds2-client to the system PATH for current user", so the proper dll files can be loaded at runtime.

Windows installer includes only the MATLAB/Java language interface.

The Windows installer does not include the Python and Octave language interfaces.

Requesting unfiltered channel lists from servers that advertise millions of channels is slow and memory intensive.

Some NDS servers (notably nds.ligo.caltech.edu) advertise millions of channels. Attempting to retrieve the full channel list from such a server may involve transferring as much as a hundred megabytes of channel metadata over the network. Moreover, preparing the representation of millions of channels as objects in Python, Octave, or MATLAB is memory-intensive, consuming a few hundred bytes per object. As a result, requesting the full, un-filtered list of all channels with find_channels() can consume hundreds of megabytes of RAM when working with such NDS servers. (Note that the legacy MATLAB interface also has this issue.) A remedy would be to have the find_channels() method return an iterator rather than a list.

Capitalization of method names in Java.

According to prevailing code conventions, in C, Python, and Octave, function names are all lowercase with underscores (_) separating words, as in find_channels. The MATLAB interface, on the other hand, follows the "camel case" convention of Java, as in findChannels.

Indexing of arrays.

In C and Python, arrays are indexed starting with 0, whereas in Octave and MATLAB arrays are indexed starting with 1. (On the other hand, if you use the MATLAB/Java bindings directly from Java, 0-based indexing applies.) Also, the syntax for accessing elements of sequences returned by find_channels(), fetch(), and next() are slightly different in the target languages: use square brackets [] in C, Python, and Java; curly braces {} in Octave; and parentheses () in MATLAB.

Java strings and MATLAB.

MATLAB does not automatically convert Java strings to MATLAB char arrays in all circumstances, so some methods in the Java interface have return types of char [][] instead of String. This subtlety does not affect MATLAB users; it is only a concern for those who wish to use the NDS interface in Java source code.

Cyrus SASL 2.1.25 and MacPorts.
NDS2 authentication currently does not work the MacPorts package cyrus-sasl2 version 2.1.25. If you are compiling nds2-client from source, build against either Apple's own distribution of cyrus-sasl2 (which comes preinstalled on every Mac) or build against the MIT Kerberos GSSAPI.

Installation

It is easy to install the NDS client on Mac OS X using the MacPorts package and on Windows using the Windows installer, or from source on Linux, UNIX, or Mac OS X.

Installing from packages

The recommended way to install the nds2 client software is from the LSC soft package repositories. You should configure your box to use the LSC soft repositories (the details of this are out of scope of this document, but you may consult software.ligo.org for information.

If you are using a Debian you can install the packages using apt-get:

$ apt-get install

If you are using a Scientific Linux 7 system you can install the packages using yum:

$ sudo yum install nds2-client-all

Installing from source

Dependencies

Before you begin installing from source, you should make sure that you have the following dependencies installed.
If you are on Mac OS X and have the MacPorts package manager installed, then you can install all of the dependencies for building from source with all language interfaces enabled by using the port command:
$ sudo port install swig-java swig-octave swig-python pkgconfig kerberos5 sqlite3 cmake
On Debian, install the dependencies using apt-get:
$ apt-get install swig libsqlite3-dev octave octave-headers default-jdk python libsasl2-dev krb5-user libsasl2-modules-gssapi-mit cmake
And on Red Hat or Scientific Linux, install the dependencies using yum:
$ yum install swig sqlite-devel octave octave-devel java-1.6.0-openjdk-devel cyrus-sasl-devel cmake

Obtaining the source code

First, download a source release tarball, unarchive it, and change directories into the downloaded source folder, using the following commands, where X.X.X is the version of nds2-client that you want:

$ wget http://www.lsc-group.phys.uwm.edu/daswg/download/software/source/nds2-client-<b>X.X.X</b>.tar.gz
$ tar -xzf nds2-client-<b>X.X.X</b>.tar.gz
$ mkdir nds2-client-<b>X.X.X</b>-build
$ cd nds2-client-<b>X.X.X</b>-build

Building

Next, configure, build, and install the package. This is done with the cmake tool. The table below lists essential command line arguments for enabling or disabling optional features.

Essential command line options for cmake
Option Purpose
-DCMAKE_INSTALL_PREFIX=/path/to/local Install files into /path/to/local/bin, /path/to/local/lib, etc.
-DPROG_MATLAB_=/path/to/matlab Use this to make sure the build finds a matlab installation.
Note
If you are on a Mac and you use MacPorts, you should add the following options to cmake:

By default, files will be installed into /usr/bin, /usr/lib, /usr/share, etc. If you would like to install into another location, such as your home directory, you should provide the -DCMAKE_INSTALL_PREFIX option when you call cmake.

For example, to enable all available features, and install into the directory /path/to/local, run the following commands:

$ cmake ../nds2-client-X.Y.Z -DCMAKE_INSTALL_PREFIX=/path/to/local
$ make
$ make install

Environment script

If you are going to install the NDS client in your own home directory, then you will need to set some environment variables every time that you log in.

The client installs a script to help setup environment variables for the specific install of the nds2-client. Assuming that you are installing into the directory /path/to/local, run the following command.

Example environments script
$ source <b>/path/to/local</b>/etc/nds2-client-user-env.sh

Installing with MacPorts

On Mac OS X, you can use the following instructions to install the latest version of nds2-client with the MacPorts package manager.

Installing with Windows installer

Note
Currently the nds2-client Windows installer is only available for 64-bit Windows.
The nds2 client (version 0.15.3) installer for Windows 7 available at the moment is only 64 bits. You can download it from https://software.ligo.org/windows/nds2-client-0.15.3-win64.exe .
Run the installer. The default options are reasonable if you don't know of a reason to change them.
An important dependency of the nds2 client is the authentification libraries. If you do not have them installed then the client will complain of missing gssapi64.dll. The recomended authentification package for Windows is MIT Kerberos. It is important that you get the 64 bit version. MIT does not have an installer for 64 bit Windows so it is advised to use the one provided by Secure Endpoints through their Heimdal release. Notice that they recomend installing first the 64 bit installer and then install the 32 bit installer on top of the previous one. I can confirm that the 2nd step was not needed, just installing the 64 bit installer was enough.
In order to test the nds2 client you would need to use the previously installed Kerberos (from the Start menu the executable is called "Network Identity Manager". Notice that you can set it up to start authomatically when you log in) to authenticate to LIGO.ORG, using the albert.einstein naming convention. From the menu 'Credential' choose 'Obtain new Credentials'. In the field 'Realm' insert: LIGO.ORG, in Username insert: your equivalent albert.einstein, then fill the Password field and you are ready to press OK.
Notice that you should only have this Credential active, if you have any other credential active at the same time causes issues with 'daq_connect' of the nds2 client. This error is obtained: error in gss_init_sec_contextError in daq_connect: Request SASL authentication protocol
Other reason to obtain a similar error message is by having the date and time on your computer considerably different to that of the nds2 server.
At this point we can confirm that the nds2 client is properly installed and communicating with the Ticket Manager by opening a command window and entering:n
cd C:\Program Files (x86)\nds2-client 0.15.3\bin
nds2_query -l -n nds.ligo.caltech.edu
Check if you see the above error messages. If everything is working you will get a (long) list of available channels. If you see the dialog box complaining of a missing MSVCR100.dll library then you will need to install the Visual C redistribution libraries which can be downloaded for free from Microsoft. They come as an installer, and need to be run from an account with administrator privileges.
You may also want to add the nds2 client bin folder to the system variable PATH. In order to do this right click in 'Computer' and select 'Properties'. There click in 'Advanced system settings' for which you will need administrator privileges. On the tab 'Advanced' click on the button 'Environment Variables' and under 'System variables' choose the variable 'Path' then click 'Edit...' and in the 'Variable value' field ADD the folder: C: Files (x86)-client 0.15.3.

Getting started

This chapter explains how to get started with the NDS client programming interface, including how to sign in to access authenticated NDS protocol 2 servers and how to complete the configuration of your MATLAB environment. It concludes with transcripts of very simple 'Hello, world'-like sessions in Python, Octave, and MATLAB.

If you are going to work with an authenticated NDS Protocol 2 server, you will first need to establish a valid Kerberos ticket by running kinit. If you have never used kinit before, you will need to pass your LIGO.ORG username to kinit, like this:

$ kinit <b>albert.einstein</b>@LIGO.ORG
Password for <b>albert.einstein</b>@LIGO.ORG:

replacing albert.einstein with your own user name. The .ORG part must be in all-caps. At the password prompt, provide your Kerberos password. Probably need to remind users here how and when they would have signed up for a Kerberos password.

If you have previously signed in, then you can renew your existing Kerberos ticket by running kinit without a username, like this:

$ kinit
Password for <b>albert.einstein</b>@LIGO.ORG:

Again, type your Kerberos password when prompted.

Setting up your MATLAB environment

If you are going to use the MATLAB/Java interface, one additional configuration step is required: you must add the NDS client to MATLAB's Java classpath. You can either modify the 'dynamic classpath' for your current MATLAB session, or permanently add NDS to MATLAB's classpath.

Adding NDS to the classpath for the current MATLAB session

You can modify the 'dynamic classpath' for the current MATLAB session by using the MATLAB command javaaddpath, like this:

> javaaddpath /path/to/local/share/java/nds2/nds2.jar
Note
For example: if you are on a Mac and you installed from MacPorts, the appropriate path is:
> javaaddpath /opt/local/share/java/nds2/nds2.jar

Permanently adding NDS to the classpath

Alternatively, you can permanently add NDS to the MATLAB classpath by editing the MATLAB configuration file classpath.txt. Inside MATLAB, open this file by issuing the edit command:

> edit classpath.txt

This command opens classpath.txt in an editor. Scroll to the end of the file, which might look something like this:

...
# Java classpath entries for webintegration
$matlabroot/java/jar/webintegration.jar
# Java classpath entries for ws_clients
$matlabroot/java/jarext/mwaws_client.jar
$matlabroot/java/jarext/dws_client.jar
$matlabroot/java/jarext/webservices/loginws_client.jar
$matlabroot/java/jarext/webservices/service_request_client.jar

Add the following line:

/path/to/local/share/java/nds2/nds2.jar

Finally, save classpath.txt and restart MATLAB.

Note on java paths, jar files versus loose classes

In previous versions the java class path was configured to point to a library directory and not to a specific file. This is a change implemented starting in NDS client 0.13.0. For the first few 0.13.x releases both the old method of referring to a directory and the new of refering to a specific file will be made available. However going forward users should configure their MATLAB installs by directly referencing the nds2.jar file.

Some explaination for the changs. The MATLAB NDS client is implemented in the java programming language. Java packages are typically distributed in .jar files (Java ARchives). In the past the java/MATLAB bindings had distributed the NDS client as a loose set of java '.class' files. This is changing to bring the NDS bindings more in line with how Java is typically used. To help users during a transition period the NDS client will be distributed with both the '.jar' file and the loose java '.class' files for the next few releases. However the end state will involve distributing only the '.jar' files.

Your first script with NDS

Getting started: your first script with NDS in Python

> import nds2
> conn = nds2.connection('nds.ligo-wa.caltech.edu', 31200)
> print conn
<nds.ligo-wa.caltech.edu:31200 (protocol version 2)>
> conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
> buffers = conn.fetch(1024417918, 1024417928, ['H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'])
> print buffers
'<H0:PEM-EY_SEISX (GPS time 953618740.000000000, 15360 samples)>', '<H0:PEM-EY_SEISY (GPS time 953618740.000000000, 15360 samples)>'
> buffers[0].data
2.39337158, 2.39943528, 2.38724732, ..., 2.39846468, 2.39515495, 2.39528942

Getting started: your first script with NDS in Octave

> conn = nds2.connection('nds.ligo-wa.caltech.edu', 31200)
conn = <nds2.ligo-wa.caltech.edu:31200 (protocol version 2)>
> conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
> buffers = conn.fetch(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'})
[1] =
<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>
[2] =
<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>
> buffers{1}.data
2.3934
2.3994
2.3872
2.3891
...

Getting started: your first script with NDS in MATLAB

> conn = nds2.connection('nds.ligo-wa.caltech.edu', 31200)
conn =
<nds.ligo-wa.caltech.edu:31200 (protocol version 2)>
> conn.setParameter('ALLOW_DATA_ON_TAPE', '1')
> buffers = conn.fetch(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'})
buffers =
nds2.buffer[]:
[nds2.buffer]
[nds2.buffer]
<![CDATA[>>]]> buffers(1).getData()
ans =
2.3934
2.3994
2.3872
2.3891
...

About the channel list cache

NDS servers typically advertise tens to hundreds of thousands of channels. Under the NDS1 protocol, the channel list must be downloaded before fetching data so that the data types of the channels are known. NDS2 does not suffer from this limitation because the data types of requested channels are always sent along with the binary data. However, when you are using the NDS2 client library interactively you may want to be able to search through the channel list at any time.

The SWIG interface deals with the channel lists is several ways. For NDS1 connections the SWIG interface caches the channel lists for all servers to which you have connected in the past 24 hours. After querying the channel list for a given server, for the next 24 hours any time you call find_channels(), instead of downloading the channel list from the server, the NDS client library searches through your local cache. After 24 hours, the next time you call find_channels() for that server, the client library will clear the cache and download that server's channel list again.

You can discard the contents of the channel cache for a given server at any time using the connection.clear_cache() method (in Java, connection.clearCache()). This command drops all the tables in the database, leaving the database empty.

Each server's cache is stored on your computer in its own Sqlite database.

For NDS2 servers the list is not cached. However you can reduce the size of the channel list that is returned by setting an epoch (a [start,stop) time range). By setting an epoch you receive only the channels that were active during that [start, stop) time frame.

Recipes

This chapter explains the NDS client programming interface by example with recipes for common tasks.

Opening a connection

To open a new connection, create a new connection object with nds2.connection(). The arguments are the server's hostname, port number, and the protocol version. If the port number is omitted, then the default port is 31200. If the protocol version is omitted, then the protocol version is automatically detected by first attempting an NDS2 connection and then falling back to NDS1.

Opening a connection in Python

# Open a connection, discovering the server's protocol version automatically.
conn = nds2.connection('example.edu')
# Equivalent to:
# conn = nds2.connection('example.edu', 31200)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TRY)
# And also equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_ONE | nds2.connection.PROTOCOL_TWO)
# Open a connection to an NDS protocol 1 server.
conn = nds2.connection('example.edu', 31200, 1)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_ONE)
# Open a connection to an NDS protocol 2 server.
conn = nds2.connection('example.edu', 31200, 2)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TWO)

Opening a connection in Octave

#Open a connection, discovering the server's protocol version automatically.
conn = nds2.connection('example.edu')
# Equivalent to:
# conn = nds2.connection('example.edu', 31200)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TRY)
# And also equivalent to:
# conn = nds2.connection('example.edu', 31200, bitor(nds2.connection.PROTOCOL_ONE, nds2.connection.PROTOCOL_TWO))
# Open a connection to an NDS protocol 1 server.
conn = nds2.connection('example.edu', 31200, 1)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_ONE)
# Open a connection to an NDS protocol 2 server.
conn = nds2.connection('example.edu', 31200, 2)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TWO)

Opening a connection in MATLAB

% Open a connection, discovering the server's protocol version automatically.
conn = nds2.connection('example.edu')
% Equivalent to:
% conn = nds2.connection('example.edu', 31200)
% Equivalent to:<
% conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TRY)
% And also equivalent to:
% conn = nds2.connection('example.edu', 31200, bitor(nds2.connection.PROTOCOL_ONE, nds2.connection.PROTOCOL_TWO))
% Open a connection to an NDS protocol 1 server.
conn = nds2.connection('example.edu', 31200, 1)
% Equivalent to:
% conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_ONE)
% Open a connection to an NDS protocol 2 server.
conn = nds2.connection('example.edu', 31200, 2)
% Equivalent to:
% conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TWO)

Setting connection parameters

The connection object has a series of parameters that can be set. Currently the parameters that can be set are "ALLOW_DATA_ON_TAPE", "GAP_HANDLER", and "ITERATE_USE_GAP_HANDLERS".

ALLOW_DATA_ON_TAPE

NDS2 only. The NDS2 server may serve data that resides on a high latency storage layer, such as a tape system. This may lead to data requests taking minutes or hours to complete, depending on the load on the storage system. As of version 0.12 of the client the default is to raise an error when accessing data that is on a high latency storage layer. This allows the application to provide feedback (if needed) to a users regarding amount of time that a request may take. If this parameter is set to a true value ("True", "1", "yes") then an error will not be raised when requesting data on a high latency storage.

As of client 0.12.0 "ALLOW_DATA_ON_TAPE" is set to False by default. You MUST enable data on tape if you want it.

GAP_HANDLER

For a given request there may not be be data available to fill the request completely. This happens due to issues upstream of the NDS server. How this is handled is application specific. Setting the "GAP_HANDLER" parameter allows the application to specify what to do. This includes options such as abort, zero fill the data, ...

Available handlers:
"ABORT_HANDLER" This aborts the request when a gap is found in the data.
"STATIC_HANDLER_ZERO" This zero fills any missing data.
"STATIC_HANDLER_ONE" This fills any missing data with ones.
"STATIC_HANDLER_NAN" This fills any missing data with NaN values (or zero for integer channels).
"STATIC_HANDLER_POS_INF" This fills any missing data with +infinity (or the maximum integer value for integer channels).
"STATIC_HANDLER_NEG_INF" This fills any missing data with -infinity (or the minimum integer value for integer channels).

ITERATE_USE_GAP_HANDLERS

The iterate methods have a special case. Unlike fetch operations which work on a single block, the iterate methods retrieve chunks of data that may not need to be contigous. Setting ITERATE_USE_GAP_HANDLERS to "false" configures the connection to simply skip any gaps in the data and only return the data that is available.

Please note that if you are asking for multiple channels that do not have identical gaps the NDS servers will return a data not found error if ITERATE_USE_GAP_HANDLERS is set to false.

Parameters can also be given default values in the system environment. The names of the environment variables are derived by prefixing them with "NDS2_CLIENT_". So the available enviornment variables are "NDS2_CLIENT_ALLOW_DATA_ON_TAPE", "NDS2_CLIENT_GAP_HANDLER", and "NDS2_CLIENT_ITERATE_USE_GAP_HANDLERS".

Setting parameters in Python with connection.set_parameter()

# Allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
# Equivalent to:
# conn.set_parameter('ALLOW_DATA_ON_TAPE', 'True')
# Set a gap handler:
conn.set_parameter('GAP_HANDLER', 'STATIC_HANDLER_ZERO')

Setting parameters in Octave with connection.set_parameter()

# Allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
# Equivalent to:
# conn.set_parameter('ALLOW_DATA_ON_TAPE', 'True')
# Set a gap handler:
conn.set_parameter('GAP_HANDLER', 'STATIC_HANDLER_ZERO')

Setting parameters in MATLAB with connection.set_parameter()

% Allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
% Equivalent to:
% conn.set_parameter('ALLOW_DATA_ON_TAPE', 'True')
% Set a gap handler:
conn.set_parameter('GAP_HANDLER', 'STATIC_HANDLER_ZERO')

Looking up and counting channels

You can look up information (channel metadata or simple channel counts) about available channels using the versatile connection.find_channels() and connection.count_channels() methods. You can look up a single channel by its name, or you can look up multiple channels with bash-compatible pattern matching. Channel name patterns support the following syntax:

As an example containing all of the above features, the pattern ?????, * {l{on,ov}e}ly {world,planet} matches the text Hello, very lovely world.

You can optionally limit the search by channel type: ONLINE, RAW, RDS, STREND, MTREND, TEST_POINT, STATIC, or any bitwise combination of these flags.

You can also limit the search by data type: INT16, INT32, and INT64 for signed 16-, 32-, and 64-bit integers respectively; FLOAT32 and FLOAT64 for 32- and 64-bit floating point, COMPLEX32 for floating point complex; or any bitwise combination of these flags.

Finally, you can restrict the search by minimum and maximum sample rate in Hz.

When requesting channel counts there are a few items to concider. On NDS1 all counts are done against the locally cached channel database. On NDS2 simple requests (filtering on name, and channel type [RAW, ONLINE, ...]) are simple requests to the servers, more detailed requests will require the server to send a partial channel list for client side filtering. For NDS2 the channel count information is also constrained to the currently set NDS2.

Looking up channel information in Python with connection.find_channels()

# List all channels.
conn.find_channels()
# Equivalent to:
# conn.find_channels('*')
<('<G1:ASC_BDO3_EP-ROT (512Hz, RAW, INT16)>', '<G1:ASC_BDO3_EP-TILT (512Hz, RAW, INT16)>', ...]]>
# Look up a single channel by name.
conn.find_channels('H0:PEM-EY_SEISX')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>',)
# Look up multiple channels by name with a <b>?</b> wildcard, standing for exactly one free character.
conn.find_channels('H0:PEM-EY_SEIS?')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISY (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISZ (256Hz, RAW, INT16)>')
# Look up multiple channels by name with a <b>?</b> wildcard, standing for any number (zero or more) free characters.
conn.find_channels('H0:PEM-EY_*')
'<H0:PEM-EY_BPO5 (16Hz, RAW, FLOAT32)>', '<H0:PEM-EY_BPO5.max (0.0166667Hz, MTREND, FLOAT32)>', '<H0:PEM-EY_BPO5.mean (0.0166667Hz, MTREND, FLOAT64)>', ...
# Look up all ONLINE-type channels with any name.
conn.find_channels('*', nds2.channel.CHANNEL_TYPE_ONLINE)
# Look up all 16-bit and 32-bit integer, ONLINE-type channels with names ending in 'SEISY'.
conn.find_channels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, nds2.channel.DATA_TYPE_INT16 | nds2.channel.DATA_TYPE_INT32)
# Look up all 16-bit integer, ONLINE-type channels with names ending in 'SEISY' and whose sample rates are between 128 Hz and 512 Hz.
conn.find_channels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, nds2.channel.DATA_TYPE_INT16, 128, 512)

Counting available channels in Python with connection.count_channels()

# Count all channels.
conn.count_channels()
9686715L
# Equivalent to:
# conn.count_channels('*')
9686715L
# Count all L1 online channels (connected to nds.ligo-la.caltech.edu)
conn.count_channels('L1:*', nds.channel.CHANNEL_TYPE_ONLINE)
1036L

Looking up channel information in Octave with connection.find_channels()

# List all channels.
conn.find_channels()
# Equivalent to:
# conn.find_channels('*')
('<G1:ASC_BDO3_EP-ROT (512Hz, RAW, INT16)>', '<G1:ASC_BDO3_EP-TILT (512Hz, RAW, INT16)>', ...)
# Look up a single channel by name.
conn.find_channels('H0:PEM-EY_SEISX')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>',)
# Look up multiple channels by name with a <b>?</b> wildcard, standing for exactly one free character.
conn.find_channels('H0:PEM-EY_SEIS?')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISY (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISZ (256Hz, RAW, INT16)>')
# Look up multiple channels by name with a <b>?</b> wildcard, standing for any number (zero or more) free characters.
conn.find_channels('H0:PEM-EY_*')
('<H0:PEM-EY_BPO5 (16Hz, RAW, FLOAT32)>', '<H0:PEM-EY_BPO5.max (0.0166667Hz, MTREND, FLOAT32)>', '<H0:PEM-EY_BPO5.mean (0.0166667Hz, MTREND, FLOAT64)>', ...)
# Look up all ONLINE-type channels with any name.
conn.find_channels('*', nds2.channel.CHANNEL_TYPE_ONLINE)
# Look up all 16-bit and 32-bit integer, ONLINE-type channels with names ending in 'SEISY'.
conn.find_channels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, bitor(nds2.channel.DATA_TYPE_INT16, nds2.channel.DATA_TYPE_INT32))
# Look up all 16-bit integer, ONLINE-type channels with names ending in 'SEISY' and whose sample rates are between 128 Hz and 512 Hz.
conn.find_channels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, nds2.channel.DATA_TYPE_INT16, 128, 512)

Counting available channels in Octave with connection.count_channels()

# Count all channels.
conn.count_channels()
9686715L
# Equivalent to:
conn.count_channels('*')
9686715L
# Count all L1 online channels (connected to nds.ligo-la.caltech.edu)
conn.count_channels('L1:*', nds.channel.CHANNEL_TYPE_ONLINE)
1036L

Looking up channel information in MATLAB with connection.findChannels()

% List all channels.
conn.findChannels()
% Equivalent to:
% conn.findChannels('*')
'<G1:ASC_BDO3_EP-ROT (512Hz, RAW, INT16)>', '<G1:ASC_BDO3_EP-TILT (512Hz, RAW, INT16)>', ...
% Look up a single channel by name.
conn.findChannels('H0:PEM-EY_SEISX')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>',)
% Look up multiple channels by name with a <b>?</b> wildcard, standing for exactly one free character.
conn.findChannels('H0:PEM-EY_SEIS?')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISY (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISZ (256Hz, RAW, INT16)>')
% Look up multiple channels by name with a <b>?</b> wildcard, standing for any number (zero or more) free characters.
conn.findChannels('H0:PEM-EY_*')
('<H0:PEM-EY_BPO5 (16Hz, RAW, FLOAT32)>', '<H0:PEM-EY_BPO5.max (0.0166667Hz, MTREND, FLOAT32)>', '<H0:PEM-EY_BPO5.mean (0.0166667Hz, MTREND, FLOAT64)>', ...
% Look up all ONLINE-type channels with any name.
conn.findChannels('*', nds2.channel.CHANNEL_TYPE_ONLINE)
% Look up all 16-bit and 32-bit integer, ONLINE-type channels with names ending in 'SEISY'.
conn.findChannels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, bitor(nds2.channel.DATA_TYPE_INT16, nds2.channel.DATA_TYPE_INT32))
% Look up all 16-bit integer, ONLINE-type channels with names ending in 'SEISY' and whose sample rates are between 128 Hz and 512 Hz.
conn.findChannels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, nds2.channel.DATA_TYPE_INT16, 128, 512)

Counting available channels in Matlab with connection.count_channels()

% Count all channels.
conn.count_channels()
9686715L
% Equivalent to:
conn.count_channels('*')
9686715L
% Count all L1 online channels (connected to nds.ligo-la.caltech.edu)
conn.count_channels('L1:*', nds.channel.CHANNEL_TYPE_ONLINE)
1036L

Fetching offline data

Use the connection.fetch() method to retrieve data spanning a range of GPS times and a list of channel names.

If the requested data range comprises multiple buffers, connection.fetch() will concatenate and trim the buffers as required to match the requested time range.

Warning
If your request includes one or more minute-trend channels, then the GPS start and stop time arguments must be divisible by 60.

Fetching offline data in Python

# As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
# Arguments are start and stop times in GPS seconds and a list of channel names.
buffers = conn.fetch(1024417918, 1024417928, ['H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'])
# The return value of <b>fetch()</b> is a sequence of <b>nds2.buffer</b> objects.
print buffers
('<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>', '<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>')
# Each buffer object has a <b>channel</b> property which provides information about the channel.
print buffers[0].channel
<H1:PSL-ISS_PDA_OUT_DQ (32768Hz, RAW, FLOAT32)>
print buffers[0].channel.sample_rate
32768.0
# Each buffer object also has properties called <b>gps_seconds</b> and <b>gps_nanoseconds</b> that relate the buffer's timestamp.
print buffers[0].gps_seconds, buffers[0].gps_nanoseconds
1024417918 0
# Finally, each buffer's numerical content is in the property called <b>data</b>.
buffers[0].data
array([ 2.39337158, 2.39943528, 2.38724732, ..., 2.39846468,
2.39515495, 2.39528942], dtype=float32)

Fetching offline data in Octave

# As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
# Arguments are start and stop times in GPS seconds and a list of channel names.
buffers = conn.fetch(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'})
# The return value of <b>fetch()</b> is a cell array of <b>nds2.buffer</b> objects.
buffers
buffers =
(
[1] =
<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>
[2] =
<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>
)
# Each buffer object has a <b>channel</b> property which provides information about the channel.
buffers{1}.channel
ans =
<H1:PSL-ISS_PDA_OUT_DQ (32768Hz, RAW, FLOAT32)>
buffers{1}.channel.sample_rate
ans = 32768
# Each buffer object also has properties called <b>gps_seconds</b> and <b>gps_nanoseconds</b> that relate the buffer's timestamp.
buffers{1}.gps_seconds
ans = 1.0244e+09
buffers{1}.gps_nanoseconds
ans = 0
# Finally, each buffer's numerical content is in the property called <b>data</b>.
buffers{1}.data
ans =
2.3934
2.3994
2.3872
2.3891
...

Fetching offline data in MATLAB

% As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
% Arguments are start and stop times in GPS seconds and a list of channel names.
buffers = conn.fetch(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'})
% The return value of <b>fetch()</b> is a cell array of <b>nds2.buffer</b> objects.
buffers
buffers =
nds2.buffer[]:
[nds2.buffer]
[nds2.buffer]
% Each buffer object has a <b>channel</b> property which provides information about the channel.
buffers(1).getChannel()
ans =
<H1:PSL-ISS_PDA_OUT_DQ (32768Hz, RAW, FLOAT32)>
buffers(1).getChannel().getSampleRate()
ans =
32768
% Each buffer object also has properties called <b>gps_seconds</b> and <b>gps_nanoseconds</b> that relate the buffer's timestamp.
buffers(1).getGpsSeconds()
ans =
1.0244e+09
buffers(1).getGpsNanoseconds()
ans =
0
% Finally, each buffer's numerical content is in the property called <b>data</b>.
buffers(1).getData()
ans =
2.4013
2.3873
2.3949
2.3995
...

Grabbing online or offline data progressively

Use the method connection.iterate() to retrieve data progressively or online. Use connection.next() to grab each chunk as it becomes available. connection.iterate() supports the following arguments:

Offline, manual step size connection.iterate(start, stop, step, list_of_channel_names) Request the channels list_of_channel_names, spanning GPS times in seconds from start to stop, in blocks of step seconds.
Offline, automatic step size connection.iterate(start, stop, list_of_channel_names) Same as above, but choose an appropriate step automatically to keep each buffer smaller than 128 MB.
Online, manual step size connection.iterate(step, list_of_channel_names) Request online data for channels list_of_channel_names starting from the current GPS time and continuing indefinitely. Transfer data in steps of step seconds.
Online, autoamtic step size connection.iterate(list_of_channel_names) Request online data, but use the shortest possible step.
Warning
If your request includes one or more minute-trend channels, then start, stop, and step must be divisible by 60.
If you specify a GPS stop time, the returned buffers may not exactly end on that GPS stop time, because connection.next() makes no attempts to trim the received buffers to size.

In Python, connection.iterate() returns the original nds2.connection object, which implements the Python iterator protocol. As a result, you can use connection.iterate() as the iterator of a for...in loop.

Grabbing online or offline data progressively in Python

# As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
# To progressively download offline data in small chunks,
# call connection.iterate() with the same arguments as connection.fetch():
# GPS start and stop times, and list of channel names.
for bufs in conn.iterate(1024417918, 1024417928, ['H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ']):
print bufs
('<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 32768 samples)>', '<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 32768 samples)>'
('<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417919.000000000, 32768 samples)>', '<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417919.000000000, 32768 samples)>')
...
# Alternatively, to initiate online data retrieval starting from the current time,
# omit the GPS start and stop time when calling connection.iterate().
for bufs in conn.iterate(['H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ']):
print bufs
...

In Octave, there is no for...in loop, but you can retrieve the buffers in a while loop, checking whether there is more data remaining with connection.has_next().

Grabbing online or offline data progressively in Octave

# As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
# To progressively download offline data in small chunks,
# call connection.iterate() with the same arguments as connection.fetch():
# GPS start and stop times, and list of channel names.
conn.iterate(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'});
# After calling connection.iterate(), call connection.next() repeatedly to retrieve bufferes repeatedly.
while conn.has_next()
bufs = conn.next();
disp(bufs);
end
(
[1] =
<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 32768 samples)>
[2] =
<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 32768 samples)>
)
(
[1] =
<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417919.000000000, 32768 samples)>
[2] =
<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417919.000000000, 32768 samples)>
)
...
# Alternatively, to initiate online data retrieval starting from the current time,
# omit the GPS start and stop time when calling connection.iterate().
conn.iterate({'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'});
while conn.has_next()
bufs = conn.next();
...

You can also use a while conn.hasNext() loop in MATLAB.

Grabbing online or offline data progressively in MATLAB

% As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
% To progressively download offline data in small chunks,
% call connection.iterate() with the same arguments as connection.fetch():
% GPS start and stop times, and list of channel names.
conn.iterate(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'});
% After calling connection.iterate(), call connection.next() repeatedly to retrieve bufferes repeatedly.
while conn.hasNext()
bufs = conn.next();
disp(bufs);
end
nds2.buffer[]:
[nds2.buffer]
[nds2.buffer]
nds2.buffer[]:
[nds2.buffer]
[nds2.buffer]
...
% Alternatively, to initiate online data retrieval starting from the current time,
% omit the GPS start and stop time when calling connection.iterate().
conn.iterate({'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'});
while conn.hasNext()
bufs = conn.next();
...

Getting channel availability

Channels are not always available. The channel may have a gap in the data for many reasons, including events such as computer and IFO downtime at the observatories (planned and unplanned) and IFO model changes (renaming, adding/removing channges, ...).

NDS2 servers have a listing of when each channel is available and can provide that data to users. NDS1 servers do not currently export this information. As such this section only applies to NDS2 connections.

The connection.get_availability() method is used to retrieve timespans when a channel is available. The call operates on a list of channel names, and returns detailed information regarding the availability of each channel requested. The information includes the start and stop times of segments of data as well as the frame types of the files these segments of data apply to. The detailed list often contains overlapping segments, often science and commissioning frames.

If all you need is a list of timespans that the data is available during, you can convert the detailed availability list to a simpler list that is only gps_start, gps_stop pairs.

Retrieving H1:GDS-CALIB_STRAIN availability during O1 in python

import nds
conn = nds.connection('nds.ligo.caltech.edu', 31200)
# only view the O1 time spans
# we could also do this as
# conn.set_epoch('1126621184-1137258496')
# or
# conn.set_epoch(1126621184, 1137258496)
#
conn.set_epoch('O1')
avail = conn.get_availability(['H1:GDS-CALIB_STRAIN'])
print "Detailed availability - %s" % avail[0].name
for entry in avail[0].data:
print "%s %d %d" % (entry.frame_type, entry.gps_start, entry.gps_stop)
# convert the detailed list to a simplified list
simple_avail = avail.simple_list()
print "Simplified availability"
for entry in simple_avail[0]:
print "%d %d" % (entry.gps_start, entry.gps_stop)
# This produces the following output (with the output truncated):
Detailed availability - H1:GDS-CALIB_STRAIN
H-H1_HOFT_C00 1126621184 1126988756
H-H1_HOFT_C00 1126989824 1127581128
H-H1_HOFT_C00 1127581340 1127581364
H-H1_HOFT_C00 1127581452 1127581772
H-H1_HOFT_C00 1127581796 1129383148
H-H1_HOFT_C00 1129396152 1129999948
H-H1_HOFT_C00 1130000036 1130014756
...
Simplified availability
1126621184 1126988756
1126989824 1127581128
1127581340 1127581364
1127581452 1127581772
1127581796 1129383148
1129396152 1129999948
1130000036 1130014756
...

Common Problems

There are a few common issues that have been seen when working with NDS1 and NDS2 using the SWIG clients.

Slow data retrieval when data is on tape.

It is simply a slow process to read back data from tape. To help manage expectations the SWIG client will now default to an error state when asked for data that resides on tape. This is not done to prevent access to the data, but to allow the user to resubmit the request with the understanding that data will take a while (you should expect to wait at least several minutes, possibly much longer depending on the load on the tape system).

You can automatically ignore errors caused by data being on tape by setting the 'ALLOW_DATA_ON_TAPE' parameter.

  • Either by setting the NDS2_CLIENT_ALLOW_DATA_ON_TAPE environment variable to 1 and restarting your analysis job (restarting python, matlab, or octave). On a UNIX type system with bash it can be set as export NDS2_CLIENT_ALLOW_DATA_ON_TAPE=1, remembering to start the analysis job from that shell.
  • Or by calling the set_parameter('ALLOW_DATA_ON_TAPE', '1') method on your connection object.

Data not found due to a gap.

This problem is mitigated with the new SWIG interface which allows fetch and iterate requests to survive gaps in the data. In addition the connection object has ability to return availability information.

NDS2 does not recognize my channel.

If the data exists, it may be that the NDS2 server is mapping your request to the wrong channel. When a channel is addressed only by its base name it may not be resolved to proper channel. This can happen when there has been a rate or data type change in the epoch you currently using. You may request a channel 'A' expecting it to be at 2kHz, but NDS2 sees two versions of 'A', one at 2kHz and one at 8kHz and NDS2 could pick the wrong one.

There are two ways to help mitigate this. First set an epoch. This can reduce the channel list that NDS2 has to choose from. Use the connection.set_epoch() method.

Second, you can more fully qualify the channel name. Fully qualified names are of the form basename,rate,type or basename,type,rate. So you could ask for the 'H1:GDS-CALIB_STRAIN' channel by requesting 'H1:GDS-CALIB_STRAIN,raw,16384' or 'H1:GDS-CALIB_STRAIN,16384,raw'. By qualifying your channel requests you can help reduce ambiguity. In this example there happen to be two H1:GDS-CALIB_STRAIN channels, one 'raw' version which is the archived version for historical data, and the 'online' data which is only useful for streaming live, no history is retained.

Other Documentation

The nds developerment group is migrating the documentation from this manual to a different format of documentation that will be easier to maintain and update as time goes forward. The new documentation is available at https://www.lsc-group.phys.uwm.edu/daswg/projects/nds-client/doc/doxygen/ .