DOPE Data Extraction and Analysis

The following will attempt to describe a "fool-proof" method of plotting analysed data given a DOPE data tape (DAT). There are 3 main tasks to undertake:
 

1 Data extraction from DAT

2 Analysis of the binary data files

3 Plotting the data

 

1 Extracting Data from DAT

To start with you need to find which tape archive contains the data files that you want. To do this type
/sabre/doppler/data/logs/Tdate YYMMDD
where YYMMDD is the data associated with your data files. The response of Tdate will be to tell you the name of the appropriate tape log file AND open it with a nedit text edit window. There are 43 data tapes from the first DOPE system (which ran from May 1995 until about 08.00 UT on 30 October 1998) and the log files are named in the form tromXX.log (eg. trom12.log). Logs for the tapes of data from the second generation of DOPE (DOPE2) are designated tro-D2-XX.log.

Once you know the log file name select the appropriate DAT tape and insert it into an available DAT drive and assign the drive to your user account by typing

mtassign DAT#
To check which DAT drives are free, typing mtstat will list each and its status. Note that the DAT drives will only respond to commands given on the file server Ion rather than a work station. To connect to Ion type rsh ion.

The method of data extraction is straightforward if long-winded. Make sure you are in the directory where you want the data to be stored. The archive exists as a series of sequentially written (using the tar function) hourly data files which is named with the time that the data in it commences (ie. YYMMDDHHMMSS.bin). You need to forward the tape to the beginning of the first file that you want and extract them from there. Examination of the tape log file will tell you where on the tape your first file is. Simply find the line number where your file is located in the file and skip forward on the tape.

Example:
Your first file is the 764th on the log file trom32.log and you want to extract that and the following  6 hours worth of data. To do so type

mt -t /dev/DAT#nr fsf 763
/people/dope/bin/get_dope_data DAT# 6
where DAT# represents the drive you have assigned. The "fsf" part of the first command means file skip forward. Be aware that this method is not perfect. If you have any problems,  rewind the tape and start again by typing
mt -t /dev/DAT#nr rewind
When you have finished put the tape away and unassign your DAT drive using the mtunassign command.

This is the most fiddly part of the process of obtaining your DOPE data plot. We feel confident, however, that the effort will be worthwhile!

2 Analysing DOPE Data Files

The data are stored in binary format and must be analysed using special software written in C. The software program /people/dope/analysis/dope/dpfft is versatile enough to allow the user to specify which system DOPE1 or DOPE2 generated the data files and whether a standard DOPE analysis (Doppler shift and amplitude information) or high time resolution signal amplitudes are required. Although there is some flexibility in the time resolution of the analysed DOPE data, the "integration" interval must be a power of 2 to speed up the frequency analysis. Thus, the integration interval selected is usually 12.8 s (equivalent to 512 samples) or 25.6 s (1024 samples), which is based on the system data sampling rate of 40 Hz. Bear in mind that the analysis performs a spectral (Fourier) analysis on consecutive blocks of data samples and saves spectral information above a specified threshold to the analysed data file. Thus, increasing the time resolution causes a reduction in the frequency resolution and, therefore, in the amplitude of the waves detectable by the system.

The simplest command which will perform routine analysis of the data files is

/people/dope/bin/dpfft <first_file_time> -indir <your_data_directory> -outdir <analysed_data_directory>
which will analyse the data with an integration period of 12.8 seconds, applying a spectral amplitude threshold of 50% (3dB). User dope has default input and output directories of respectively /sabre/doppler/data/raw and /sabre/doppler/data/analysed, however a general user does not have write permission to these areas and must include the –in and –out flags to the program. The argument first_file_time is the name of the first file that you wish to analyse without its ".bin" extension and represents the time in UT when the data in the file starts (eg. 980516042346). The default for dpfft is to analyse data in 8 hour blocks and so it will attempt to access this file and the subsequent 7 hourly files if they are present.

The output files are assigned names in the form yymmddhhmmss.chnXX where XX represents the receiver or channel number sampled by DOPE. This ranges from 01-04 for DOPE2, although there were only two receivers in the original system.

Typing dpfft will invoke the program to print its usage statement to the user’s screen. This information includes details of all the flags which dpfft will accept as inputs and attempts to demonstrate in what forms the user can modify the analysis. This usage statement is reproduced below for your information.

Usage statement for dpfft:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Usage: dpfft <start time> [options]

where

<start time> is in the form YYMMDDHHMMSS - the file start time
-int <interval>         integration interval (based on radix2)
                        - 12.8 (def), 25.6, 51.2, 102.4 sec
-dope1                  analyse data from the first DOPE system
                        (do not use with -dope2)
-dope2                  analyse data from the current DOPE (DOPE2)
                        system (do not use with -dope1)
-hours <number>         number of files to analyse (max 8)
-threshold <thrs>       percentage for threshold value (default 50%)
-amp                    output amplitude information only (may not
                        be used with -int or -threshold options)
-indir <directory1>      alternative input data area
                        (default: /sabre/doppler/data/raw)
-outdir <directory2>    alternative output directory
                        (default: /sabre/doppler/data/analysed)
Examples:
 
dpfft 990101120345 -int 25.6


will analyse 8 hours of data from the current system using an
integration interval of 25.6 seconds commencing with file
990101120345.bin and for a threshold of 50%
 

dpfft 980304011512 -int 25.6 -dope1 -hours 4 -threshold 90


will analyse 4 hours of data for both receivers on the old system
commencing with file 980304011512.bin and for an integration period
of 25.6 sec and a threshold of 90%

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

3 Plotting the Analysed Data

Now that you have your analysed data file, you can set about plotting it. The software employed to do this is written in IDL (Interactive Data Language) and is designed as an add-on to the Go software written and maintained by the Radio and Space Plasma Physics group, University of Leicester, for plotting CUTLASS HF radar data. To run the software effectively, the user must first set up and run Go. Then, at the IDL prompt, type
@/people/dmw7/idl/Cutlass/go_extras
This command will compile the various routines necessary to manipulate and plot DOPE data files.

Generating Simple Plots:
To quickly plot an analysed data file for a particular receiver channel, first load the file and generate the plot:

dopefile, 'yymmddhhmmss.chnXX', dir=<your_data_directory>
plot_dope
The plot_dope command creates a two panel plot - the "Doppler trace" (frequency shift) and the signal amplitude, both as a function of time.

To generate a postscript version (ie. a printable copy) of this, type

ps_start, 'filename.ps', /bw
plot_dope
ps_close
which will generate a black and white postcript file called filename.ps. To print or view this file type respectively
ps_print
ps_preview
Note that specifying "/bw" implies that the file will by default be printed to a black and white printer.

Loading FVC data:
To load data from the DOPE Frequency to Voltage Converter (FVC):

fvcfile, 'yymmddhhmmssfvc.out', dir=<your_data_directory>

Adjusting Annotation:
The font size of the plot annotation can be increased by specifying it before generating the plot. A charsize in the range 0.7-1.0  is generally acceptable:

set_min_charsize, 0.8
Setting the format of the time axis annotation to hh:mm can be achieved by typing
set_time_format, 'hh:mm'
Summary Plots:
To quickly generate a plot showing the data from all 2/4 receivers simultaneously type:
dope_summary, 'yymmddhhmmss', dir=<your_data_directory>
This will (for the DOPE2 system) generate a two page plot - 4 Doppler traces followed by 4 signal amplitude plots.

Other Commands:
Plot a DOPE data panel in a predefined position:

plot_dope_panel, columns, rows, this_column, this_row, time=time, trace=trace, peaks=peaks, amp=amp, y=y, bar=bar,  no_x=no_x, high=high, low=low, int=int, autoscale=autoscale, fvc=fvc, smoothfvc=smoothfvc
where
columns, rows, this_column, this_row define the position of the plot on the page (see define_panel below),
time=[start,end] defines the time range to be plotted in the format HHMM,
/trace will overlay the peak trace (join together spectral maxima for consecutive integration intervals) of the Doppler trace,
/peaks does the same as /trace but without plotting the Doppler trace first,
/amp will plot the signal amplitude instead of the Doppler frequency shift (Doppler trace),
y=[upper,lower] defines the ordinate range
/bar will adjust the size of the plot to be the same as another one will a colour bar scale,
/no_x stops the abscissa (time) annotation from being plotted - useful in conjunction with the set_format, /sardines command
high=high_cutoff specifies the long period cut off if the peak trace is being filtered,
low=low_cutoff specifies the short period cut off if the peak trace is being filtered,
int=integration_interval specifies the data interval if the peak trace is being filtered,
/autoscale will force the ordinate axis to automatically scale to the exact range.
/fvc will plot the FVC data instead of the Doppler frequency shift (Doppler trace),
smoothfvc=interval_in_seconds specifies a smoothing interval in seconds for the FVC data. Otherwise it is plotted at the sampling rate (40 Hz)
Place a title on the DOPE plot with a specific subtitle
plot_dope_title, 'subtitle'
Print to screen information on the path (transmitter/receiver) appropriate to the current file
print_dope_info
Perform a Fourier power spectral analysis on the peak trace derived from the DOPE Doppler trace using an FFT routine.  It is important that the sample interval between consecutive data points is constant throughout the time series.
dopespec, time=time, int=int, fmax=fmax, fmin=fmin, high=high,low=low, no_x=no_x, normalise=normalise
where
time=[start,end] defines the time range to be plotted in the format HHMM,
int=integration_interval specifies the data interval if the peak trace is being filtered,
fmin=min_frequency is the lowest frequency component plotted (usually 0 Hz)
fmax=max_frequency is the highest frequency component plotted (default is 1/(2*int) Hz)
high=high_cutoff specifies the long period cut off if the peak trace is being filtered,
low=low_cutoff specifies the short period cut off if the peak trace is being filtered,
/no_x stops the abscissa (time) annotation from being plotted - useful in conjunction with the set_format, /sardines command
/normalise forces the spectrum to be normalised to a peak power of 1.0
Filter a time series without using dopespec. This will involve removing the mean from the data (in order to minimise the DC (0 Hz) component of the spectrum), tapering the data with a cosine bell function at the ends of the time series (in order to remove edge effects, which can generate multiple signal harmonics), and then filtering the data with a convolution routine. It is important that the sample interval between consecutive data points is constant throughout the time series.
(1) Remove DC Offset (mean)
mean=total(data_array)/float(n_elements(data_array))
data_array=data_array - mean

(2) Taper the Time Series
cosbel, numsamp, fdatain, fdataout

where
numsamp is the number of points in the array [=n_elements(fdatain)],
fdatain contains the input data,
fdataout contains the tapered data

(3) Filter Data
filter_time_series, data, interval,high=high, low=low

where
data is the array to be filtered,
interval is the sample interval in seconds,
high=high_cutoff specifies the long period cut off if the peak trace is being filtered,
low=low_cutoff specifies the short period cut off if the peak trace is being filtered
Many of the routines above are designed to be compatible with the routines in Go and some familiarity with this software would be advantageous. Important functions might be:
define_panel, columns, rows, this_column, this_row
which defines the position of a plot panel and is useful for routines which do not utilise panel position information (eg. dopespec can be forced to plot a spectrum at the specified position). For example,
define_panel, 1, 2, 0, 0
defines that there will be two plots stacked vertically and this plot will be on top (position 0,0).

Simple routines for loading magnetometer data are as follows:

magfile, 'filename', dir=<data_directory>
plot_mag, /all, time=[start, end]
plot_mag, filt=['x','y'], time=[start,end], high=high_cutoff, low=low_cutoff
These functions are desribed in detail in the Magnetometer Plotting section of the Go user guide.


If you require further help or information on DOPE please contact

Dr Darren Wright [Darren.Wright@ion.le.ac.uk] or Dr Tim Yeoman [Tim.Yeoman@ion.le.ac.uk]


This web page was created on 20th January 2000 by D. Wright
Modified on 25th January 2000