.. _tut-data-conversion:

Tutorial 01: How to generate GazeParser data file
==================================================

GazeParser.Converter module includes conversion tools to generate GazeParser data file.
Currently, SimpleGazeTracker CSV file (\*.csv) and Eyelink EDF file (\*.edf) are supported.
GazeParser.Converter has TobiiToGazeParser() method, but it is under development and has only limited functions.

SimpleGazeTracker CSV to GazeParser
-------------------------------------

To generate GazePaser datafile from SimpleGazeTracker CSV file, use :func:`~GazeParser.Converter.TrackerToGazeParser` function.
In the following example, 'data.csv' is a SimpleGazeTracker data file to be converted (data.csv is in the current directory). ::

    >>> import GazeParser.Converter
    >>> GazeParser.Converter.TrackerToGazeParser('data.csv')

If this method run successfully, a file named 'data.db' is generated.
This file can be read with GazeParser.load() method.
TrackerToGazeParser accepts following optional parameters.

================= =======================================================================
Name               Description
================= =======================================================================
overwrite         By default, TrackerToGazeParser does not overwrite existing file.
                  If overwrite=True is specified, TrackerToGazeParser overwrites 
                  existing file. Default value is *False*.
config            Specify a GazeParser.Configuration.Config object where conversion
                  parameters are defined.  If no configuration object is specified
                  (i.e. config=None),
                  TrackerToGazeParser uses default GazeParser configuration file.
                  See :ref:`config-directory` and :ref:`config-gazeparser` for .
                  details of default GazeParser configuration file.
                  Default value is *None*
useFileParameters If this parameter is True, conversion parameters are overwritten by 
                  parameters recorded in SimpleGazeTracker data file.
                  Default value is *True*
================= =======================================================================

Figure 1 shows a flow chart of setting 'config' and 'useFileParameters'.

.. figure:: conversion001.png
    
    Figure 1

GazeParser has a viewer of GazeParser data file.  Executable file of the Viewer is app/Viewer.py in the GazeParser installation directory.
If you are not sure of GazeParser installation directory, start pyhton interpreter and type as following.::

    >>> import GazeParser
    >>> print GazeParser.__file__
    C:\Python27\lib\site-packages\gazeparser-0.6.0-py2.7.egg\GazeParser\__init__.pyc

In this case, C:\\Python27\\lib\\site-packages\\gazeparser-0.6.0-py2.7.egg\\GazeParser is the GazeParser installation directory.
Viewer.py is at C:\\Python27\\lib\\site-packages\\gazeparser-0.6.0-py2.7.egg\\GazeParser\\app\\Viewer.py.
If you installed GazeParser using win32 installer, you will find a shortcut for the Viewer in 'GazeParser' program group (Figure 2).

.. figure:: conversion002.png
    
    Figure 2

Start Viewer and choose [Open] from [File] menu.

.. figure:: conversion003.png
    
    Figure 3

Figure 4 shows an example of data. Raw gaze data is plotted at 'plot area'.
Abscissa and ordinate represents time (in msec) and gaze posiiton (in screen coordinate), respectively.
Vertical green lines represent inserted messages. Hatched pale bule areas represent detected saccades.
To pane or zoom plot, click 'pane/zoom button' and then left-drag (pane) or right-drag (zoom) on the plot area.

'Event list' is a list of fixation, saccades and messages.  By double-clicking an item on the event list, 
plot area is redrawn so that double-clicked event is at the center of the plot area.

.. figure:: conversion004.png
    
    Figure 4

In Figure 4, **obviously too many saccades are detected**. The range of the abscissa is 3000 msec.
Usually, saccades occur 3-4 times per second at most.
In addition, some saccades have very long duration for their amplitutde.
This is a typical conversion result when velocity threshold for saccade detection is too low.
In such a case, prepare a GazeParser configuration file and re-convert the data.
Copy the configuration file that was inserted when the data is recorded (See also :ref:`config-gazeparser`).
Suppose that SACCADE_VELOCITY_THRESHOLD in the configuration file is modified from 22.0 to 40.0.
Then the configuration file is saved as 'velthreshold40.cfg'.

To re-convert data.csv using velthreshold40.cfg, execute following command from python interpreter.::

    >>> newconfig = GazeParser.Configuration.Config('velthreshold40.cfg')
    >>> GazeParser.Converter.TrackerToGazeParser('data.csv', config=newconfig, overwrite=True, useFileParameters=False)

In GazeParser 0.6.1 or later, :func:`~GazeParser.Converter.TrackerToGazeParser` reads configuration file when name of the configuration file is passed as 'config'.::

    >>> GazeParser.Converter.TrackerToGazeParser('data.csv', config='velthreshold40.cfg', overwrite=True, useFileParameters=False)

.. note::
    You can directly edit inserted parameters in SimpleGazeTracker CSV file.
    In such a case, you don't have to create and load a new configuration file.
    If you want to re-convert a few files, this method is handy; however, if you have to 
    re-convert many files, probably it is easy to create a new configuration file.
    See :ref:`simplegazetracker-data-format053` for format of SimpleGazeTracker CSV file.

Figure 5 shows the reuslt of re-conversion. Saccades with unnaturally long duration are disappeared.

.. figure:: conversion005.png
    
    Figure 5

A value of 40.0 may be somewhat high as velocity threshold for saccade detection.
Appropriate threshould depends on measurement noise.
For example, when recording data with a high-speed camera, small measurement noises are likely to misdetected as a saccade.
The data used for this example was recorded at 400Hz and measurement noise constantly exceeded the threshold inserted to SimpleGazeParser CSV file (22.0 deg/s).
**It is recommended that record saccades and determine parameters suitable for your camera at first.**

GazeParser.Converter has several parameters besides SACCADE_VELOCITY_THRESHOLD.  For example, SACCADE_MINIMUM_DURATION prevents to detect too short saccades.
Low-pass filers can be applied to the data by setting FILTER_* parameters.
See :ref:`apply_lpf` for detail of low-pass fileter.


Eyelink EDF to GazeParser
-----------------------------------

Use :func:`~GazeParser.Converter.EyelinkToGazeParser` method to convert an Eyelink EDF file to GazeParser data file. ::

    import GazeParser.Converter
    GazeParser.Converter.EyelinkToGazeParser('data.edf')

EyelinkToGazeParser accepts the same parameters as those of TrackerToGazeParser except useFileParameters.
Saccades, fixations, blinks and events are imported from EDF file.
Therefore, parameters concerning saccade detection in GazeParser.Configuration object (e.g. SACCADE_VELOCITY_THRESHOLD) don't affect conversion result.
Parameters concerning display setting (e.g. SCREEN_ORIGIN, SCREEN_HEIGHT and SCREEN_WIDTH) affect data visualization with GazeParser data viewer.


Batch conversion
------------------------------------------------------------------

:func:`os.listdir` and :func:`os.walk` functions are useful to convert multiple SimpleGazeTracker CSV files.
For example, following script converts CSV files in 'data' directory.::

    #!/usr/bin/env python
    import os
    import GazeParser.Converter

    for file in os.listdir('data'):
        if os.path.splitext(file)[1].lower() == '.csv': # Is file extension 'csv'?
            GazeParser.Converter.TrackerToGazeParser(os.path.join('data',file))

To convert all CSV files in subdirectories, use :func:`os.walk`.::

    #!/usr/bin/env python
    import os
    import GazeParser.Converter
    
    for root, dirs, files in os.walk('data'):
        for file in files:
            if os.path.splitext(file)[1].lower() == '.csv': # Is file extension 'csv'?
                GazeParser.Converter.TrackerToGazeParser(os.path.join(root,file))

GazeParser data viewer can also be used to convert multiple files (see below).


Converting SimpleGazeTracker CSV to GazeParser with Data Viewer
------------------------------------------------------------------

If you use inserted parameters to convert SimpleGazeTracker CSV file, you can directly select SimpleGazeTracker CSV file from [Open] in the [File] menu.
If CSV file is not shown in the dialog, select 'SimpleGazeTracker CSV file (\*.csv)' as file type (Figure 6).

.. figure:: conversion006.png
    
    Figure 6

If corresponding .db file exists, a confirmation dialog for overwriting is presented.

.. figure:: conversion007.png
    
    Figure 7

To specify conversion parameters, open data conversion dialog from [Convert SimpleGazeTracker CSV] in [Convert] menu.

.. note::
    In version 0.6.0, menu item is mistyped as [Convert GazeTracker CSV].
    Data conversion dialog is a prototype and will be modified in future.

.. figure:: conversion008.png
    
    Figure 8

To customize conversion parameters, select 'Use following parameters' (box 1).
This radiobutton corresponds to useFileParameters optin of :func:`~GazeParser.Converter.TrackerToGazeParser`.
You can directly edit parameters or load GazeParser configuration file by clicking 'Load configuration file' (box 2).
To start conversion, clikc 'Convert files' (box 3).
If you want to overwrite existing .db files, check 'Overwrite' (box 4).

Multiple files in a directory can be selected at once (Figure 9).

.. figure:: conversion009.png
    
    Figure 9

An error dialog is presented if any errors occur during conversion.
Figure 10 shows a dialog which is presented when output file cannot be opened.
In many cases, the cause for this error is that 'Overwrite' is not checked.

.. figure:: conversion010.png
    
    Figure 10

If other error is presented, please confirm that appropriate parameters are set.
When all files have been processed, conversion results are reported (Figure 11).

.. figure:: conversion011.png
    
    Figure 11


.. _apply_lpf:

Applying low-pass filter
------------------------------------------------------------------

As sampling frequency becomes higher, measurement noise will disturb saccade detection more severely.
GazeParser.Converter has following built-in low-pass filters to reduce measurement noise.

================================== =============== ===========================================================
filter                             FILTER_TYPE     Description
================================== =============== ===========================================================
Moving average                     ma              A sequence of averages of N consecutive samples.
                                                   N is determined by **FILTER_SIZE**.
Butterworth filter                 butter          Applying an N-th order Butterworth low-pass filter.
                                                   Order is determined by **FILTER_ORDER**. Cut off frequency
                                                   is determined by **FILTER_WN**.  **FILTER_WN** must be
                                                   from 0.0 to 1.0. The value of 1.0 correspond to the
                                                   Nyquist frequency. That is, if sampling frequency is 
                                                   250Hz, the value of 1.0 corresponds to 125Hz.
Forwar-backward Butterworth filter butter_filtfilt Applying an N-th order *zero-phase* Butterworth low-pass
                                                   filter. *Zero-phase* means that no phase shift occurs in 
                                                   the output. The order and the cut off frecuency of the 
                                                   filter is determined by **FILTER_ORDER** and **FILTER_WN**,
                                                   respectively.
(No filter)                        identity        
================================== =============== ===========================================================

Figure 12 and 13 show raw data and outputs of these filters.
A curve labeled 'raw' represents raw data.
Curves labeled 'ma', 'butter' and 'butter_filtfilt' represent the output of corresponding filters (FILTER_SIZE=5, FILTER_WN=0.2 and FILTER_ORDER=3).

.. figure:: lpf001.png
    
    Figure 12

.. figure:: lpf002.png
    
    Figure 13

Figure 14 shows the effect of FILTER_WN on the output of the Butterworth filter.
Smaller FILTER_WIN yields in smoother output.
Notably, the output is delayed from the raw data due to phase shift of Butterworth filter.


.. figure:: lpf003.png
    
    Figure 14

Figure 15 shows the effect of FILTER_ORDER on the output of the Butterworth filter. 
Similar to FILTER_WN, larger FITER_ORDER yields smoother and delayed output.

.. figure:: lpf004.png
    
    Figure 15

In contrast, output of the zero-phase Butterworth filter is not delayed from the raw data.

.. figure:: lpf005.png
    
    Figure 16


.. _resampling_data:


Resampling data
------------------------------------------------------------------

Ideally, interval between successive samples is constant throughout recording.;
however, the interval fluctuates because of several reasons such as version of camera's driver, compatibility between camera and frame grabber and so on.
Because GazeParser's low-pass filter function assumes that the interval is constant, low-pass filter cannot be used if fluctuation of the interval is not negligibly small.

RESAMPLING option is added to GazeParser 0.6.6 to "rectify" data with fructuated interval. If positivie value is set to RESAMPLING attribute of :class:`~GazeParser.Configuration` object, :func:`~GazeParser.Converter.TrackerToGazeParser` resamples raw data at a frequency specified by RESAMPLING attribute. Following code makes GazeParser resample raw data at 200Hz before applying low-pass filter.::

    newconfig = GazeParser.Configuration.Config()
    newconfig.RESAMPLING = 200
    GazeParser.Converter.TrackerToGazeParser('data.csv', config=newconfig, useFileParameters=False)

Resampling is performed by using SciPy's linear interpolation function. Resampling rate should not be set to a value far heigher than the original frequency.