%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% FOR PRINTING
\documentclass[
    11pt,
    a4paper,
    openright,
    cleardoubleplain,
    appendixprefix,
    chapterprefix,
    numbers=noenddot,
    DIV11,
    BCOR15mm
]{report}
               
\newcommand{\eztrace}{EZTrace}
\newcommand{\reporttitle}{\eztrace}               
\newcommand{\reportsubtitle}{A Generic Framework for\\ Instrumenting Applications}
\newcommand{\reportsubsubtitle}{User Manual}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{etex}
\usepackage{savesym}
\usepackage[utf8]{inputenc}

\usepackage[T1]{fontenc}
\usepackage[small,bf]{caption}

% for plotting 
\usepackage{tikz}
\usepackage{pgfplots}
\usepackage{subfigure}
\usepackage{graphicx}
\usepackage{color}
\usepackage{xcolor}
    \definecolor{ListingsKeywordColor}{rgb}{0,0,0.4}
    \definecolor{ListingsIdentifierColor}{rgb}{0,0.5,0}
    \definecolor{ListingsCommentColor}{rgb}{0.4,0.4,0.4}
    \definecolor{ListingsStringColor}{rgb}{0.6000,0.3333,0.7333}%{0.8,0,0}
    \definecolor{ListingsRuleSepColor}{rgb}{0,0,0}
    \definecolor{ListingsEmphColor}{rgb}{0,0.6667,0.6667}
    \definecolor{ListingsBreakSymbolColor}{rgb}{0.780,0.082,0.522}
    \definecolor{LinkColor}{rgb}{0,0,0.5}
    \definecolor{UnitColor}{rgb}{0,0,0}
    \definecolor{MathsVectorColor}{rgb}{0,0,0}
    \definecolor{MathsMatrixColor}{rgb}{0,0,0}
    \definecolor{MyGreen}{HTML}{228B22}
%     \definecolor{TikzGreen}{HTML}{}
    \definecolor{MyBlue}{HTML}{0000FF}
    \colorlet{MatrixElementsLight}{gray!20!white}
    \colorlet{MatrixElementsDark}{gray!40}

% for listing
\usepackage{listings}
  \lstset{
    basicstyle=\scriptsize\ttfamily,
    tabsize=3,
    showtabs=false,
    showspaces=false,
    showstringspaces=false,
    tab=\rightarrowfill,
    keywordstyle=\color{ListingsKeywordColor},
    identifierstyle=\color{ListingsIdentifierColor},
    commentstyle=\color{ListingsCommentColor},
    stringstyle=\color{ListingsStringColor},
    emphstyle=\color{ListingsEmphColor}\bfseries\underbar,
    frame=none,
    rulesepcolor=\color{ListingsRuleSepColor},
    numbers=left,
    numberstyle=\tiny,
    numbersep=5pt,
    captionpos=top,
    frame=tb,
    firstnumber=1,
    stepnumber=1,
%     numberfirstline=false,
    breaklines=true,
    breakatwhitespace=true,
%     prebreak=\mbox{\,$\color{ListingsBreakSymbolColor}\mathbf{\hookleftarrow}$},
    mathescape=true,
    morekeywords={},
}

\usepackage{url}
\usepackage[final]{hyperref}
\hypersetup{
   % Farben fuer die Links
   colorlinks=false,         % Links erhalten Farben statt Kaeten
   urlcolor=LinkColor,    % \href{...}{...} external (URL)
   filecolor=LinkColor,  % \href{...} local file
   linkcolor=LinkColor,  %\ref{...} and \pageref{...}
   menucolor=LinkColor,
   citecolor=LinkColor,
   % Links
   raiselinks=true,       % calculate real height of the link
   breaklinks,              % Links berstehen Zeilenumbruch
   %backref=page,            % Backlinks im Literaturverzeichnis (section, slide, page, none)
   %pagebackref=true,        % Backlinks im Literaturverzeichnis mit Seitenangabe
   verbose,
   hyperindex=true,         % backlinkex index
   linktocpage=true,        % Inhaltsverzeichnis verlinkt Seiten
   hyperfootnotes=false,     % Keine Links auf Fussnoten
   % Bookmarks
   bookmarks=true,          % Erzeugung von Bookmarks fuer PDF-Viewer
   bookmarksopenlevel=1,    % Gliederungstiefe der Bookmarks
   bookmarksopen=false,      % Expandierte Untermenues in Bookmarks
   bookmarksnumbered=true,  % Nummerierung der Bookmarks
   bookmarkstype=toc,       % Art der Verzeichnisses
   % Anchors
   plainpages=false,        % Anchors even on plain pages ?
   pageanchor=true,         % Pages are linkable
   % PDF Informationen
   pdftitle={\reporttitle: \reportsubtitle. \reportsubsubtitle},             % Titel
   pdfauthor={Roman Iakymchuk},            % Autor
   pdfcreator={LaTeX, hyperref, KOMA-Script},
   pdfstartview=Fit,       % Dokument wird Fit Height geaefnet
   pdffitwindow=true,
   pdfpagemode=UseOutlines, % Bookmarks im Viewer anzeigen
   pdfpagelabels=true,      % set PDF page labels
}

\usepackage{cleveref}
\Crefname{algorithm}{Alg.}{Algs.}
\Crefname{figure}{Fig.}{Figs.}
\Crefname{table}{Tab.}{Tabs.}
\Crefname{equation}{Eq.}{Eqs.}
\Crefname{eqnarray}{Eq.}{Eqs.}

\usepackage{float}
\newfloat{algorithm}{htb}{lob}
\floatname{algorithm}{Algorithm}

\newlength{\figlength}
\setlength{\figlength}{7.6cm} %8.2cm
\newlength{\subfiglength}
\setlength{\subfiglength}{7.3cm}%7.8

\newcommand{\becomes}{:=}
\newcommand{\bytes}{b}
\newcommand{\kb}{Kb}
\newcommand{\mb}{Mb}
\newcommand{\ghz}{GHz}
\newcommand{\flop}{Flop}
\newcommand{\flops}{Flops}
\newcommand{\gflops}{G\flops}
\newcommand{\svn}{SVN}
\newcommand{\fxt}{FxT}
\newcommand{\gtg}{GTG}
\newcommand{\mpi}{MPI}



% Title Page
\title{
{\Huge\bf \reporttitle{}}\\[6mm]
{\LARGE\bf \reportsubtitle}\\[12mm]
{\Large\bf \reportsubsubtitle}}
% \author{Roman Iakymchuk}

\begin{document}
\maketitle

\tableofcontents

\chapter{License of \eztrace}

\eztrace{} is a computer program whose purpose is to analyze the performance
 of parallel applications.

\eztrace{} is governed by the CeCILL-B license under French law and
abiding by the rules of distribution of free software.  You can  use,
modify and/ or redistribute the software under the terms of the CeCILL-B
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and  rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty  and the software's author,  the holder of the
economic rights,  and the successive licensors  have only  limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading,  using,  modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean  that it is complicated to manipulate,  and  that  also
therefore means  that it is reserved for developers  and  experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and,  more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL-B license and that you accept its terms.

\chapter{Overview of \eztrace{}}
\eztrace{}~\cite{Aulagnon12EZTrace,TraRue11Ccgrid,Coulomb10periscope} is a tool 
that aims at generating automatically execution traces from High-Performance 
Computing (HPC) programs. It generates execution trace files that can be 
interpreted by visualization tools such as ViTE~\cite{vite}.

\eztrace{} has been designed to provide a simple way to trace parallel 
applications. This framework relies on plugins in order to offer a generic way 
to analyze programs; depending on the application to analyze or on the point to 
focus on, several modules an be loaded. \eztrace{} provides pre-defined plugins 
that give the ability to the user to analyze applications that use \mpi{} 
libraries, OpenMP, or Pthreads. However, user-defined plugins can also be 
loaded in order to analyze application functions or custom libraries.

\chapter{Installation}
\section{Requirements}
In order to run \eztrace{}, the following software needs to be installed:
\begin{enumerate}
 \item autoconf of version 2.63;
 
 \item {[}Optional\,{]} Any \mpi{} implementation. Make sure your \mpi{} 
       installation provides dynamic libraries.
\end{enumerate}

We use custom versions of the following software:
\begin{enumerate}
 \item LiTL v.0.1 (\url{});
 \item \fxt{} v.0.2.10 (\url{http://download.savannah.gnu.org/releases/fkt/});       
 \item \gtg{} v.0.2 (\url{http://gforge.inria.fr/projects/gtg/}).
\end{enumerate}
Those are already included in \eztrace{}, but you can also provide your own 
versions.

\section{Getting \eztrace}
\begin{enumerate}
 \item You can get the latest stable release from the \eztrace{} website,
       \url{http://eztrace.gforge.inria.fr/};

 \item Current development version is available via \svn{}\\
       \texttt{svn checkout svn+ssh://scm.gforge.inria.fr/svn/eztrace}.
\end{enumerate}
After getting the latest development version (from \svn{}), you need to run\\
\hspace*{0.9cm}\texttt{./bootstrap}\\
and only then build the tool.

\section{Building \eztrace{}}
At first, you need to configure \eztrace{} by invoking the configure script:\\
\hspace*{0.9cm}\texttt{./configure -\,-prefix=<WHERE\_YOU\_INSTALL\_EZTRACE>}\\
Options to configure. You can pass several options to the configure script for
specifying where it should find the required libraries:\\
\begin{tabular}{@{\hskip .9cm}l}
\texttt{-\,-with-fxt=\$FXT\_ROOT} -- specify where \fxt{} is installed;\\
\texttt{-\,-with-gtg=\$GTG\_ROOT} -- specify where \gtg{} is installed;\\
\texttt{-\,-with-mpi=\$MPI\_ROOT} -- specify where \mpi{} is installed. 
 The \texttt{mpi.h}\\
\hspace*{0.4cm}file must be located in the 
 \texttt{\$MPI\_ROOT/include/directory};\\
\texttt{-\,-with-mpi-include=<PATH\_TO\_MPI.H>} -- specify the directory that\\
\hspace*{0.4cm}contains the \texttt{mpi.h} file.
\end{tabular}

Once \eztrace{} is configured, just run\\
\hspace*{0.9cm}\texttt{make}\\
\hspace*{0.9cm}\texttt{make install}


\chapter{How to Use \eztrace?}
\section{Recording Events}
\subsection{Selecting Functions to Instrument}
First, you should select functions in your application that you want to 
instrument. For this, you can set the \texttt{EZTRACE\_TRACE} environment 
variable to the list of plugins that should be used. For instance, set\\
\hspace*{0.9cm}\texttt{export EZTRACE\_TRACE="pthread mpi"}\\
if you want to instrument the PThread and \mpi{} functions.

You can get the list of available plugins using the \texttt{eztrace\_avail} 
command:\\
\hspace*{0.9cm}\texttt{eztrace\_avail}\\[1mm]
\begin{tabular}{@{\hskip 1.3cm}lll}
\tt 3 & \tt stdio & \tt Module for stdio functions (read, write,\\
 & & \tt select, poll, etc.)\\
\tt 2 & \tt pthread & \tt Module for PThread synchronization functions\\
 & & \tt (mutex, semaphore, spinlock, etc.)\\
\tt 6 & \tt papi & \tt Module for PAPI Performance counters\\
\tt 1 & \tt omp & \tt Module for OpenMP parallel regions\\
\tt 4 & \tt mpi & \tt Module for \mpi{} functions\\
\tt 5 & \tt memory & \tt Module for memory functions (malloc, free,\\
 & & \tt etc.)
\end{tabular}

You can get the list of selected modules with the \texttt{eztrace\_loaded} 
command:\\
\hspace*{0.9cm}\texttt{export EZTRACE\_TRACE="pthread mpi"}\\
\hspace*{0.9cm}\texttt{eztrace\_loaded}\\[1mm]
\begin{tabular}{@{\hskip 1.3cm}lll}
\tt 2 & \tt pthread & \tt Module for PThread synchronization functions\\
 & & \tt (mutex, semaphore, spinlock, etc.)\\
\tt 4 & \tt mpi & \tt Module for \mpi{} functions
\end{tabular}

\subsection{Generating Execution Traces}
Once the list of plugins is selected, you can run your application with 
\eztrace{}. For example:\\
\hspace*{0.9cm}\texttt{eztrace ./my\_program my\_arg1 my\_arg2}\\
This command line executes your program and generates a trace file in the 
\texttt{/tmp} directory (usually the file is named as 
\texttt{/tmp/<username>\_eztrace\_log\_\\rank\_<rank>}).

\subsection{Using \eztrace{} for \mpi{} Applications}
\eztrace{} needs to instrument each \mpi{} process. Thus, you can run the 
following command:\\
\hspace*{0.9cm}\texttt{mpirun -np nproc eztrace ./my\_program},\\
where {\tt npoc} is the number of \mpi{} processes. When your application ends. 
Each process writes a file named\\
\hspace*{0.9cm}\texttt{/tmp/<username>\_eztrace\_log\_rank\_<rank>}.

\subsection{Changing the Output Directory}
\label{sec:changing_dir}
By default, each process saves its trace in the local \texttt{/tmp} directory. 
You can change this by setting the \texttt{EZTRACE\_TRACE\_DIR} environment 
variable.

\section{Post-Mortem Analysis}
\subsection{Merging Execution Traces}
Once the execution traces are recorded, you can merge and convert them into a
file format that can be read by your visualization software:\\
\hspace*{0.9cm}\texttt{eztrace\_convert -o my\_paje.trace 
 /tmp/<username>\_eztrace\_}\\
\hspace*{1.3cm}\texttt{log\_rank\_0 /tmp/<username>\_eztrace\_log\_rank\_1}
This converts the trace files into the Paje format. If \gtg{} is installed with 
OTF support (this is enabled by default), you can choose to convert into the OTF
file format with the \texttt{-t OTF} option:\\
\hspace*{0.9cm}\texttt{eztrace\_convert -t OTF 
 /tmp/<username>\_eztrace\_log\_rank\_0}\\
\hspace*{1.3cm}\texttt{/tmp/<username>\_eztrace\_log\_rank\_1}

\subsection{Filtering Events}
You can select the plugins to use for the conversion phase by using the 
\texttt{EZTRACE\_TRACE} environment variable. For instance, if your traces 
contains \mpi{} and other events, then by setting \texttt{EZTRACE\_TRACE} to 
\texttt{mpi} and calling \texttt{eztrace\_convert} you will receive an output 
trace that contains only \mpi{} events.

\subsection{Computing Statistics}
Instead of creating a merged trace file, you can tell \eztrace{} to compute 
statistics on the recorded traces:\\
\hspace*{0.9cm}\texttt{eztrace\_stats /tmp/<username>\_eztrace\_log\_rank\_0}\\
\hspace*{1.3cm}\texttt{/tmp/<username>\_eztrace\_log\_rank\_1}\\[1mm]
\begin{tabular}{@{\hskip 1.7cm}l}
\tt [...]\\
\tt PThread:\\
\tt --------------\\
\tt 6 locks acquired\\
\\
\tt MPI:\\
\tt -----\\
\tt 27 messages sent\\
\tt MPI\_RECV: 10 calls\\
\tt MPI\_BARRIER: 11 calls\\
\tt 163 events handled
\end{tabular}

\section{User-defined Plugins}
\subsection{Defining Custom Plugins}
Since \eztrace{} works with plugins, you can create one and instrument the 
functions that you want. An example of a custom plugin is available in the 
\texttt{example} directory.

Once your plugin is created, you should tell \eztrace{} where to find it. For 
this, just set the \texttt{EZTRACE\_LIBRARY\_PATH} variable to the appropriate 
directory(-ies):\\
\hspace*{0.9cm}\texttt{export EZTRACE\_LIBRARY\_PATH=plugin1:plugin2}\\
\hspace*{0.9cm}\texttt{eztrace\_avail}\\[1mm]
\begin{tabular}{@{\hskip 1.3cm}lll}
\tt 3 & \tt stdio & \tt Module for stdio functions (read, write,\\
 & & \tt select, poll, etc.)\\
\tt 2 & \tt pthread & \tt Module for PThread synchronization func-\\
 & & \tt tions (mutex, semaphore, spinlock, etc.)\\
\tt 6 & \tt papi    & \tt Module for PAPI Performance counters\\
\tt 1 & \tt omp     & \tt Module for OpenMP parallel regions\\
\tt 4 & \tt mpi     & \tt Module for MPI functions\\
\tt 5 & \tt memory  & \tt Module for memory functions (malloc, free,\\
 & & \tt etc.)\\
\tt 99 & \tt plugin1 & \tt Example module for libplugin1\\
\tt 98 & \tt plugin2 & \tt Example module for the plugin2 library\\
\end{tabular}

\subsection{Generating Custom Plugins}
You can generate one plugin and instrument the functions you want.
In order to generate your plugin, you need to create a file containing:
\begin{enumerate}
 \item The name of the library you want to trace (\texttt{libNAME.so});
 \item {[}Optional{]} A brief description of the library;
 \item An ID to identify the module. 0 is reserved for \eztrace{} internal
  use. Thus, you can use any between 10 and ff;
 \item The prototype of the functions you want to instrument.
\end{enumerate}
As a result, your file should look as follow
\lstset{language=C, caption={\tt example.tpl}}
\begin{lstlisting}
BEGIN_MODULE 
NAME example_lib 
DESC "module for the example library"
ID 99
int example_do_event(int n)
double example_function1(double* array, int array_size)
END_MODULE
\end{lstlisting}

Now use \texttt{eztrace\_create\_plugin} to generate the plugin source
code:\\
\hspace*{0.9cm}\texttt{eztrace\_create\_plugin example.tpl}\\
\begin{tabular}{@{\hskip 1.3cm}l}
\tt New Module\\
\tt Module name: 'example\_lib'\\
\tt Module description: '"module for the example library"'\\
\tt Module id: '99'\\
\tt \hspace*{0.8cm} emulate record\_state for 'example\_do\_event'\\
\tt Function 'example\_do\_event' done\\
\tt \hspace*{0.8cm} emulate record\_state for 'example\_function1'\\
\tt Function 'example\_function1' done\\
\tt End of Module example\_lib
\end{tabular}\\
The source code is generated in the output directory. Just type:\\
\hspace*{0.9cm}\texttt{make}\\
Then, set the \texttt{EZTRACE\_LIBRARY\_PATH} to the appropriate directory. Now,
your custom plugin is ready to be used.

You can also determine (in the \texttt{example.tpl} file) the way a function is
depicted in the output trace. For instance,
\lstset{language=C, caption={}}
\begin{lstlisting}
int submit_job(int* array, int array_size)
BEGIN
 ADD_VAR("job counter", 1)
END
\end{lstlisting}
specifies that when the \texttt{submit\_job} function is called, the output
trace should increment the "job counter" variable. You can now track the value
of a variable.

The \texttt{test/module\_generator} directory contains several scripts that
demonstrate the various commands available.

\section{Environment Variables}
Here is a list of the environment variables that can be used for tuning 
\eztrace{}.
\begin{itemize}
 \item General-purpose variables:
  \begin{itemize}
   \item \texttt{EZTRACE\_TRACE\_DIR} specifies the directory in which trace 
    files are stored (by default it is \texttt{/tmp});
   \item \texttt{EZTRACE\_LIBRARY\_PATH} specifies the directories in which 
    \eztrace{} can find \eztrace{} modules (by default, it is none);
   \item \texttt{EZTRACE\_TRACE} specifies the list of \eztrace{} modules to 
    load (by default, it is the list of all available modules);
   \item \texttt{EZTRACE\_BUFFER\_SIZE} specifies the size of the buffer in 
    which \eztrace{} stores events (by default, the size is 32\,MB);
   \item \texttt{EZTRACE\_FLUSH} specifies the behavior of \eztrace{} when the 
    event buffer is full. If it is set to one, the buffer is flushed. This 
    permits to record traces that are larger than 
    \texttt{EZTRACE\_BUFFER\_SIZE}, but this may impact the application 
    performance. Otherwise, if it is set to zero, which is a default option,
    any additional event will be recorded. The trace is, thus, truncated and 
    there is no impact on performance.
  \end{itemize}
 \item Error-handling variables:
  \begin{itemize}
   \item \texttt{EZTRACE\_NO\_SIGNAL\_HANDLER} disables \eztrace{} signal 
    handling (by default, it is zero).
  \end{itemize}

 \item Hardware counters-related variables:
  \begin{itemize}
   \item \texttt{EZTRACE\_PAPI\_COUNTERS} selects hardware events to trace 
    using the PAPI library, e.g. export 
    \texttt{EZTRACE\_PAPI\_COUNTERS="PAPI\_L3\\\_TCM PAPI\_FP\_INS"}.
    Please note that the list of supported events as well as the number of 
    events, which can be traced simultaneously, vary depending on the processor 
    type. This information can be retrieved using \texttt{papi\_avail} and the 
    processor documentation.
  \end{itemize}
  
 \item MPI-related variables:
  \begin{itemize}
   \item \texttt{EZTRACE\_MPI\_DUMP\_MESSAGES} tells \eztrace{} to dump the 
    list of messages into a file. You can then compute your own statistics on 
    MPI messages.
  \end{itemize}    
\end{itemize}

\section{Known Limitations}
 \begin{itemize}

  \item If \eztrace{} is compiled with a particular \mpi{} implementation such 
   as OpenMPI, it will not work if you run your application with another, e.g. 
   MPICH2. So make sure your application uses the same \mpi{} implementation as 
   \eztrace{}.
 \end{itemize}
 

\chapter{\eztrace{} in Details}
\eztrace{} uses a two-phases mechanism for analyzing performance. During the 
first phase that occurs while the application is executed, functions are 
intercepted and events are recorded. After the execution of the application, 
the post-mortem analysis phase is in charge of interpreting the recorded events.
This two-phase mechanism permits the library to separate the recording of a 
function call from its interpretation. It thus allows the user to interpret a 
function call event in different ways depending on the point he/she wants to 
focus on. It also reduces the overhead of profiling a program; during the 
execution of the application, the analysis tool should avoid performing 
time-consuming tasks such as computing statistics or interpreting function calls.

During the execution of the application, \eztrace{} intercepts calls to the 
functions specified by plugins and records events for each of them. Depending 
on the type of functions, \eztrace{} uses two different mechanisms for 
the interception. The functions defined in shared libraries can be overridden 
using \texttt{LD\_PRELOAD}: When the \eztrace{} library is loaded, it retrieves 
the addresses of the functions to instrument. When the application calls one of
these functions, the version implemented in \eztrace{} is called. This function 
records events and calls the actual function. The \texttt{LD\_PRELOAD} 
mechanism cannot be used for functions defined in the application since there 
is no symbol resolution. In that case, \eztrace{} uses the 
DynInst~\cite{Buck00DynInst} tool for instrumenting the program on the fly. 
Using DynInst, \eztrace{} modifies the program to record events at the 
beginning and/or at the end of each function to instrument.

\eztrace{} is structurally divided into two parts: the \eztrace{} core and 
the \eztrace{} modules.

\section{\eztrace{} Core}
The \eztrace{} core is composed of several libraries such as the Fast Kernel and
User Trace (\fxt{})~\cite{Danjean05FxT} and the Generic Trace Generator 
(\gtg{})~\cite{Coulomb10periscope,gtg} libraries. 

\subsection{\fxt{}}
For recording events, \eztrace{} relies on the \fxt{} library. Each process 
being instrumented by \eztrace{} generates a trace file using \fxt{}. In order 
to keep the trace size as compact as possible, \fxt{} records events in a 
binary format that contains only the minimum amount of information: a 
timestamp, an event code, and optional parameters.

\subsection{\gtg{}}
During the post-mortem analysis phase, \eztrace{} browses the recorded events 
and interprets them. It can then generate statistics -- such as the length of 
messages, the duration of critical sections, etc.-- or create a trace file for 
visualizing the application behavior. For generating trace files, \eztrace{} 
relies on the \gtg{} library. \gtg{} provides an abstract interface for 
recording traces. This permits the application to use a single interface for 
creating traces in multiple formats. Thus, an application can generate PAJ\'E 
traces or OTF files without any modification.

Althought PAJ\'E and OTF are both traces format, they have some differences. 
Thus, adding a meaning to a raw \fxt{} event is the critical part and the event 
must be interpreted in a way that is conformed to the output format chosen by 
the user. Otherwise, the traces will not represent the data they should.

\section{\eztrace{} Modules}
Since \eztrace{} uses the two-phases mechanism, plugins are organized in two 
parts, accordingly: the description of the functions to instrument, and the 
interpretation of each function call. During the execution of the application, 
the first part of the plugin is in charge of recording calls to a set of 
functions. The second part of the plugin is in charge of adding semantic to the
trace. \eztrace{} provides plugins for major parallel programming libraries 
(MPI, OpenMP, PThread, etc.), but also allows user-defined plugins designed for 
custom libraries or applications. For example, the PLASMA linear algebra 
library~\cite{Agullo09Plasma} is shipped with an EZTRACE plugin.


\chapter{Frequently Asked Questions}
\begin{itemize}
 \item[\bf Q.] When I run my \mpi{} application with \eztrace{}, all the 
  processes generate the \texttt{/tmp/<username>\_eztrace\_log\_rank\_1} file. 
  What is going wrong?

 \item[\bf A.] This happens when \eztrace{} fails to intercept calls to 
  \texttt{MPI\_Init} or \texttt{MPI\_Init\_thread}. There can be several reasons
  for that:
  \begin{itemize}
   \item The \eztrace{} \mpi{} module was not compiled. For intercepting calls 
    to \mpi{} functions, you need the MPI module in your installation (look for
    the \texttt{EZTRACE\_ROOT/lib/libeztrace-autostart-\\mpi.so} file). If you 
    do not see that file, it means that something went wrong during the 
    configuration of \eztrace{}, so check for errors or warnings in the 
    \texttt{config.log} file.
   \item You specified the list of modules to use and the \mpi{} module was not 
    there. Check your \texttt{EZTRACE\_TRACE} variable or use 
    \texttt{eztrace\_\\loaded}.
  \end{itemize}
  If you still experience problems, please contact the \eztrace{} development 
  team and we will fix your problem.\\[1mm]
  
  \item[\bf Q.] What if I do not want to trace the whole application, but only 
   a part of it?

  \item[\bf A.] Then, you can call \texttt{eztrace\_start()} and 
   \texttt{eztrace\_stop()} specify in the code which part to trace. You will 
   need to add \texttt{\#include <eztrace.h>} and to link with 
   \texttt{libeztrace}. Afterwards, you can run your application as usual, i.e.
   \texttt{./my\_program my\_arg1}.\\[1mm]
  
  \item[\bf Q.] I need to trace my program while using GDB, how can I do that?

  \item[\bf A.] Just add the \texttt{-d} option to \eztrace{} to enable GDB:\\
   \hspace*{0.9cm}\texttt{eztrace -d ./my\_program my\_arg1 my\_arg2}\\
   Please note that this should be applied only when a bug occurs while using 
   \eztrace{}.\\[1mm]
  
  \item[\bf Q.] I want my trace to be saved in a specific directory, how can I 
   do that?

  \item[\bf A.] Please take a look in \Cref{sec:changing_dir}.\\[1mm]
  
  \item[\bf Q.] What if I do not care about OpenMP and I only want to see 
   \mpi{} communication?

  \item[\bf A.] You can set \texttt{EZTRACE\_TRACE} to the list of modules you 
   want to activate. By default, all the available modules are enabled, but you 
   can tell \eztrace{} to trace only \mpi{}, OpenMP, or both \mpi{} and OpenMPI 
   functions:\\[1mm]
   \begin{tabular}{@{\hskip 0.9cm}l}
    \tt export EZTRACE\_TRACE=mpi\\
    \tt export EZTRACE\_TRACE=omp\\
    \tt export EZTRACE\_TRACE="omp mpi"
   \end{tabular}\\[1mm]
  
  \item[\bf Q.] Can \eztrace{} generate an OTF trace file so that I can 
   visualize it with Vampir?

  \item[\bf A.] Yes, since \eztrace{} relies on GTG for writing output traces, 
   it can generate OTF trace files. When converting the trace with 
   \texttt{eztrace\_con\\vert}, just add the \texttt{-t OTF} option:\\
   \hspace*{0.9cm}\texttt{eztrace\_convert -t OTF 
     /tmp/<username>\_eztrace\_log\_\\}
   \hspace*{1.3cm}\texttt{rank\_0 /tmp/<username>\_eztrace\_log\_rank\_1}
\end{itemize}

\section{Troubleshooting}
In case you have not found an answer to your question in the FAQ section, you 
encounter a bug, or want some explanation about \eztrace{}, please contact and 
ask our development team:
\begin{itemize}
 \item On the development mailing list\\
       \url{https://gforge.inria.fr/mail/?group_id=2774}

 \item On our IRC channel:
  \begin{itemize}
   \item Server: \texttt{chat.freenode.net}
   \item Channel: \texttt{\#eztrace}
  \end{itemize}
\end{itemize}

\bibliographystyle{plain}
\cleardoublepage
\phantomsection
\addcontentsline{toc}{chapter}{Bibliography}
\small
\bibliography{references}
\normalsize

\end{document}
