public documents.sextractor_doc

\section{Filtering}
 
\gam{Filtering suggests that some data will be left out. Shouldn't we call
  this ``Smoothing'' instead?}
 
\subsection{Convolution}
\label{filter-conv}
Detectability is generally limited at the faintest flux levels by the background noise.
The power-spectrum of the noise and that of the superimposed signal can be significantly different.
Some \index{gain} gain in the ability to detect sources may therefore be obtained simply through
appropriate \index{linear filtering} linear filtering of the data, prior to \index{segmentation} segmentation. In low density fields,
an optimal \index{convolution} convolution kernel $h$ (``matched filter'') can be found that maximizes
detectability. An estimator of detectability is for instance the \index{signal-to-noise ratio} signal-to-noise ratio
at source position $(x_0,y_{\,0}) \equiv (0,0)$:
\begin{equation}
\left[ \frac{\rm S}{\rm N}\right)^2 \equiv \frac{\left( (s * h)(x_0,y_{\,0}) \right]^2}
{\overline{(n * h)^2}}\,,
\end{equation}
where $s$ is the signal to be detected, $n$ the noise, and `$*$' the \index{convolution} convolution operator.
Moving to Fourier space, we get:
\begin{equation}
\left( \frac{\rm S}{\rm N}\right)^2 = \frac{\left(\int{{\cal S}{\cal H}\,d\omega}\right)^2}
					{\int{|{\cal N}|^2 |{\cal H}|^2\,d\omega}}\,,
\end{equation}
where ${\cal S}$ and ${\cal H}$ are the \index{Fourier-transforms} Fourier-transforms of $s$ and $h$, respectively, and
$|{\cal N}|^2$ is the power-spectrum of the noise. 
\gam{This equation seems dimensionally correct only if $\omega$ is dimensionless.}
Remarking, using \index{Schwartz inequality} Schwartz inequality, that
\begin{equation}
\label{eq:schwartz1}
\left|\int{{\cal S}{\cal H}\, d\omega}\right|^2 \leq
		\int{\frac{|{\cal S}|^2}{|{\cal N}|^2} d\omega} \, \int{|{\cal N}|^2 |{\cal H}|^2
		d\omega}\,,
\end{equation}
we see that
\begin{equation}
\label{eq:schwartz2}
\left( \frac{\rm S}{\rm N}\right)^2 \leq \int{\frac{|{\cal S}|^2}{|{\cal N}|^2} d\omega}\,.
\end{equation}
Equality (maximum S/N) in (\ref{eq:schwartz1}) and (\ref{eq:schwartz2}) is achieved for
\begin{equation}
\frac{\cal S}{|{\cal N}|} \propto |{\cal N}| {\cal H}^*\,,\, {\rm that\, is}
\end{equation}
\begin{equation}
\label{eq:conv}
{\cal H} \propto \frac{{\cal S}^*}{|{\cal N}|^2}.
\end{equation}
In the case of white noise (a valid approximation for many astronomical \index{image} images, especially
\index{CCD} CCD ones), $|{\cal N}|^2 = \rm cst$; the optimal \index{convolution} convolution kernel for detecting \index{stars} stars is
then the \emph{point spread function}\footnote{The \index{PSF} PSF is the \index{convolution} convolution of
  the instrumental \index{PSF} PSF and the atmospheric seeing.} (PSF) flipped over the $x$ and $y$ directions. It may also be described as the
cross-correlation with the template of the sources to be detected (for more
details see e.g., \cite{bijaoui:dantel:1970}) \gam{missing a recent book citation here}.
 
There are of course a few problems with this method. First of all,
many sources of unquestionable interest, like galaxies, appear in a variety of shapes and scales
on astronomical \index{image} images.
A perfectly optimized detection routine should ultimately apply all relevant
\index{convolution} convolution kernels one after the other in order to make a complete catalogue. Approximations
to this approach are the (isotropic) \index{wavelet} wavelet analysis mentioned earlier, or the more empirical
\index{ImCat} ImCat algorithm (Kaiser \etal 1995),  both of which assume that
the sources are
reasonably round. The impact on \index{memory} memory usage and processing speed of such refinements is currently
judged too severe to be applied in {\sc SExtractor}. Simple filtering does a good job in general:
the topological constraints added by the \index{segmentation} segmentation process make the detection somewhat tolerant
towards larger objects. Extended, very Low-Surface-Brightness (LSB) features found in astronomical
\index{image} images are often artifacts (flat-fielding errors, optical ``ghosts'' or halos). However, it is
true that some of them can be genuine objects, like \index{LSB} LSB galaxies, or distant \index{galaxy clusters} galaxy clusters
buried in the background noise. For detecting those with software like {\sc SExtractor}, a
specific processing is needed (see for instance Dalcanton \etal 1997 and references therein). The
simplest way to achieve the detection of extended \index{LSB} LSB objects in {\sc SExtractor} is to work
on {\tt MINIBACK} \index{check-image} \index{check-images} check-images (see \S\ref{chap:miniback}).
 
A second problem may occur because of overlaps with other objects. Convolving with a low-pass
filter (the \index{PSF} PSF has no negative side-lobes) diminishes the contrast between objects, and makes
\index{segmentation} segmentation less effective in isolating individual sources. This can to some extent be recovered
by \index{deblending} deblending (see \S\ref{chap:deblending}). In severely crowded fields however, confusion noise
becomes the limiting factor for detection, and it is then advisable not to filter at all, or to
use a bandpass-filter (compensated filter \gam{what is this? One with
  negative side-lobes?}).
 
Finally, the \index{PSF} PSF can vary across the field. The \index{convolution} convolution mask
should ideally follow these variations in order to allow for optimal detection everywhere in the
\index{image} image. However, considering approximately-Gaussian \index{PSF} PSF cores and \index{convolution} convolution kernels,
detectability is a rather slow function of their \index{FWHM} FWHMs\footnote{Full-Width at Half-Maximum}: a
mismatch as large as 50\% between the kernel \index{FWHM} FWHM and that of the \index{PSF} PSF will lead to no more than a
10\% loss in peak S/N (Irwin 1985). Considering that \index{PSF} PSF variations are generally much smaller
than this, filtering in {\sc SExtractor} is limited to constant kernels.
 
\subsection{Non-linear filtering}
\label{filter-non-linear}
There are many situations in which \index{convolution} convolution is of little help:
filtering of (strongly) non-Gaussian noise, extraction of specific \index{image} image patterns,...
In those cases, one would like to extend the concept of a \index{convolution} convolution kernel to that of a more
general stationary filter, able for instance to mimic boolean-like operations on pixels. What
one wants is thus a mapping from ${\mathbf R}^n$ to ${\mathbf R}$ around each pixel. But the
more general the filter, the more difficult it is to design ``by hand'' for each case, specifying
how input pixel \#i should be taken into account with respect to input pixel \#j to form the
output, etc.. The solution to this is \index{machine-learning} machine-learning. Given a training set containing input and
output pixels, a \index{machine-learning} machine-learning software will adapt its internal parameters in order to minimize
a ``cost function'' (generally a $\chi^2$ error) and converge toward the desired mapping-function.
 These parameters can then for example be reloaded by a ``read-only'' routine to provide the
actual filtering.
 
{\sc SExtractor} implements this kind of ``read-only'' functionality in the form of the so-called
``retina filtering''. The {\sc \index{EyE} EyE}\footnote{{\em Enhance Your Extraction} \gam{URL?}} software (Bertin
1997) performs neural-network-learning on input and output \index{image} images to produce
``retina files''.
These files contain weights that describe the behaviour of the \index{neural network} neural network. The neural network
can thus be seen as an ``artificial retina'' that takes its stimuli from a small rectangular array
of pixels and produces a response according to prior learning (for more details, see the {\sc \index{EyE} EyE}
documentation). Typical applications of the retina are the identification of \index{glitch} \index{glitches} glitches.
 
\subsection{What is filtered, and what isn't}
Although filtering is a benefit for detection, it distorts profiles
and correlates the noise; it is therefore detrimental for most measurement tasks. Because of this,
filtering is applied ``on the fly'' to the \index{image} image, and {\em directly} affects only the detection
process and the isophotal parameters described in \S\ref{chap:isoparam}. Other catalogue parameters
are indirectly affected --- through the exact position of the \index{barycenter} barycenter and typical object extent
---, but the effect is considerably less. Obviously, in \index{double-image \index{mode} mode} double-image mode, filtering is only
applied to the {\em detection}\, \index{image} image. 
 
\subsection{Image \index{boundaries} boundaries and \index{bad pixel} \index{bad pixels} bad pixels}
``Virtual'' pixels that lie outside \index{image} image \index{boundaries} boundaries are arbitrarily set to zero. This makes sense
since filtering occurs on a background-subtracted \index{image} image. When weighting is applied
(\S\ref{chap:weight}), \index{bad pixel} \index{bad pixels} bad pixels (pixels with weight $<$ {\tt WEIGHT\_THRESH}) are interpolated
by default (\S\ref{chap:interp}) and should therefore not cause much trouble. It is recommended
not to turn-off \index{interpolation} interpolation of \index{bad pixel} \index{bad pixels} bad pixels when filtering is on.
 
\subsection{Configuration parameters}
Filtering is triggered when the {\tt FILTER} keyword is set to {\tt Y}. If active, a file with name
specified by {\tt FILTER\_NAME} is searched for and loaded. Filtering with large retinas can be
extremely time consuming. In many cases, one is only interested in filtering pixels whose values
stand out from the background noise. The {\tt FILTER\_THRESH keyword} can be given to specify the
range of pixel values within which retina-filtering will be applied, in units of background noise
\index{standard deviation} standard deviation. If one value is given, it is interpreted as a lower \index{threshold} threshold. For instance:
\begin{verbatim}
FILTER_THRESH	3.0
\end{verbatim}
will allow filtering for pixel values exceeding $+3\sigma$ above the \index{local background} local background, whereas
\begin{verbatim}
FILTER_THRESH	-10.0,3.0
\end{verbatim}
will only allow filtering for pixel values between $-10\sigma$ and $+3\sigma$.
{\tt FILTER\_THRESH} has no effect on \index{convolution} convolution.
 
The result of the filtering process can be verified through a {\tt FILTERED} \index{check-image} check-image: see
\S\ref{chap:check}.
 
\subsection{CPU cost}
The {\sc SExtractor} filtering routine is particularly optimized for small kernels. It thus
provides a convenient way of filtering large \index{image} image data. On a 2GHz machine, a \index{convolution} convolution by a
$5\times5$ kernel will contribute less than 1 second to the processing time of a $2048\times4096$
\index{image} image. The numbers for non-linear (retina) filtering depend on the complexity of the neural
network, but can be a hundred times larger.
\gam{Update time?}
 
\subsection{Filter file formats}
As described above, two kinds of filter
files are recognized by {\sc SExtractor}: \index{convolution} convolution files (traditionaly suffixed with
``{\tt .conv}''), and ``retina'' files (``{\tt .ret}'' extensions\footnote{In {\sc SExtractor},
file name extensions are just conventions; they are not used by the software to distinguish
between different file formats.}).
 
Retina files are written exclusively by the {\sc \index{EyE} EyE} software, as \index{FITS binary-tables} FITS binary-tables.
 
Convolution files are in ASCII format. The following example shows the content of the
{\tt gauss\_2.0\_5x5.conv} file which can be found in the {\tt config/} sub-directory of the
{\sc SExtractor} distribution:
\begin{verbatim}
CONV NORM
# 5x5 \index{convolution} convolution mask of a gaussian \index{PSF} PSF with \index{FWHM} FWHM = 2.0 pixels.
0.006319 0.040599 0.075183 0.040599 0.006319 
0.040599 0.260856 0.483068 0.260856 0.040599 
0.075183 0.483068 0.894573 0.483068 0.075183 
0.040599 0.260856 0.483068 0.260856 0.040599 
0.006319 0.040599 0.075183 0.040599 0.006319 
\end{verbatim}
The {\tt CONV} keyword appearing at the beginning of the first line
tells {\sc SExtractor} that the file contains the description of a
\index{convolution} convolution mask (kernel). It can be followed by {\tt NORM} if the
mask is to be normalized to 1 before being applied, or {\tt NONORM}
otherwise\footnote{If the sum of the kernel coefficients happens to be
exactly zero, the kernel is normalized to variance unity.}. The
following lines should contain an equal number of kernel coefficients,
separated by $<$space$>$ or $<$TAB$>$ characters. Coefficients in the
example above are read from left to right and top to bottom,
corresponding to increasing {\tt NAXIS1} ($x$) and {\tt NAXIS2} ($y$)
in the \index{image} image. Formatting is free, and number representations like {\tt
-0.14}, {\tt -0.1400}, {\tt -1.4e-1} or {\tt -1.4E-01} are equivalent.
The width of the kernel is set by the number of values per line, and
its height is given by the number of lines. Lines beginning with
``{\tt \#}'' are treated as comments.