public documents.sextractor_doc

\section{Filtering}

\gam{Filtering suggests that some data will be left out. Shouldn't we call

  this ``Smoothing'' instead?}

\subsection{Convolution}

\label{filter-conv}

Detectability is generally limited at the faintest flux levels by the background noise.

The power-spectrum of the noise and that of the superimposed signal can be significantly different.

Some \index{gain} gain in the ability to detect sources may therefore be obtained simply through

appropriate \index{linear filtering} linear filtering of the data, prior to \index{segmentation} segmentation. In low density fields,

an optimal \index{convolution} convolution kernel $h$ (``matched filter'') can be found that maximizes

detectability. An estimator of detectability is for instance the \index{signal-to-noise ratio} signal-to-noise ratio

at source position $(x_0,y_{\,0}) \equiv (0,0)$:

\begin{equation}

\left[ \frac{\rm S}{\rm N}\right)^2 \equiv \frac{\left( (s * h)(x_0,y_{\,0}) \right]^2}

{\overline{(n * h)^2}}\,,

\end{equation}

where $s$ is the signal to be detected, $n$ the noise, and `$*$' the \index{convolution} convolution operator.

Moving to Fourier space, we get:

\begin{equation}

\left( \frac{\rm S}{\rm N}\right)^2 = \frac{\left(\int{{\cal S}{\cal H}\,d\omega}\right)^2}

                                        {\int{|{\cal N}|^2 |{\cal H}|^2\,d\omega}}\,,

\end{equation}

where ${\cal S}$ and ${\cal H}$ are the \index{Fourier-transforms} Fourier-transforms of $s$ and $h$, respectively, and

$|{\cal N}|^2$ is the power-spectrum of the noise.

\gam{This equation seems dimensionally correct only if $\omega$ is dimensionless.}

Remarking, using \index{Schwartz inequality} Schwartz inequality, that

\begin{equation}

\label{eq:schwartz1}

\left|\int{{\cal S}{\cal H}\, d\omega}\right|^2 \leq

                \int{\frac{|{\cal S}|^2}{|{\cal N}|^2} d\omega} \, \int{|{\cal N}|^2 |{\cal H}|^2

                d\omega}\,,

\end{equation}

we see that

\begin{equation}

\label{eq:schwartz2}

\left( \frac{\rm S}{\rm N}\right)^2 \leq \int{\frac{|{\cal S}|^2}{|{\cal N}|^2} d\omega}\,.

\end{equation}

Equality (maximum S/N) in (\ref{eq:schwartz1}) and (\ref{eq:schwartz2}) is achieved for

\begin{equation}

\frac{\cal S}{|{\cal N}|} \propto |{\cal N}| {\cal H}^*\,,\, {\rm that\, is}

\end{equation}

\begin{equation}

\label{eq:conv}

{\cal H} \propto \frac{{\cal S}^*}{|{\cal N}|^2}.

\end{equation}

In the case of white noise (a valid approximation for many astronomical \index{image} images, especially

\index{CCD} CCD ones), $|{\cal N}|^2 = \rm cst$; the optimal \index{convolution} convolution kernel for detecting \index{stars} stars is

then the \emph{point spread function}\footnote{The \index{PSF} PSF is the \index{convolution} convolution of

  the instrumental \index{PSF} PSF and the atmospheric seeing.} (PSF) flipped over the $x$ and $y$ directions. It may also be described as the

cross-correlation with the template of the sources to be detected (for more

details see e.g., \cite{bijaoui:dantel:1970}) \gam{missing a recent book citation here}.

There are of course a few problems with this method. First of all,

many sources of unquestionable interest, like galaxies, appear in a variety of shapes and scales

on astronomical \index{image} images.

A perfectly optimized detection routine should ultimately apply all relevant

\index{convolution} convolution kernels one after the other in order to make a complete catalogue. Approximations

to this approach are the (isotropic) \index{wavelet} wavelet analysis mentioned earlier, or the more empirical

\index{ImCat} ImCat algorithm (Kaiser \etal 1995),  both of which assume that

the sources are

reasonably round. The impact on \index{memory} memory usage and processing speed of such refinements is currently

judged too severe to be applied in {\sc SExtractor}. Simple filtering does a good job in general:

the topological constraints added by the \index{segmentation} segmentation process make the detection somewhat tolerant

towards larger objects. Extended, very Low-Surface-Brightness (LSB) features found in astronomical

\index{image} images are often artifacts (flat-fielding errors, optical ``ghosts'' or halos). However, it is

true that some of them can be genuine objects, like \index{LSB} LSB galaxies, or distant \index{galaxy clusters} galaxy clusters

buried in the background noise. For detecting those with software like {\sc SExtractor}, a

specific processing is needed (see for instance Dalcanton \etal 1997 and references therein). The

simplest way to achieve the detection of extended \index{LSB} LSB objects in {\sc SExtractor} is to work

on {\tt MINIBACK} \index{check-image} \index{check-images} check-images (see \S\ref{chap:miniback}).

A second problem may occur because of overlaps with other objects. Convolving with a low-pass

filter (the \index{PSF} PSF has no negative side-lobes) diminishes the contrast between objects, and makes

\index{segmentation} segmentation less effective in isolating individual sources. This can to some extent be recovered

by \index{deblending} deblending (see \S\ref{chap:deblending}). In severely crowded fields however, confusion noise

becomes the limiting factor for detection, and it is then advisable not to filter at all, or to

use a bandpass-filter (compensated filter \gam{what is this? One with

  negative side-lobes?}).

Finally, the \index{PSF} PSF can vary across the field. The \index{convolution} convolution mask

should ideally follow these variations in order to allow for optimal detection everywhere in the

\index{image} image. However, considering approximately-Gaussian \index{PSF} PSF cores and \index{convolution} convolution kernels,

detectability is a rather slow function of their \index{FWHM} FWHMs\footnote{Full-Width at Half-Maximum}: a

mismatch as large as 50\% between the kernel \index{FWHM} FWHM and that of the \index{PSF} PSF will lead to no more than a

10\% loss in peak S/N (Irwin 1985). Considering that \index{PSF} PSF variations are generally much smaller

than this, filtering in {\sc SExtractor} is limited to constant kernels.

\subsection{Non-linear filtering}

\label{filter-non-linear}

There are many situations in which \index{convolution} convolution is of little help:

filtering of (strongly) non-Gaussian noise, extraction of specific \index{image} image patterns,...

In those cases, one would like to extend the concept of a \index{convolution} convolution kernel to that of a more

general stationary filter, able for instance to mimic boolean-like operations on pixels. What

one wants is thus a mapping from ${\mathbf R}^n$ to ${\mathbf R}$ around each pixel. But the

more general the filter, the more difficult it is to design ``by hand'' for each case, specifying

how input pixel \#i should be taken into account with respect to input pixel \#j to form the

output, etc.. The solution to this is \index{machine-learning} machine-learning. Given a training set containing input and

output pixels, a \index{machine-learning} machine-learning software will adapt its internal parameters in order to minimize

a ``cost function'' (generally a $\chi^2$ error) and converge toward the desired mapping-function.

 These parameters can then for example be reloaded by a ``read-only'' routine to provide the

actual filtering.

{\sc SExtractor} implements this kind of ``read-only'' functionality in the form of the so-called

``retina filtering''. The {\sc \index{EyE} EyE}\footnote{{\em Enhance Your Extraction} \gam{URL?}} software (Bertin

1997) performs neural-network-learning on input and output \index{image} images to produce

``retina files''.

These files contain weights that describe the behaviour of the \index{neural network} neural network. The neural network

can thus be seen as an ``artificial retina'' that takes its stimuli from a small rectangular array

of pixels and produces a response according to prior learning (for more details, see the {\sc \index{EyE} EyE}

documentation). Typical applications of the retina are the identification of \index{glitch} \index{glitches} glitches.

\subsection{What is filtered, and what isn't}

Although filtering is a benefit for detection, it distorts profiles

and correlates the noise; it is therefore detrimental for most measurement tasks. Because of this,

filtering is applied ``on the fly'' to the \index{image} image, and {\em directly} affects only the detection

process and the isophotal parameters described in \S\ref{chap:isoparam}. Other catalogue parameters

are indirectly affected --- through the exact position of the \index{barycenter} barycenter and typical object extent

---, but the effect is considerably less. Obviously, in \index{double-image \index{mode} mode} double-image mode, filtering is only

applied to the {\em detection}\, \index{image} image.

\subsection{Image \index{boundaries} boundaries and \index{bad pixel} \index{bad pixels} bad pixels}

``Virtual'' pixels that lie outside \index{image} image \index{boundaries} boundaries are arbitrarily set to zero. This makes sense

since filtering occurs on a background-subtracted \index{image} image. When weighting is applied

(\S\ref{chap:weight}), \index{bad pixel} \index{bad pixels} bad pixels (pixels with weight $<$ {\tt WEIGHT\_THRESH}) are interpolated

by default (\S\ref{chap:interp}) and should therefore not cause much trouble. It is recommended

not to turn-off \index{interpolation} interpolation of \index{bad pixel} \index{bad pixels} bad pixels when filtering is on.

\subsection{Configuration parameters}

Filtering is triggered when the {\tt FILTER} keyword is set to {\tt Y}. If active, a file with name

specified by {\tt FILTER\_NAME} is searched for and loaded. Filtering with large retinas can be

extremely time consuming. In many cases, one is only interested in filtering pixels whose values

stand out from the background noise. The {\tt FILTER\_THRESH keyword} can be given to specify the

range of pixel values within which retina-filtering will be applied, in units of background noise

\index{standard deviation} standard deviation. If one value is given, it is interpreted as a lower \index{threshold} threshold. For instance:

\begin{verbatim}

FILTER_THRESH   3.0

\end{verbatim}

will allow filtering for pixel values exceeding $+3\sigma$ above the \index{local background} local background, whereas

\begin{verbatim}

FILTER_THRESH   -10.0,3.0

\end{verbatim}

will only allow filtering for pixel values between $-10\sigma$ and $+3\sigma$.

{\tt FILTER\_THRESH} has no effect on \index{convolution} convolution.

The result of the filtering process can be verified through a {\tt FILTERED} \index{check-image} check-image: see

\S\ref{chap:check}.

\subsection{CPU cost}

The {\sc SExtractor} filtering routine is particularly optimized for small kernels. It thus

provides a convenient way of filtering large \index{image} image data. On a 2GHz machine, a \index{convolution} convolution by a

$5\times5$ kernel will contribute less than 1 second to the processing time of a $2048\times4096$

\index{image} image. The numbers for non-linear (retina) filtering depend on the complexity of the neural

network, but can be a hundred times larger.

\gam{Update time?}

\subsection{Filter file formats}

As described above, two kinds of filter

files are recognized by {\sc SExtractor}: \index{convolution} convolution files (traditionaly suffixed with

``{\tt .conv}''), and ``retina'' files (``{\tt .ret}'' extensions\footnote{In {\sc SExtractor},

file name extensions are just conventions; they are not used by the software to distinguish

between different file formats.}).

Retina files are written exclusively by the {\sc \index{EyE} EyE} software, as \index{FITS binary-tables} FITS binary-tables.

Convolution files are in ASCII format. The following example shows the content of the

{\tt gauss\_2.0\_5x5.conv} file which can be found in the {\tt config/} sub-directory of the

{\sc SExtractor} distribution:

\begin{verbatim}

CONV NORM

# 5x5 \index{convolution} convolution mask of a gaussian \index{PSF} PSF with \index{FWHM} FWHM = 2.0 pixels.

0.006319 0.040599 0.075183 0.040599 0.006319

0.040599 0.260856 0.483068 0.260856 0.040599

0.075183 0.483068 0.894573 0.483068 0.075183

0.040599 0.260856 0.483068 0.260856 0.040599

0.006319 0.040599 0.075183 0.040599 0.006319

\end{verbatim}

The {\tt CONV} keyword appearing at the beginning of the first line

tells {\sc SExtractor} that the file contains the description of a

\index{convolution} convolution mask (kernel). It can be followed by {\tt NORM} if the

mask is to be normalized to 1 before being applied, or {\tt NONORM}

otherwise\footnote{If the sum of the kernel coefficients happens to be

exactly zero, the kernel is normalized to variance unity.}. The

following lines should contain an equal number of kernel coefficients,

separated by $<$space$>$ or $<$TAB$>$ characters. Coefficients in the

example above are read from left to right and top to bottom,

corresponding to increasing {\tt NAXIS1} ($x$) and {\tt NAXIS2} ($y$)

in the \index{image} image. Formatting is free, and number representations like {\tt

-0.14}, {\tt -0.1400}, {\tt -1.4e-1} or {\tt -1.4E-01} are equivalent.

The width of the kernel is set by the number of values per line, and

its height is given by the number of lines. Lines beginning with

``{\tt \#}'' are treated as comments.