The old version of the manual used to include an explanation for fitting generalised linear models, including binomial and poisson distributions. It is not in the current user manual, but the explanations are in the beta manual, which you can get from Obtaining AS Reml.
The following explanation is based on emails by Bruce Southey and Arthur Gilmour.
How does one calculate heritability for binomial or poisson models?
Just as you would a normally distributed trait. This is in the old manuals for the logistic models. Basically you use the variance of the link function as the residual variance. For example, the variance of the logit link is pi*pi/3 (i.e., approximately 3.29).
Are there any good references regarding generalized mixed models, particularly for genetic analysis?
There are papers that use ASReml, like Southey et al. 2003. Journal of Animal Science 81:1399-1405. Another good source is Steve Kachman’s work. He will give a course at Iowa State University on AS Reml in summer 2005.
It seems to be easier starting on binary data and David Collett’s book Modelling Binary Data is an excellent starting point. Then work up from that.
How good are ASReml results for GLMM?
Generalised Linear Mixed Models (GLMM) procedures in ASReml (which use the Schall method) may be seriously biased when applied to animal models (which is why they have not been described in the User Guide). The situation for Sire (Family) models seems to be a bit better. It is not easy to quantify the bias in any particular case without doing a simulation study.
Arthur’s explanation for this problem is that as the mean incidence becomes more extreme, the likelihood of fixed classes having all 0’s or all 1’s increases, and then there is no genetic information available from that class. However, dropping such classes from the analysis, is likely to change the overall mean, hence the overall model. There are strategies for handling this, like making contemporary groups RANDOM at the lowest level, or collapsing classes.
Even if all fixed effects have a welled defined mean (not 0 or 1), the animal model is still fitting an effect for every individual and the only constraint on that effect being ± infinity is that it has a correlation with other effects which may have the opposite sign. My own studies 20 years ago suggested that there was a tendency for the estimated variance component to be biased down (it is too small).
Other formulations of the GLMM model (e.g. using a linear predictor ignoring the animal effect) have slightly different behaviour but ASReml only has the one option in this regard.
As the writer of ASReml I am not prepared to say you can confidently accept answers you might get from ASReml when estimating heritability on the liability scale from BINARY data. Sometimes they will be fine but I am not able to give the rules when that is the case. Apparently Bruce has had no problems with the situations he has encountered which is encouraging.
Log likelihood is not suitable for comparing binomial models
The LogL value reported by ASReml is for the analysis of the working variable. As this working variable changes between iterations and especially between models, the LogL values are not comparable - even to confirm convergence of a particular model.
ASReml also calculates a ‘variance heterogenity factor’, which is the Deviance/DF (the deviance is calculated for the underlying maximum likelihood model). In a GLM (no random effects) this deviance can be used to compare models. However, for GLMM (with random effects) it does not represent the total variation, because it ignores the part related to the random effects. So it also is unsuitable as a basis for testing in GLMM models.