If you wants to find out whether a transaction is doomed, you are supposed to use xact_state, which would return -1 in this case.
However, xact_state() returns 0 if there is no active user-defined transaction. This means that if the transaction is a system transaction, you cannot tell whether the transaction is committable or not.
There are at least two context where an SQL module can execute in the context of a system-defined transaction: triggers and code run from INSERT-EXEC. See the repro
section for details.
Give that Books Online says that xact_state gives information about user-defined transactions only, the behaviour cannot be construed to be a bug in the true sense of
the word. However, the current behaviour limits the usefulness of the function, and makes it more or less unreliable. (You cannot tell as an SP programmer whether your proc will called from a trigger or INSERT-EXEC three years from now.) So I file this as engine bug,
for reconsideration for SQL 11.
But if the current behaviour is to stand, Books Online should point this out in all places where it discusses xact_state()