The ASN.1 language is a good language for specifying protocols. It is widely used by the telecommunications industry. However it is seldom good for "Internet based" protocols. For example IP/TCP/UDP/PPP.
For these kinds of protocols you have to have some other approach. One approach is to write your own language specification (grammar) and a compiler for this. From my own experience this is a risky path because you have to think really carefully how you design the grammar and compiler.
Does anyone know of any open or free implementations for a grammar and compiler that is suitable for Internet based protocols?