Documentation 
The General TCM Encoder block implements trelliscoded modulation (TCM) by convolutionally encoding the binary input signal and mapping the result to an arbitrary signal constellation. The Signal constellation parameter lists the signal constellation points in setpartitioned order. This parameter is a complex vector with a length, M, equal to the number of possible output symbols from the convolutional encoder. (That is, log_{2}M is equal to n for a rate k/n convolutional code.)
If the convolutional encoder represents a rate k/n code, then the General TCM Encoder block's input must be a binary column vector with a length of L*k for some positive integer L.
This block accepts a binaryvalued input signal. The output signal is a complex column vector of length L. For information about the data types each block port supports, see Supported Data Types.
To define the convolutional encoder, use the Trellis structure parameter. This parameter is a MATLAB^{®} structure whose format is described in the section Trellis Description of a Convolutional Code in the Communications System Toolbox™ documentation. You can use this parameter field in two ways:
If you want to specify the encoder using its constraint length, generator polynomials, and possibly feedback connection polynomials, then use a poly2trellis command within the Trellis structure field. For example, to use an encoder with a constraint length of 7, code generator polynomials of 171 and 133 (in octal numbers), and a feedback connection of 171 (in octal), set the Trellis structure parameter to
poly2trellis(7,[171 133],171)
If you have a variable in the MATLAB workspace that contains the trellis structure, then enter its name as the Trellis structure parameter. This way is faster because it causes Simulink^{®} software to spend less time updating the diagram at the beginning of each simulation, compared to the usage in the previous bulleted item.
The encoder registers begin in the allzeros state. You can configure the encoder so that it resets its registers to the allzeros state during the course of the simulation. To do this, set the Operation mode to Reset on nonzero input via port. The block then opens a second input port, labeled Rst. The signal at the Rst port is a scalar signal. When it is nonzero, the encoder resets before processing the data at the first input port.
The trelliscoded modulation technique partitions the constellation into subsets called cosets so as to maximize the minimum distance between pairs of points in each coset.
Note When you set the Signal constellation parameter, you must ensure that the constellation vector is already in setpartitioned order. Otherwise, the block might produce unexpected or suboptimal results. 
As an example, the diagram below shows one way to devise a setpartitioned order for the points for an 8PSK signal constellation. The figure at the top of the tree is the entire 8PSK signal constellation, while the eight figures at the bottom of the tree contain one constellation point each. Each level of the tree corresponds to a different bit in a binary sequence (b_{3},b_{2},b_{1}), while each branch in a given level of the tree corresponds to a particular value for that bit. Listing the constellation points using the sequence at the bottom of the tree leads to the vector
exp(2*pi*j*[0 4 2 6 1 5 3 7]/8)
which is a valid value for the Signal constellation parameter in this block.
For other examples of signal constellations in setpartitioned order, see [1] or the reference pages for the MPSK TCM Encoder and Rectangular QAM TCM Encoder blocks.
Coding gains of 3 to 6 decibels, relative to the uncoded case can be achieved in the presence of AWGN with multiphase trellis codes [3].
MATLAB structure that contains the trellis description of the convolutional encoder.
In Continuous mode (default setting), the block retains the encoder states at the end of each frame, for use with the next frame.
In Truncated (reset every frame) mode, the block treats each frame independently. I.e., the encoder states are reset to allzeros state at the start of each frame.
In Terminate trellis by appending bits mode, the block treats each frame independently. For each input frame, extra bits are used to set the encoder states to allzeros state at the end of the frame. The output length is given by $$y=n\cdot (x+s)/k$$, where x is the number of input bits, and $$s=\text{constraintlength}1$$ (or, in the case of multiple constraint lengths, s =sum(ConstraintLength(i)1)). The block supports this mode for column vector input signals.
In Reset on nonzero input via port mode, the block has an additional input port, labeled Rst. When the Rst input is nonzero, the encoder resets to the allzeros state.
A complex vector that lists the points in the signal constellation in setpartitioned order.
The output type of the block can be specified as a single or double. By default, the block sets this to double.
Port  Supported Data Types 

Input 

Output 

[1] Biglieri, E., D. Divsalar, P. J. McLane, and M. K. Simon, Introduction to TrellisCoded Modulation with Applications, New York, Macmillan, 1991.
[2] Proakis, John G., Digital Communications, Fourth edition, New York, McGrawHill, 2001.
[3] Ungerboeck, G., "Channel Coding with Multilevel/Phase Signals", IEEE Trans. on Information Theory, Vol IT28, Jan. 1982, pp. 55–67.