DXT3
The colour information in DXT3 is saved similar to the DXT1 format without an alpha channel. So a palette of 4 colours is used to represent 16 pixels of the texture. But the alpha information is stored with 4 bit in the DXT3 format. That means that in total 128 bit are used for the 16 pixels (32 for the palette, 32 for the colour indices and 64 for the alpha information). So this results in a compression ratio of 1:4.
Because the alpha information is stored in 4 bit, the smallest step in the alpha value that can be stored is 16. This means that very smooth transitions in the alpha channel can not be stored in the DXT3 format.
DXT5
The DXT5 formats differs from the DXT3 format in the way the alpha information is saved. The colour information is stored in the same way.
For the alpha information it uses a palette, similar to the way the colour information is also stored. This palette contains a minimum and maximum alpha value. Then 6 other alpha values are interpolated between this minimum and maximum. This thus allows more gradual changes of the alpha value.
A second variant does interpolate only 4 other alpha values between the minimum and maximum, but also adds an alpha value of 0 and 1 (for fully transparent and not transparent). For some textures this might give better results.