your issue is the DXT3. i recently tried applying various types of normal maps to the terrain, but i never got what you are seeing here.

Originally the normal map files are dxt5_nm im pretty sure. this produces a greyscale image when viewed in RGB. but the info lies on a specific channel. i think green and alpha channell. try save as the "nm" variant in photoshops.
----------------
From dxt 1,3,5 its all about how transparency is handled and how much compression is given to the alpha and general size of files. (alphas not necessarily being used for transparency via the shader etc)


DXT 1 with alpha
The above holds when your texture does not have any transparency, if you want to add a 1 bit alpha channel things change slightly. A 1 bit alpha channel means that the transparency is either on or off. To store this one of the colours of the colour palette will get the meaning completely transparent.

The other three colours are left to store the colour information of your pixels. So for the two colours stored in this case only one has to be interpolated, colour3 is 1/2 colour1 plus 1/2 colour2. This means even less resolution is left for different shades of colours.

DXT3
The colour information in DXT3 is saved similar to the DXT1 format without an alpha channel. So a palette of 4 colours is used to represent 16 pixels of the texture. But the alpha information is stored with 4 bit in the DXT3 format. That means that in total 128 bit are used for the 16 pixels (32 for the palette, 32 for the colour indices and 64 for the alpha information). So this results in a compression ratio of 1:4.

Because the alpha information is stored in 4 bit, the smallest step in the alpha value that can be stored is 16. This means that very smooth transitions in the alpha channel can not be stored in the DXT3 format.

DXT5

The DXT5 formats differs from the DXT3 format in the way the alpha information is saved. The colour information is stored in the same way.

For the alpha information it uses a palette, similar to the way the colour information is also stored. This palette contains a minimum and maximum alpha value. Then 6 other alpha values are interpolated between this minimum and maximum. This thus allows more gradual changes of the alpha value.

A second variant does interpolate only 4 other alpha values between the minimum and maximum, but also adds an alpha value of 0 and 1 (for fully transparent and not transparent). For some textures this might give better results.