My first question is how old is your new heater and was there a problem before?
If the answer is somewhat new to new and it started after it was installed, take a flashlight light and look at all the new fittings that were used for the installation.
What you are discribing could very easily be a rust problem. Looking at the fitting should tell you if brass, copper, or galvanized was used. Be sure to carefully check out the threaded fitting going directly into the tank. There could be several configurations of fittings, but regardless, a threaded fitting of some type had to be used. If you see a black iron pipe fitting, that’s a problem and that’s your rust.