One of the prospect in hardening SQL Server includes changing the port from default 1433 to a customized one.
And while making such changes, application teams unknowingly try to establish connections and in turn, we notice the below errors in SQL Server error log file-
Error: 17836, Severity: 20, State: 17.
Length specified in network packet payload did not match number of bytes read; the connection has been closed. Please contact the vendor of the client library.
Users can lose connectivity to SQL Server if such error message is reported in the SQL Server error log file.
This issue occurs when the Detect services running on non-standard ports option is enabled in the Discovery module. The event is logged because SQL Server 2005/8/8R2 is unable to interpret the incoming packet.
Perform a nslookup of the CLIENT IP Address that is listed in the error message and find out what computer it is that is connecting. Then you need to check that machine and determine what specifically is connecting to the SQL Server. You might get more infromation from doing a SQL Trace for the Errors and Warnings Event Class and have the ClientProcessID column in the trace data. When the error spikes, you might get the PID for the process that is connecting from that machine, and then you can find that process in Task Manager on that machine by adding the PID to the data displayed (View -> Select Columns).
If you have antivirus running on your system and SQL Server is not running default port then Disable the Detect services running on non-standard ports option in the scan configuration.
Hope this helps.