System.Net.Http.HttpIOException: The response ended prematurely

[fatihsimsek]

We try to migrate our .NetFramework application to .NetCore 8.0.
Our Application process thousands of requests per minute.

After this migration, we get 30/40 errors per day. (different times / different load periods – It does not depends load traffic)
How do we fix this error? Do you have any suggestions about code?

Trace:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Code:

using (var serviceScope = SiriusServiceScopeFactory.Instance.CreateScope())
{
         var client = serviceScope.GetHttpClient(HttpClientConstants.CoreHttpClient);
         using (var httpRequest = new HttpRequestMessage(HttpMethod.Post, requestUri))
         {
                        SetHttpHeaders(httpRequest);

                        byte[] zippedBytes;
                        using (var ms = new MemoryStream())
                        {
                            CoreGZipUtils.Zip(ms, request);
                            zippedBytes = ms.ToArray();
                        }

                        HttpContent content = new ByteArrayContent(zippedBytes);
                        content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
                        
                        httpRequest.Content = content;
                        using (var cts = new CancellationTokenSource(timeout))
                        {
                            HttpResponseMessage response = null;
                            try
                            {
                                response = await client.SendAsync(httpRequest, cts.Token).ConfigureAwait(false);
                            }
                            catch (TaskCanceledException ex)
                            {
                                throw new SrSystemException(string.Empty, string.Format("Timeout exception occured while connecting to plm. PLM Uri:{0}, Service Name:{1}, Timeout:{2} second", this.plmUri, this.corePlmClientHeaderInfo.ServiceName, timeout));
                            }
                            return await ReadHttpResponseFromMessageAsync<TResponse>(response).ConfigureAwait(false);
                        }
        }
}

internal static IServiceCollection AddCoreHttpClient(this IServiceCollection services)
{
    services.AddHttpClient(HttpClientConstants.CoreHttpClient).ConfigurePrimaryHttpMessageHandler(x =>
          new HttpClientHandler
          {
              AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
              UseCookies = false,
              AllowAutoRedirect = false,
              UseDefaultCredentials = true,
          });

    return services;
}

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[karelz]

Looks like the server sent invalid response, or perhaps closed the connection too early? (see The response ended prematurely).
I recommended via email to collect Wireshark traces and dig deeper.

[wfurt]

Is it transient e.g. when you repeat identical request it works or is it specific to some particular payload?

Do you have access to the server and can you get logs for the requests? We've seen similar errors in cases when server rejects the request before it is completly sent. Since this is POST one example may be size limits. For that, using 100 expect/continue may resolve it. On occasion, we also seen servers that send early error code and close the connection. In either case the request would not succeed.

I think the difference is that .NET Framework does not read from server while sending the request so it would not know. We talk about surfacing some info if the server sends error response early but HttpClient does not handle that very well at the moment.

Getting Wireshark may help but it is less useful for encrypted HTTPS traffic - unless you have option to decrypt. It seems like this is also authenticated access and in case of NTLM/Negotiate there may be several rounds working on authentication before the POST actually completes. #104745

[fatihsimsek]

It is transient and it works on bigger request bodies. We could access server and could not see any traces on server access logs. Our server is Linux and client is Windows server so (UseDefaultCredentials = true) property is sensible or not? Linux server does not use credential infos this request sent.
I try to delete setting of UseDefaultCredentials but result is not changed :(