Skip to content
This repository has been archived by the owner on Dec 18, 2018. It is now read-only.

rc1 INCOMPLETE_CHUNKED_ENCODING when using nginx #341

Closed
staff0rd opened this issue Nov 8, 2015 · 39 comments
Closed

rc1 INCOMPLETE_CHUNKED_ENCODING when using nginx #341

staff0rd opened this issue Nov 8, 2015 · 39 comments
Assignees
Milestone

Comments

@staff0rd
Copy link

staff0rd commented Nov 8, 2015

Opened this here also, but I'm getting the feeling this more specific to kestrel rc1.

I have a docker container that has an aspnet5-rc1 application inside, listening on port 5000. If I make a request directly to that port, the response is 200 and quick.

If I make a request via nginx, then the response data comes back immediately (200), but does not terminate and rather about a minute later the response fails with; (in chrome) ERR_INCOMPLETE_CHUNKED_ENCODING.

Error does not occur when using nginx with aspnet-beta7. Headers for both aspnet-beta7 and aspnet-rc1 are the same;

HTTP/1.1 200 OK
Server: nginx/1.9.6
Date: Sun, 08 Nov 2015 04:38:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
@Tratcher
Copy link
Member

I have a local repro that looks similar. If I throw an exception after the first response write then there is no chunked terminator and the connection is closed gracefully rather than reset.

            app.Run(async (context) =>
            {
                await context.Response.WriteAsync("hello world");
                throw new Exception("Test exception");
            });

The behavior is even stranger behind HttpPlatformHandler. It appears to suppress the chunking error and re-chunk the response including a terminator. Verify if it behaves correctly after kestrel changes to a reset.

@Tratcher Tratcher added the bug label Nov 10, 2015
@Tratcher Tratcher added this to the 1.0.0-rc2 milestone Nov 10, 2015
@staff0rd
Copy link
Author

@Tratcher any hints as to why I'm seeing this behind nginx reverse proxy but not when accessing it directly?

@Tratcher
Copy link
Member

@staff0rd That's odd. In my case the reverse proxy (HttpPlatformHandler) hid the behavior by re-chunking the response.

You just posted the headers, did any of the body make it through? Or did you do an explicit Flush to send the headers before an exception occurred?

@staff0rd
Copy link
Author

@Tratcher if you browse to http://test.atqu.in you can see the issue occur. If you browse to http://test.atqu.in:5000 you'll reach the same site directly (without reverse proxy) and you'll see the issue does not occur. As you'll see, the whole site is is delivered both times, but the connection stays open on the reverse-proxy for about a minute until it finally dies with INCOMPLETE_CHUNKED_ENCODING.

The code is here and is just a fresh MVC 6 beta8 project updated to rc1, so no extra flushes occur, nor are any exceptions thrown.

@Tratcher
Copy link
Member

@staff0rd Let's re-test your scenario after we get the identified bug fixed.

@halter73 halter73 assigned cesarblum and unassigned pakrym Nov 18, 2015
@cesarblum
Copy link
Contributor

The problem here is the same as what we're facing in #368: we're not sending an RST when we expect to (ConnectionControl.End(ProduceEndType.SocketDisconnect)).

Libuv currently does not expose the underlying socket enough for us to force an RST. Someone tried to submit a PR in the past but it seems like that never got in: joyent/libuv#498

Basically what needs to be done is to set SO_LINGER on the socket with a timeout of 0. That should force an RST.

@cesarblum
Copy link
Contributor

Forget my previous comment 😄

nginx makes HTTP/1.0 requests when acting as a proxy. There is a known issue for Connection: close requests (which HTTP/1.0 requests implicitly are): #406

There's an easy fix for putting Kestrel behind an nginx reverse proxy: set proxy_http_version 1.1; to have nginx send HTTP/1.1 requests.

@staff0rd
Copy link
Author

This is already configuring HTTP/1.1. Looking at the response headers of test.atqu.in also confirms that HTTP/1.1 is being used - same headers as in first post above.

@cesarblum
Copy link
Contributor

Can you try a different version of nginx? I repro'd your issue on 1.8 and found that using that setting fixed the problem. Then I switched to 1.9.6 and on that specific version the requests seem to take a long time to be served (but I didn't see the original issue), even when running on localhost. I don't see that happening on 1.9.7 though.

@staff0rd
Copy link
Author

Updated, same symptoms - 1.3min response time.

Response headers;

HTTP/1.1 200 OK
Server: nginx/1.9.7
Date: Thu, 19 Nov 2015 23:49:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

@cesarblum
Copy link
Contributor

I can hit that URL just fine. Wasn't that the URL where you were seeing the error?

@staff0rd
Copy link
Author

Seriously? I just tried chrome + firefox + edge. Same symptoms on all - page immediately pops up, but loading doesn't stop for 1 min. However, the page imediately pops up and stops loading if I go around nginx. (http://test.atqu.in:5000)

@cesarblum
Copy link
Contributor

Ohh, sorry, I didn't notice it was still loading 😞

What addresses are you binding to? I figured out why 1.9.6 was taking so long to forward the request to Kestrel. I was binding Kestrel to my IPv6 loopback interface only, but nginx was first trying to forward the request to the IPv4 interface. It took a minute to time out and try the IPv6 interface. Maybe something similar is happening in your setup?

@staff0rd
Copy link
Author

Do you mean this or something on the nginx side?

@cesarblum
Copy link
Contributor

Yep, that. Although you're already binding to an IPv4 interface... Can you try binding to localhost or * and see if that makes a difference?

@staff0rd
Copy link
Author

Its hosted in a docker container so I don't think that binding to localhost will work. What's the syntax for *, http://*:5000?

@cesarblum
Copy link
Contributor

That's correct.

@staff0rd
Copy link
Author

Nah, same result;

Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING

@paralin
Copy link

paralin commented Apr 8, 2016

@benaadams Any way a fix for this can be made for rc1? I don't think switching the entire stack to the AspCore dev versions is wise but I can apply a single patch to rc1.

@davidfowl
Copy link
Member

See #341 (comment) for a workaround:

proxy_set_header Connection keep-alive;

@amcdnl
Copy link

amcdnl commented May 23, 2016

@benaadams I'm experiencing similar to this in RC2. See: #636 (comment)

@muratg muratg reopened this May 23, 2016
@muratg muratg added 1 - Ready and removed 3 - Done labels May 23, 2016
@muratg muratg assigned mikeharder and unassigned cesarblum May 23, 2016
@muratg muratg modified the milestones: 1.0.0, 1.0.0-rc2 May 23, 2016
@cesarblum
Copy link
Contributor

#636 #734 seem to be the same as this one (actually #636 is about BrowserSync). Let's center the discussion here, since this was the original issue.

@amcdnl Which version of nginx are you using, and on which OS?

@halter73
Copy link
Member

This issue is related to Kestrel not sending the chunked response suffix. This was fixed by e4fd91b which closed this issue and #406.

Unless people are still seeing INCOMPLETE_CHUNKED_ENCODING in chrome after the nginx timeout, I would close this.

@amcdnl
Copy link

amcdnl commented May 23, 2016

@CesarBS I updated the details in #636 to reflect your questions.

@mikeharder
Copy link
Contributor

@amcdnl: Can we close this issue, or are you still seeing INCOMPLETE_CHUNKED_ENCODING with RC2?

@amcdnl
Copy link

amcdnl commented May 24, 2016

@mikeharder Good to go.

@halter73 halter73 modified the milestones: 1.0.0-rc2, 1.0.0 May 24, 2016
@halter73 halter73 assigned cesarblum and unassigned mikeharder May 24, 2016
@jmbalanag
Copy link

Im currently encoutering this issue:

"Microsoft.AspNetCore.Server.Kestrel": "1.0.0",

@davidfowl
Copy link
Member

@jmbalanag can you provide more details about your setup and provide specific repro steps?

@jmbalanag
Copy link

jmbalanag commented Jul 27, 2016

This is the problem code:
[Route("api/lib_fund_source")]
public ActionResult GetFundSource()
{
return Json(db.lib_fund_source.Where(x => x.is_active == true).Select(x => new { Id = x.fund_source_id, Name = x.name }));

    }

if i remove, Where(x => x.is_active == true) it works just fine.

I think Its no longer related with kestrel.

@muratg
Copy link
Contributor

muratg commented Jul 27, 2016

@jmbalang Yeah, doesn't look like a Kestrel issue

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests