This was also against server version 1.15.2~0trusty

@mlucy can you take a look at this?

@danielmewes -- will do.

https://gist.github.com/jhstatewide/df4842f63efb20d7eb3e Here is a stack trace from the hanging program. I generated this with "jstack", hopefully it is of some use.

@jhstatewide -- does this happen every time?

@mlucy, yes, 100% reproducible with this script

Alright, planning to work on this next once my current changes are in. It's probably a result of JRuby having real threads and us not testing the multithreaded part of our driver in anything except CRuby.

@larkost -- once this is fixed, would it be possible to add JRuby to the list of Ruby versions we test with?

I have further information... I generated random strings at increasing lengths... the following script works fine until the document reaches ~60k bytes...

https://gist.github.com/jhstatewide/70ea290678ac2eda4ce1

It also works great on MRI... 200k+ and counting

Haven't investigated it, but it could be the query is filling up the TCP send buffer which the driver may not be handling correctly.

If this is the problem, it may be that changing the @socket.write(packet) in net.rb to @socket.sendmsg(packet) would solve it. That would block the query thread until all the data has made it into the TCP send buffer.

@Tryneus sendmsg is not implemented on JRuby but I can confirm that the hang is on that @socket.write

I reproduced this on newton using JRuby 1.17.19, and it appears that the @socket.write call hangs inside JRuby. The server read out the beginning of the query and is waiting for more, but the main thread of execution in the client doesn't resume sending it.

This would likely correspond to this area of the stack trace from @jhstatewide:

"main" #1 prio=5 os_prio=0 tid=0x00007ffb9c00a000 nid=0x15ae waiting for monitor entry [0x00007ffba38c2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.jruby.RubyThread.select(RubyThread.java:1264)
    - waiting to lock <0x00000000f60247d0> (a java.lang.Object)
    at org.jruby.RubyThread.select(RubyThread.java:1256)
    at org.jruby.RubyIO.waitWritable(RubyIO.java:1436)
    at org.jruby.RubyIO.fwrite(RubyIO.java:1500)
    at org.jruby.RubyIO.write(RubyIO.java:1410)
    at org.jruby.RubyIO$INVOKER$i$1$0$write.call(RubyIO$INVOKER$i$1$0$write.gen)
...

Perhaps the JRuby waitWritable implementation is broken. Trying to find if there's an alternative way to do this.

I wouldn't recommend using this as the solution, but on newton, changing the @socket.write to @socket.syswrite seems to work.

Ok, my last comment was a little misleading as I hadn't had a chance to fully test it. At this point, I'm willing to accept this as a solution (up in code review 2630).

The new Connection::send function is:

def send packet
  written = 0
  while written < packet.length
    written += @socket.syswrite(packet.slice(written, packet.length))
  end
end

It seems that JRuby's IO::write has a hanging problem with waiting on the socket to become writable again. If we use JRuby's IO::syswrite, it will not hang, but it may return partial writes. MRI does not appear to have either of these problems.

Tested with extremely large rows, resulting in many partial writes on JRuby, everything seems to be in working order.

Tested with JRuby 1.7.19 (built from source) and Ruby 2.0.0.

@Tryneus: I can confirm your findings. Both of my test cases now pass with your updates to net.rb. Thanks very much for the fast response!

👍

Fix has been approved and merged to next in commit 1f5dce2, and cherry-picked into v1.16.x in commit 5b6e928. Will be in release 1.16.3 (or earlier if we do a ruby driver release for this).

Thanks for reporting this, @jhstatewide, glad we could get to the root of it!

@AtnNn Can we push out an update of just the Ruby driver?

I'll do that now.

The new version of the ruby driver, 1.16.0.1, has ben released and includes this fix.

Cool, thank you @AtnNn