Sunday, November 4, 2007

Is Werewolf Killing the Conference Hackfest?

The latest craze at conferences, especially those associated with O'Reilly or Ruby, is the game Werewolf (historically known as Mafia, but Werewolf has become more popular). The premise of the game is simple: (from Wikipedia's Mafia article)
Mafia (also known under the variant Werewolf or Vampire) is a party game modeling a battle between an informed minority and an uninformed majority. Mafia is usually played in groups with at least five players. During a basic game, players are divided into two teams: 'Mafia members', who know each other; and 'honest people', who generally know the number of Mafia amongst them. The goal of both teams is to eliminate each other; in more complicated games with multiple factions, this generally becomes "last side standing".
Substitute "villagers" for "honest people" and "werewolves" for "mafia members" and you get the general idea. They're the same game.

Again from Wikipedia:
Mafia was created by Dimma Davidoff at the Psychological Department of Moscow State University, in spring of 1986, and the first players were playing in classrooms, dorms, and summer camps of Moscow University. [citation needed] The game became popular in other Soviet colleges and schools and in the 1990s it began to be played in Europe (Hungary, Poland, England, Norway) and then the United States. Mafia is considered to be one of "the 50 most historically and culturally significant games published since 1800" by about.com.
So you get the idea.

It's a fun game. I first played it at Foo Camp 2007, and I've played at each Ruby-related conference since then. It's a great way to get to know people, and an interesting study in social dynamics. I have my own opinions about strategy and in particular a fatal flaw in the game, but I'll leave those for another day. Instead, I have a separate concern:

Is Werewolf killing the conference hackfest?

Last year at RubyConf, Nick Sieger and I sat up until 4AM with Eric Hodel, Evan Phoenix, Zed Shaw and others hacking on stuff. Eric showed a little Squeak demo for those of us who hadn't used it, Zed and I talked about getting a JRuby-compatible Mongrel release out (which finally happened a year later) and I think we all enjoyed the time to hack with a few others who code for the love of coding.

As another example, RubyGems, the definitive and now official packaging system for Ruby apps and libraries, was written during a late-night hackfest at an early RubyConf (2002?). I'm not remembering my history so well, but I believe Rake had a similar start, as well as several other projects including the big Rails 1.0 release's final hours.

This year, and for the past several conferences, there's been a significant drop in such hackfests. And it's because of Werewolf.

Immediately after the Matz's keynote last night, many of the major Ruby players sequestered themselves in isolated groups to play Werewolf for hours (and yes, I know many did not). They did not write the next RubyGems. They did not plant the seeds of the next great Ruby web framework. They did not advance the Ruby community. They played a game.

Don't get me wrong...I have been tempted to join every Werewolf game I can. I enjoy the game, and I feel like I'm at least competent at it. And I can appreciate wanting to blow off steam and play a game after a long conference day. I often feel the same way.

But I'm worried about the general trend. Not only does Werewolf seem to be getting more popular, it seems to draw in many of the best and brightest conference attendees, attendees who might otherwise be just as happily hacking on the "hard problems" we still face in the software world.

So what do you think? Is this a passing trend, or is it growing and spreading? Does it mean the doom of the late-night, post-conference hackfest and the inspirational products it frequently produces? Or is it just a harmless pastime for a few good folks who need an occasional break?

Friday, November 2, 2007

Top Five Questions I Get Asked

I'm at RubyConf this weekend, the A-list gathering of Ruby dignitaries. Perhaps one day I'll be considered among them.

At any rate, here's the top five questions I get asked. Perhaps this will help people move directly on to more interesting subjects :)

#1: Are you Charles Nutter

Yes, I am. I look like that picture up in the corner of my blog.

#2: How's life been at Sun/How's Sun been treating you?

Coming to Sun has been the best professional experience of my life. When they took us in over a year ago, many worried that they'd somehow ruin the project or steer it in the wrong direction (and I had my occasional concerns as well). The truth is that they've remained entirely hands-off with regards to JRuby while simultaneously putting a ton of effort into NetBeans Ruby support and committing mission-critical internal projects to running on JRuby. Beyond that, I've become more and more involved in the growing effort to support many languages on the JVM, through changes to Java and the JVM specification, public outreach, and cultivating and assisting language-related projects. It's been amazing.

Along with this question I often get another: "What's it like to work at Sun?" Unfortunately, this one is a little hard for me to answer. Tom Enebo and I work from our homes in Minneapolis, so "working at Sun" basically means hacking on stuff in my basement for 16 hours a day. In that regard, it's like...great.

#3 What's next for JRuby?

Usually this comes from people who haven't really been following JRuby much, and that's certainly understandable. We had our 1.0 release just five months ago, and we've had two maintenance releases since then plus a major release coming out in beta this weekend. So there's a lot to keep track of.

JRuby 1.0 was released in RC form at JavaOne 2007 and in final form at Ruby Kaigi 2007 in Tokyo. It was focused largely on compatibility, but had early bits of the Ruby-to-bytecode compiler (perhaps 25% complete). Over the summer, we simultaneously worked on improving compatibility (adding missing features, fixing bugs) and preparing for another major release (with me spending several months working on the compiler and various performance optimizations). That leads us to JRuby 1.1, coming out in beta form this weekend. JRuby 1.1 represents a huge milestone for JRuby and for Ruby in general. It's now safe to say it's the fastest way to run Ruby code in a 1.8-compatible environment. It includes the world's first Ruby compiler for a general-purpose VM, and the first complete and working 1.8-compatible compiler of any kind. And it has finally reached our largest Rails milestone to date: it runs Rails faster than Ruby 1.8.x does.

What comes after JRuby 1.1? Well, we need to finally clean up our association/integration with the other half of JRuby: Java support. Java integration works great right now--probably good enough for 90% of users. But JRuby still doesn't quite fit into the Java ecosystem as well as we'd like. Along with continuing performance and compatibility work, JRuby 2.0 should finally run the last mile of Java integration. Then, maybe we'll be "done"?

#4 What do you think of [Ruby implementation X]?

I usually hear this as "what bad things do you want to say about the other implementations", even though people probably don't intend it that way. So, for the record, here's my current position on the big five alternative Ruby implementations.

Rubinius: Evan and I talk quite a bit about implementation design, challenges of implementing Ruby, and so on. JRuby uses Rubinius's specs for testing, and Rubinius borrows/ports many of our tests for their specs. I've even made code contributions to Rubinius and Sun sponsored the Ruby Hackfest in Denver this past September. We're quite cozy.

Technical opinion? It's a great idea to implement Ruby in Ruby...I'd rather use Ruby than Java in JRuby whenever possible. However the challenge of making it perform well is a huge one. When all your core classes are implemented in Ruby, you need your normal Ruby execution to be *way* faster than Ruby 1.8 to have comparable application performance...like orders of magnitude faster. Think about it this way: if each core class method makes ten calls to more core class methods, and each of those methods calls ten native methods (finally getting to the VM kernel), you've got 100x more method invocations than an implementation with a "thicker" kernel that's immediately a native call. Now the situation probably isn't that bad in Rubinius, and I'm sure Evan and company will manage to overcome performance challenges, but it's a difficult problem to solve writing a VM from scratch.

We all like difficult problems, don't we?

XRuby: XRuby is the "other" implementation of Ruby for the JVM. Xue Yong Zhi took the approach of writing a compile-only implementation, but that's probably the only major difference from the JRuby approach. XRuby has similar core class structure to JRuby, and when I checked a few months ago, it could run some benchmarks faster than JRuby. Although it's still very incomplete, it's a good implementation that's probably not getting enough attention.

Xue and I talk occasionally on IM about implementation challenges. I believe we've both been able to help each other.

Ruby.NET: Ruby.NET is the first implementation of Ruby for the CLR, and by far the most complete. Wayne Kelly and John Gough from the University of Queensland ran this Microsoft-funded project when it was called "Gardens Point Ruby.NET Compiler", and Wayne has continued leading it in its "truly open source" project form on Google Code. I exchanged emails many times with Wayne about strategies for opening up the project, and I involved myself in many of the early discussions on their mailing list. I'm also, oddly enough, listed as a project owner on the project page. I'd like to see them succeed...more Ruby options means Ruby gains more validation on all platforms, and that means JRuby will be more successful. Ironic, perhaps, that long-term Ruby and JRuby success depends on all implementations doing well...but I believe that to be true.

IronRuby: IronRuby is the current in-progress implementation of Ruby for the CLR, primarily being developed by Microsoft's John Lam and his team. IronRuby is still very early in their implementation, but they've already achieved one big milestone: they have their Ruby core classes available as a public, open-source project on RubyForge, and you, dear reader, can actually contribute. In my book, it qualifies as the first "real" open source project out of Microsoft since it finally recognizes that OSS is a two-way street.

Technically, IronRuby is also an interesting project because it's stretching Microsoft's DLR in some pretty weird directions. DLR was originally based on IronPython, and it sounds like there's been growing pains supporting Ruby. Ruby is a much more complicated language to implement than Python, and from what I've heard the IronRuby team has had to diverge from DLR in a few areas (perhaps temporarily). But it also sounds like IronRuby is helping to expand DLR's reach, which is certainly good for MS and for folks interested in the DLR.

John and I talk at conferences, and I monitor and participate in the IronRuby mailing list and IRC channel. We Rubyists are a close-knit bunch.

Ruby 1.9.1/YARV: Koichi Sasada has done the impossible. He's implemented a bytecode VM and compiler for Ruby without altering the memory model, garbage collector, or core class implementations. And he's managed to deliver targeted performance improvements for Ruby that range from two times to ten times faster. It's an amazing achievement.

Ruby 1.9.1 will finally come out this Christmas with Koichi's YARV as the underlying execution engine. I talk with Koichi on IRC frequently, and I've been studying his optimizations in YARV for ideas on how to improve JRuby performance in the future. I've also offered suggestions on Koichi's Fiber implementation, resulting in him specifying a core Fiber API that can safely be implemented in JRuby as well. Officially, Ruby 1.9.1 is the standard upgrade path for Rubyists interested in moving on from Ruby 1.8 (and not concerned about losing a few features). JRuby will support Ruby 1.9 execution in the near future, and we already have a partial YARV VM implementation. JRuby and YARV will continue to evolve together, and I expect we'll be cooperating closely in the future.

#5 How's JRuby's Performance?

This is the first question that finally gets into interesting details about JRuby. JRuby's performance is great, and improving every day. Straight-line execution is now typically 2-4x better than Ruby 1.8.6. Rails whole-app performance ranges from just slightly slower to slightly better, though there have been reports of specific apps running many times faster. There's more work to do across the board, but I believe we've finally gotten to a point where we're comfortable saying that JRuby is the fastest complete Ruby 1.8 implementation available.

See also my previous posts and posts by Ola Bini and Nick Sieger for more information on performance. There's more detail there.

Friday, October 26, 2007

Easy JRuby 1.0.2 Bugs

For those of you looking to start helping JRuby along, here's a few open 1.0.2 bugs that would be pretty easy for a newb to look at. They'd also help you get your first taste of JRuby internals. Good eatin!

This should help get you started:
FYI: Most of these are also 1.1 bugs, so you'll be helping two releases at once (provided you fix them and provide patches for both branches, of course!)

This is fairly simple, isolated core class code. Have a look at src/org/jruby/RubyArray.java

There's a patch here that looks pretty clean, but needs updating to apply cleanly to trunk and 1.0 branch.

We have File.sysopen defined, but for some reason we don't have IO.sysopen. If you can't fix it, at least do some research as to *why*.

This one throws a NullPointerException, which are always really ugly, really broken, and usually really easy to figure out and fix. The spec in question is our (oldish) copy in test/externals/rubinius/spec/core.

This one just needs proper error handling, and there's a preliminary patch provided.

I'm not sure if we can support these, but if we can we should. If we can't, we should raise an error. Either way it's probably not hard to resolve.

There's a patch here that just needs a little more work.

I don't even understand what's happening in this one, because I keep falling asleep reading the description. I think it's probably easy.

Again, there's a patch here that appears to just need a bit more work.

This one has all the background work done on the JRuby side. We're simply not using the right algorithm for Range#step over character ranges. A quick peek at MRI source or a little insight on how to "just fix it" is all that's needed here, and the code is pretty simple. (Update: Fixed, thanks to Mathias Biilmann Christensen.)

This has a patch that's been applied to trunk, but it didn't apply cleanly to the 1.0 branch. Tidy it up, and it's done.

...but Kernel#trap does. Huh? (Update: Fixed! Ola had already repaired it, but not closed the bug.)

This is pretty straightforward...we just don't have Q/q implemented. (Update: Fixed, thanks to Riley Lynch.)

Thursday, October 25, 2007

JRuby 1.1 on Rails: More Performance Previews

Nick Sieger, a Sun coworker and fellow JRuby core team member, has posted the results of benchmarking his team's large Rails-based project on MRI and the codebase soon to be released as JRuby 1.1 (trunk). Read it here:


Now there's a couple things I get out of this post:
  1. JRuby on Rails is starting, at least for this app, to surpass MRI + Mongrel for simple serial-request benchmarks. And this a week before the 1.1 beta release and all the continuing performance enhancements we know are still out there. I think we're gonna make it, folks. If this continues, there's no doubt in my mind JRuby 1.1 will run Rails fastest.
  2. JRuby on Rails will perform at least as well as MRI + Mongrel for the app Nick and his teammates are building. Several months ago they committed to eventually deploying on JRuby, and if you knew what I know about this app, you'd know why that scared the dickens out of me. But I'm glad the team had faith in JRuby and the JRuby community, and I'm glad we've been able to deliver.
I hope we're approaching a time when people believe that JRuby is the real deal--an excellent choice for running Rails now and potentially the best choice in the near future. There's always more work to do, but it's good to see the effort paying off.

Give it a try today, why not?

Tuesday, October 23, 2007

Help Us Complete JRuby 1.1 and 1.0.2 Releases

We're looking to do releases of a 1.1 beta and 1.0.2 over the next couple weeks, and we're hoping to pull in help from the community to turn over as many bugs as possible. A lot of the bugs we'd like to fix for each release wouldn't be very difficult for folks to get into, even if you only have a bit of Ruby experience and a good Java background. Currently we're looking at the following counts:
  • JRuby 1.0.2: 54 scheduled - These are almost exclusively compatibility issues, though there's a few minor performance items and several stability fixes.
  • JRuby 1.1: 147 scheduled - There's almost all the 1.0.2 bugs in here plus many additional performance enhancements and bug fixes that can't be cleanly/easily backported to 1.0.2.
  • Post 1.1: 7 JRuby 1.x, 71 unscheduled - These bugs are in danger of getting punted to a future release, so if you have a bug in these lists or you see something you think you'd like to/want to fix, now's the time.
The 1.0.2 release is basically a maintenance release to the 1.0 branch, and as such includes primarily compatibility and stability fixes. There's a couple minor perf issues that will be resolved, generally only where they were considered so slow as to be "broken".

The big story is going to be 1.1. It will include a massive number of performance enhancements and bug fixes. It will include a complete Ruby-to-Java bytecode compiler, which is responsible for much of the performance improvements. And it should also include a completely reworked regular expression subsystem that eliminates one of the last really major bottlenecks running Rails code. We may not have that done for the 1.1 beta release at RubyConf, but it will be there for 1.1 final around a month later.

So if you've been looking for a chance to get involved in JRuby, hop on the mailing lists or start browing the bug lists above, and feel free to email me directly (firstname.lastname@sun.com) if you have questions about any specific bug. JRuby needs your support!

Wednesday, October 17, 2007

Another Performance Discovery: REXML

I've discovered a really awful bottleneck in REXML processing.

Look at these results for parsing our build.xml:
read content from stream, no DOM
2.592000 0.000000 2.592000 ( 2.592000)
1.326000 0.000000 1.326000 ( 1.326000)
0.853000 0.000000 0.853000 ( 0.853000)
0.620000 0.000000 0.620000 ( 0.620000)
0.471000 0.000000 0.471000 ( 0.471000)
read content once, no DOM
5.323000 0.000000 5.323000 ( 5.323000)
5.328000 0.000000 5.328000 ( 5.328000)
5.209000 0.000000 5.209000 ( 5.209000)
5.173000 0.000000 5.173000 ( 5.173000)
5.138000 0.000000 5.138000 ( 5.138000)

When reading from a stream, the content is read in in chunks, with each chunk being matched in turn. Because our current regexp engine uses char[] instead of byte[], each chunk must be decoded into UTF-16 characters, matched, and encoded back into UTF-8 bytes.

For small chunks, like those read off a stream, this decode/encode cycle is fairly quick. Here, the streamed numbers are pretty close to MRI. However, when an XML string is parsed from memory, the process goes like this:
  1. set buffer to entire string
  2. match against the buffer
  3. set buffer to post match (remainder of the string)
Now this is obviously a little inefficient, since it creates a lot of extra strings, but a copy-on-write String implementation helps a lot. However in our case it also means that we decode/encode the entire remaining XML string for every element match. For any nontrivial file, this is *terrible* overhead.

So what's the fix? Here's the same second benchmark using a StringIO object passed to the parser instead, with a simple change to rexml/source.rb:

Diff:
Index: lib/ruby/1.8/rexml/source.rb
===================================================================
--- lib/ruby/1.8/rexml/source.rb (revision 4596)
+++ lib/ruby/1.8/rexml/source.rb (working copy)
@@ -1,4 +1,5 @@
require 'rexml/encoding'
+require 'stringio'

module REXML
# Generates Source-s. USE THIS CLASS.
@@ -8,7 +9,7 @@
# @return a Source, or nil if a bad argument was given
def SourceFactory::create_from arg#, slurp=true
if arg.kind_of? String
- Source.new(arg)
+ IOSource.new(StringIO.new(arg))
elsif arg.respond_to? :read and
arg.respond_to? :readline and
arg.respond_to? :nil? and

New numbers:
read content once, no DOM
0.640000 0.000000 0.640000 ( 0.640000)
0.693000 0.000000 0.693000 ( 0.693000)
0.542000 0.000000 0.542000 ( 0.542000)
0.349000 0.000000 0.349000 ( 0.349000)
0.336000 0.000000 0.336000 ( 0.336000)

This is a perfect indication why JRuby's Rails performance is currently nowhere near what it will be. We continue to find these little gems...and there's no telling how many more are out there. With recent execution performance numbers looking extremely solid and recent Rails performance getting closer and closer, the upcoming 1.1 release ought to be amazing.

Friday, October 12, 2007

Performance Update

As some of you may know, I've been busily migrating all method binding to use Java annotations. The main reasons for this are to simplify binding and to provide end-to-end metadata that can be used for optimizing methods. It has enabled using a single binding generator for 90% of methods in the system (and increasing). And today that has enabled making some impressive perf improvements.

The method binding ends up looking like this:
@JRubyMethod(name = "[]", name2 = "slice",
required = 1, optional = 1)
public IRubyObject aref(IRubyObject[] args) {
...

This binds the aref Java method to the two Ruby method names [] and slice and enforces a minimum of one argument and a maximum of two. And it does this all automatically; no manual arity checking or method binding is necessary. Neat. But that's not the coolest result of the migration.

The first big step I took today was migrating all annotation-based binding to directly generate unique DynamicMethod subclasses rather than unique Callback subclasses that would then be wrapped in a generic DynamicMethod implementation. This moves generated code closer to the actual calls.

The second step was to completely disable STI dispatch. STI, we shall miss you.

So, benchmarks. Of course fibonacci numbers are indicative of only a very narrow range of performance, but I think they're a good indicator of where general performance will go in the future, as we're able to expand these optimizations to a wider range of methods.

JRuby before the changes:
$ jruby -J-server -O bench_fib_recursive.rb
1.039000 0.000000 1.039000 ( 1.039000)
1.182000 0.000000 1.182000 ( 1.182000)
1.201000 0.000000 1.201000 ( 1.201000)
1.197000 0.000000 1.197000 ( 1.197000)
1.208000 0.000000 1.208000 ( 1.208000)
1.202000 0.000000 1.202000 ( 1.202000)
1.187000 0.000000 1.187000 ( 1.187000)
1.188000 0.000000 1.188000 ( 1.188000)

JRuby after:
$ jruby -J-server -O bench_fib_recursive.rb
0.864000 0.000000 0.864000 ( 0.863000)
0.640000 0.000000 0.640000 ( 0.640000)
0.637000 0.000000 0.637000 ( 0.637000)
0.637000 0.000000 0.637000 ( 0.637000)
0.642000 0.000000 0.642000 ( 0.642000)
0.643000 0.000000 0.643000 ( 0.643000)
0.652000 0.000000 0.652000 ( 0.652000)
0.637000 0.000000 0.637000 ( 0.637000)

This is probably the largest performance boost since the early days of the compiler, and it's by far the fastest fib has ever run. Here's MRI (Ruby 1.8) and YARV (Ruby 1.9) numbers for comparison:

MRI:
$ ruby bench_fib_recursive.rb
1.760000 0.010000 1.770000 ( 1.813867)
1.750000 0.010000 1.760000 ( 1.827066)
1.760000 0.000000 1.760000 ( 1.796172)
1.760000 0.010000 1.770000 ( 1.822739)
1.740000 0.000000 1.740000 ( 1.800645)
1.750000 0.010000 1.760000 ( 1.751270)
1.750000 0.000000 1.750000 ( 1.778388)
1.740000 0.000000 1.740000 ( 1.755024)

And YARV:
$ ./ruby -I lib bench_fib_recursive.rb
0.390000 0.000000 0.390000 ( 0.398399)
0.390000 0.000000 0.390000 ( 0.412120)
0.400000 0.010000 0.410000 ( 0.424013)
0.400000 0.000000 0.400000 ( 0.415217)
0.400000 0.000000 0.400000 ( 0.409039)
0.390000 0.000000 0.390000 ( 0.415853)
0.400000 0.000000 0.400000 ( 0.415201)
0.400000 0.000000 0.400000 ( 0.504051)

What I think is really awesome is that I'm comfortable showing YARV's numbers, since we're getting so close--and YARV has a bunch of additional integer math optimizations we don't currently support and thought we'd never be able to compete with. Well, I guess we can.

However a more reasonable benchmark is the "pentomino" benchmark in the YARV suite. We've always been slower than MRI...much slower some time ago when nothing compiled. But times they are a-changin'. Here's JRuby before the changes:
$ time jruby -J-server -O sbench/bm_app_pentomino.rb

real 1m50.463s
user 1m49.990s
sys 0m1.131s

And after:
$ time jruby -J-server -O bench/bm_app_pentomino.rb

real 1m25.906s
user 1m26.393s
sys 0m0.946s

MRI:
$ time ruby test/bench/yarv/bm_app_pentomino.rb

real 1m47.635s
user 1m47.287s
sys 0m0.138s

And YARV:
$ time ./ruby -I lib bench/bm_app_pentomino.rb

real 0m49.733s
user 0m49.543s
sys 0m0.104s

Again, keep in mind that YARV is optimized around these benchmarks, so it's not surprising it would still be faster. But with these recent changes--general-purpose changes that are not targeted at any specific benchmark--we're now less than 2x slower.

My confidence has been wholly restored.