- Best Open Source Programming Language - Summary: Ruby gets mad props for a vibrant community and a diverse range of implementations (e.g. JRuby), and then they go squishy and say "and these other languages are great too!"
- Best Open Source IDE - InfoWorld heaps praise on NetBeans largely because it has *not* gone the way of an amorphous, all-encompassing "platform" and has continued to take risks to improve the overall experience of the IDE. Before the work on NetBeans 6 (don't download M10; use a daily build) I was a nay-sayer myself; having used NetBeans 6 for many months now, I can honestly say it's far better than NetBeans 5.5 and has caught up or passed Eclipse in many ways. There's more to do, but I'm extremely impressed with the progress.
Wednesday, September 12, 2007
InfoWorld Bossies Close to my Heart
A couple interesting "Bossies" were awarded by InfoWorld this week:
Tuesday, September 11, 2007
JRuby Compiler Update, and a Nice Peformance Milestone
Hello again friends! It's time to update you on the status of the JRuby compiler.
Compiler Status
I've been working feverishly for the past several weeks to get the rest of the compiler complete. Currently, it's able to handle the majority of Ruby syntax. Here's a list of the remaining language features that do not compile:
We will likely have to solve this complication in one of two ways:
A Nice Performance Milestone
And on the topic of performance, the recent compiler work has allowed us to reach a new milestone: we now exceed Ruby 1.8.6's performance on M. Edward (Ed) Borasky's MatrixBenchmark.
Some months back, after the Mountain West RubyConf in Salt Lake City, Ed posted an interesting blog entry where he professed a lot of confidence in JRuby's future. We emailed a bit offline, and he pointed me to this matrix benchmark he'd been using to measure the relative performance of Ruby 1.8.6 and Ruby 1.9 (YARV). I told him I'd give it a try.
Originally, we were perhaps 50% to 100% slower than Ruby 1.8.6. This was back when hardly anything was compiling, and there had been few serious efforts to optimize the JRuby runtime. Performance slowly crept up as time went on. But as recent as a week ago, JRuby performance was still roughly 20-25% slower than 1.8.6.
So last week, I dug into it a bit more. I turned on JRuby's JIT logging (-J-Djruby.jit.logging=true) and verbose logging (-J-Djruby.jit.logging.verbose=true) to log compiling and non-compiling methods, respectively. As it turned out, the "inverse_from" method in matrix.rb was not yet compiling...and it was where the bulk of MatrixBenchmark's work was happening.
The final sticking point in the compiler for this method was "operator element assignment" syntax, or basically anything that looks like a[0] += 5. It's a little involved to compile; you have to retrieve the element, calculate the value, call the operator method, and reassign all in one operation. For the ||= or &&= versions, you have to perform a boolean check against the element to see if you should proceed to the assignment. A good bit of compiler code, but it had to be done.
So then, with "OpElementAsgn" compiling, it was time to re-run the numbers. And finally, finally, we were comfortably exceeding Ruby 1.8.6 performance:
On Beyond Zebra
I believe we're pretty well on-target to have the compiler completed by RubyConf in November. I'm about to embark on a refactoring adventure to prepare for the stack-juggling I'll have to do to support rescue blocks. That will mean minimal progress on adding to the compiler until the end of the month, but ideally the refactoring will make it easy to get rescue compilation complete. The others are just a matter of spending some time.
Once the JRuby compiler is complete, we will start testing in earnest against a fully pre-compiled Ruby stdlib. Along with that, we'll wire in support for pre-compiling RubyGems as they install and pre-compiling Ruby scripts as they are executed and loaded. Much of this works already in prototype form, but it waits for the completion of the compiler to go into general use.
I also have plans for a "static" compiler for JRuby that enable compiling Ruby classes into normal, instantiable, callable, static Java classes. This would bring us on par with other compiled languages on the JVM, and allow you to directly instantiate and invoke JRuby/Ruby objects from within your Java code.
Beyond all this work, Tom and I have been discussing a whole raft of performance improvements we could make to the underlying JRuby runtime. There's a lot more performance to be had, and it's just around the corner.
Exciting times, friends. Exciting times.
Compiler Status
I've been working feverishly for the past several weeks to get the rest of the compiler complete. Currently, it's able to handle the majority of Ruby syntax. Here's a list of the remaining language features that do not compile:
- "rescue" blocks; exception handling in Ruby is rather complicated, and there's some particularly odd uses of rescue that will be a bit tricky to support with normal Java exception-handling.
- "class var declaration" is not yet supported. This is when you declare a class variable (@@foo) from within the body of a class or module. This primarily affects compiling class bodies, so although it prevents AOT compilation of some scripts, it doesn't usually affect individual methods.
- "opt n" execution. This is specifying "-n" to the Ruby runtime, and it loops the provided script as though it were surrounded by "while gets(); ... end". It's useful for line-by-line processing of stdin.
- "post execution" blocks. Post exe blocks are when you specify an END { ... } block somewhere in your script. These blocks are saved up and executed at the end of the script execution, regardless of where they appear in the script. They're a bit like Kernel#at_exit blocks.
- "retry". Tell me friends, do you know what "retry" actually does? Retry is used within a block/closure, and it causes the method containing the closure to be re-called anew. And as an interesting quirk, the original arguments to the method are re-evaluated, so if you call foo(bar()) and a retry is triggered within foo(), bar() will get invoked again for the retried call to foo(). Weird, eh? Update: I didn't explain this well. Here's another attempt: if you have the following code:
def foo(x = bar()); 1.times {retry}; endAnd you call foo with no arguments, allowing the default argument logic to fire, retry will cause that logic to fire again and again. It's essentially re-entering the method anew with the original arguments, but causing *argument processing* to be revisited. I'm not sure why you'd want this behavior, since it could frequently result in default arguments to re-call methods that might only be valid the first time. - Some non-local flow control is not yet complete. Non-local flow control happens any time you return, break, or next from within a block (when not immediately inside a normal loop construct). Much of non-local flow control is working, but I need to flush out any remaining cases that aren't running correctly.
a = [1, 2, (begin; raise; rescue; 3; end)]When this code is compiled, it turns into a local variable assignment. The value assigned is a literal array construction with three elements: a Fixnum 1, a Fixnum 2, and a rescued block of code. The typical way to construct the array then is to follow these steps:
- Construct an array of the appropriate size
- Dup the array reference
- Push a constant integer zero
- Push Fixnum 1
- Insert Fixnum 1 at index zero in the array. This consumes the dup'ed array, the index, and the Fixnum1.
- Dup the array reference again
- Push a constant integer one
- Push Fixnum 2
- Insert Fixnum 2
- Dup the array reference again
- Push a constant integer two
- Now it gets complicated; we must recurse in the compiler to handle the rescue block
- The rescue block is compiled and a "raise" is triggered in the code
- The exception raised is handled, resulting in the whole rescue leaving a Fixnum 3 on the stack
- Insert the Fixnum 3
- Construct a RubyArray object with the remaining object array
We will likely have to solve this complication in one of two ways:
- We could save off the stack when entering code that might trigger exception handling
- We could put exception-handling logic in a separate method and invoke it in-place, thereby protecting our executing stack from clearage.
A Nice Performance Milestone
And on the topic of performance, the recent compiler work has allowed us to reach a new milestone: we now exceed Ruby 1.8.6's performance on M. Edward (Ed) Borasky's MatrixBenchmark.
Some months back, after the Mountain West RubyConf in Salt Lake City, Ed posted an interesting blog entry where he professed a lot of confidence in JRuby's future. We emailed a bit offline, and he pointed me to this matrix benchmark he'd been using to measure the relative performance of Ruby 1.8.6 and Ruby 1.9 (YARV). I told him I'd give it a try.
Originally, we were perhaps 50% to 100% slower than Ruby 1.8.6. This was back when hardly anything was compiling, and there had been few serious efforts to optimize the JRuby runtime. Performance slowly crept up as time went on. But as recent as a week ago, JRuby performance was still roughly 20-25% slower than 1.8.6.
So last week, I dug into it a bit more. I turned on JRuby's JIT logging (-J-Djruby.jit.logging=true) and verbose logging (-J-Djruby.jit.logging.verbose=true) to log compiling and non-compiling methods, respectively. As it turned out, the "inverse_from" method in matrix.rb was not yet compiling...and it was where the bulk of MatrixBenchmark's work was happening.
The final sticking point in the compiler for this method was "operator element assignment" syntax, or basically anything that looks like a[0] += 5. It's a little involved to compile; you have to retrieve the element, calculate the value, call the operator method, and reassign all in one operation. For the ||= or &&= versions, you have to perform a boolean check against the element to see if you should proceed to the assignment. A good bit of compiler code, but it had to be done.
So then, with "OpElementAsgn" compiling, it was time to re-run the numbers. And finally, finally, we were comfortably exceeding Ruby 1.8.6 performance:
Ruby 1.8.6:Or should I say vastly exceeding? By my calculation this is an easy 2x performance increase, and perhaps a 70% improvement just by getting this one extra method to compile.
Hilbert matrix of dimension 128 times its inverse = identity? true
586.110000 5.710000 591.820000 (781.251569)
JRuby trunk, Java 6 server, ObjectSpace disabled:
Hilbert matrix of dimension 128 times its inverse = identity? true
372.950000 0.000000 372.950000 (372.950000)
On Beyond Zebra
I believe we're pretty well on-target to have the compiler completed by RubyConf in November. I'm about to embark on a refactoring adventure to prepare for the stack-juggling I'll have to do to support rescue blocks. That will mean minimal progress on adding to the compiler until the end of the month, but ideally the refactoring will make it easy to get rescue compilation complete. The others are just a matter of spending some time.
Once the JRuby compiler is complete, we will start testing in earnest against a fully pre-compiled Ruby stdlib. Along with that, we'll wire in support for pre-compiling RubyGems as they install and pre-compiling Ruby scripts as they are executed and loaded. Much of this works already in prototype form, but it waits for the completion of the compiler to go into general use.
I also have plans for a "static" compiler for JRuby that enable compiling Ruby classes into normal, instantiable, callable, static Java classes. This would bring us on par with other compiled languages on the JVM, and allow you to directly instantiate and invoke JRuby/Ruby objects from within your Java code.
Beyond all this work, Tom and I have been discussing a whole raft of performance improvements we could make to the underlying JRuby runtime. There's a lot more performance to be had, and it's just around the corner.
Exciting times, friends. Exciting times.
Sunday, September 2, 2007
Java Native Access + JRuby = True POSIX
I know, I know. It's got native code in it, and that's bad. It's using JNI, and that's bad. But damn, for the life of me I couldn't see enough negatives to using Java Native Access that would outweigh the incredible positives. For example, the new POSIX interface I just committed:
Given this library, it takes a whopping *one line of code* to wire up the standard C library on a given platform:
That's. It.
So the idea behind JNA (jna.dev.java.net) is to use a foreign function interface library (libffi in this case) to wire up C libraries programmatically. This allows loading a library, inspecting its contents, and providing interfaces for those functions at runtime. They get bound to the appropriate places in the Java interface implementation, and JNA does some magic under the covers to handle converting types around. And so far, it appears to do an outstanding job.
With this code in JRuby, we now have fully complete chmod and chown functionality. Before, we had to either shell out for chmod or use Java 6 APIs that wouldn't allow setting group bits, and we always had to shell out for chown because there's no Java API for it. Now, both work *flawlessly*.
The potential of this is enormous. Any C function we haven't been able to emulate, we can just use directly. Symlinks, fcntl, file stats and tests, process spawning and control, on and on and on. And how's this for a wild idea: with a bit of trickery, we could actually wire up Ruby C extensions, just by providing JNI/JRuby-aware implementations of the Ruby C API functions.
So there it is. I went ahead and committed a trunk build of JNA, and I'm working with the JNA guys to get a release out. We'll have to figure out the platform-specific bits, probably by just shipping a bunch of pre-built libraries (not a big deal; the JNI portion is about 26k), but the result will be true POSIX functionality in JRuby...the "last mile" of compatibility will largely be solved.
Many thanks to Timothy Wall, who's been nursemaiding me through getting JNA working well, and who appears to be the primary driver of the JNA project right now. If you're interest in this stuff, check out JNA, start using it, make it popular. As far as I'm concerned this is how native access is supposed to be.
Update: My mistake, Wayne Meissner has also been pouring a metric buttload of effort into JNA. Credit where credit's due!
import com.sun.jna.Library;
public interface POSIX extends Library {
public int chmod(String filename, int mode);
public int chown(String filename, int owner, int group);
}
Given this library, it takes a whopping *one line of code* to wire up the standard C library on a given platform:
import com.sun.jna.Native
POSIX posix = (POSIX)Native.loadLibrary("c", POSIX.class);
That's. It.
So the idea behind JNA (jna.dev.java.net) is to use a foreign function interface library (libffi in this case) to wire up C libraries programmatically. This allows loading a library, inspecting its contents, and providing interfaces for those functions at runtime. They get bound to the appropriate places in the Java interface implementation, and JNA does some magic under the covers to handle converting types around. And so far, it appears to do an outstanding job.
With this code in JRuby, we now have fully complete chmod and chown functionality. Before, we had to either shell out for chmod or use Java 6 APIs that wouldn't allow setting group bits, and we always had to shell out for chown because there's no Java API for it. Now, both work *flawlessly*.
The potential of this is enormous. Any C function we haven't been able to emulate, we can just use directly. Symlinks, fcntl, file stats and tests, process spawning and control, on and on and on. And how's this for a wild idea: with a bit of trickery, we could actually wire up Ruby C extensions, just by providing JNI/JRuby-aware implementations of the Ruby C API functions.
So there it is. I went ahead and committed a trunk build of JNA, and I'm working with the JNA guys to get a release out. We'll have to figure out the platform-specific bits, probably by just shipping a bunch of pre-built libraries (not a big deal; the JNI portion is about 26k), but the result will be true POSIX functionality in JRuby...the "last mile" of compatibility will largely be solved.
Many thanks to Timothy Wall, who's been nursemaiding me through getting JNA working well, and who appears to be the primary driver of the JNA project right now. If you're interest in this stuff, check out JNA, start using it, make it popular. As far as I'm concerned this is how native access is supposed to be.
Update: My mistake, Wayne Meissner has also been pouring a metric buttload of effort into JNA. Credit where credit's due!
Saturday, August 25, 2007
JRuby 1.0.1 and Jython 2.2 Released
I'm in the airport, but I wanted to add to the blogs reporting that JRuby 1.0.1 and Jython 2.2 have been released.
JRuby 1.0.1 is basically just a maintenance release to 1.0. It includes various compatibility fixes that have filtered in since the release, and doesn't do much to improve performance. JRuby 1.1 will come out by November, and should have all the latest research and perf work, including a complete compiler to JVM bytecode.
Jython 2.2 is much bigger news. After years of sleepily crawling along, the 2.2 release of Jython has finally been released. With 2.2 behind them, the Jython team can now be more ambitious about adding Python 2.3, 2.4, and 2.5 features. Much of that work has already started. Also coming up is a new JVM bytecode compiler. If you're a Python guy, give Jython a try today...those guys have put a lot of time and effort in.
I also understand that Groovy is supposed to have their 1.1 release out some time in October. Happy times for dynlangs on the JVM!
JRuby 1.0.1 is basically just a maintenance release to 1.0. It includes various compatibility fixes that have filtered in since the release, and doesn't do much to improve performance. JRuby 1.1 will come out by November, and should have all the latest research and perf work, including a complete compiler to JVM bytecode.
Jython 2.2 is much bigger news. After years of sleepily crawling along, the 2.2 release of Jython has finally been released. With 2.2 behind them, the Jython team can now be more ambitious about adding Python 2.3, 2.4, and 2.5 features. Much of that work has already started. Also coming up is a new JVM bytecode compiler. If you're a Python guy, give Jython a try today...those guys have put a lot of time and effort in.
I also understand that Groovy is supposed to have their 1.1 release out some time in October. Happy times for dynlangs on the JVM!
Tuesday, August 14, 2007
NetBeans Ruby Support is the BOMB
OMG NetBeans Ruby support is so awesome. I just picked up some of the recent dailies, and it does stuff I just can't believe. But don't take my word for it.
Today I stumbled back into the NetBeans Ruby Wiki, expecting to just find the same old "how to download", "how to install", and "how to build" instructions. Instead I find a nicely-organized set of pages describing (with screenshots!) all of the really awesome features.
Refactoring? Check. Debugging? Check. YAML and RHTML editing? Check. Test running? Check.
It just goes on and on.
If you haven't given it a try, and you're a Ruby programmer, you owe it to yourself to check it out. And if you ph34r a gigantic IDE download with support for Java and UML and BPEL and other stuff you'll never used, there's a Ruby-only IDE available too; check the Installation page. Incredible.
Direct links to feature pages:
Today I stumbled back into the NetBeans Ruby Wiki, expecting to just find the same old "how to download", "how to install", and "how to build" instructions. Instead I find a nicely-organized set of pages describing (with screenshots!) all of the really awesome features.
Refactoring? Check. Debugging? Check. YAML and RHTML editing? Check. Test running? Check.
It just goes on and on.
If you haven't given it a try, and you're a Ruby programmer, you owe it to yourself to check it out. And if you ph34r a gigantic IDE download with support for Java and UML and BPEL and other stuff you'll never used, there's a Ruby-only IDE available too; check the Installation page. Incredible.
Direct links to feature pages:
- RubyEditing describes the editing features available.
- RubyRefactoring describes the refactoring features available.
- RubyProjects describes the project features available.
- RubyOnRails describes the Ruby On Rails features available.
- RubyDebugging describes the debugging features available.
- RubyTesting describes the testing features available.
- RubyCodeTemplates describes the live code templates feature.
- RubyHints describes the quickfixes and hints feature.
- RubyShortcuts lists keyboard shortcuts relevant to Ruby
Sunday, August 5, 2007
Widening the JVM Languages Group: We Need You!
We need diversity in the JVM Languages group, and it's been brought to my attention that some popular/key/interesting languages may not have representation. So we need to change that.
If you are interested in the future of non-Java languages on the JVM, you should be on this list. Yes, we talk about a lot of JVM language implementation challenges, we discuss compilers and stack frames and call-site optimizations, but we also talk about features peripheral to language implementation like package indexing and retrofitting Java 5+ code. We need your help.
Once you've joined, or if you're already member, you have a second task
I respectfully request that each of you search out one individual you think would be interested in the list and try to get them involved. Toss them a quick email, invite them to describe their project or language or implementation to us, and promise them they're joining a very interesting and entertaining community. History will thank you, and so will I.
If you are interested in the future of non-Java languages on the JVM, you should be on this list. Yes, we talk about a lot of JVM language implementation challenges, we discuss compilers and stack frames and call-site optimizations, but we also talk about features peripheral to language implementation like package indexing and retrofitting Java 5+ code. We need your help.
Once you've joined, or if you're already member, you have a second task
I respectfully request that each of you search out one individual you think would be interested in the list and try to get them involved. Toss them a quick email, invite them to describe their project or language or implementation to us, and promise them they're joining a very interesting and entertaining community. History will thank you, and so will I.
A Business Case for Supporting Jython
The facts:
Put simply: if you want to become an OSS rockstar tomorrow, get Mercurial running on Jython.
(credit for this idea goes to OpenJDK ambassador Tom Marble...I'm just trying to needle the OSS world into making it happen)
- Sun and many other organizations have started considering moves to Mercurial. In Sun's case, it's a mandate for all Sun-managed OSS projects (OpenSolaris, OpenJDK, etc).
- Moving projects to Mercurial frequently (usually?) requires IDE/tool support.
- Sun's IDE/tools and those of many other organizations are Java-based (NetBeans, Eclipse, and so on).
- Mercurial is written in Python.
- Jython is an implementation of Python for the JVM.
Put simply: if you want to become an OSS rockstar tomorrow, get Mercurial running on Jython.
(credit for this idea goes to OpenJDK ambassador Tom Marble...I'm just trying to needle the OSS world into making it happen)
Subscribe to:
Posts (Atom)