Ruby on Windows: A Note for Microsoft
21 / Sep 2006This article was modified from its original published form. The most recent modification was on2014-09-29.
Update: This article is no longer relevant, but has been republished for the historical record. I am no longer involved in these discussions, and the excellent work by the RubyInstaller for Windows team has rendered the points covered moot.
I have finally sat down to try to pull together the issues related to building Ruby with Visual Studio 8. I apologize for the lengthy delay in producing this so that the conversation can continue on this matter.
As I said back in June, the primary concern that most Ruby developers have is the difficulty in compiling extensions on Windows, especially those developers who have extensive background with UNIX-like environments, where you can easily do:
./configure && make && make test && sudo make install
or
ruby extconf.rb && make && make install
or
gem install <name-of-extension-gem>
Visual Studio 2005 (VS8) does not make this easier because of the choice to require manifests for the unmanaged assemblies with SxS installation. Much of this can be automated, but some common tools are completely missing from the VS8 tool chain that make building harder than it needs to be. To be fair, there are tools missing from the Ruby environment that make building extensions harder than it needs to be.
Ruby needs tools similar to the Python distutils, but this is something that the Ruby community must provide, not Microsoft. So far, though, no one has stepped forward to create such a beast, although tools like mkrf from this year’s Google Summer of Code could be used to improve this situation considerably.
This does not mean that Microsoft has nothing that it can do to help here. If Microsoft wishes to win the Ruby community toward using VS8 instead of continuing to use VC6 or stepping sideways toward the MinGW/MSYS combination, it needs to offer positive steps toward increased compatibility even between Microsoft’s tools and offer some way of making it easier to build tools that are primarily developed on UNIX-like platforms for Windows.
Let me be clear: I consider using MinGW/MSYS a step backwards (not sideways) for the Ruby community to take, because it goes to a known-buggy runtime (VC6’s MSVCRT.DLL) and will have a harder time interfacing with Microsoft’s modern technologies moving forward. Not everyone considers this a problem; they need Windows versions of tools that they have been using on UNIX-like environments more than they need the advanced capabilities offered by the later runtimes. This is a compelling argument, but I do not find it wholly convincing.
There is a vocal contingent of Ruby developers that would like to be able to have a version of the Ruby One-Click Installer that includes a compiler as part of its distribution, possibly as a separate download. This is something that would matter significantly for people who use Ryan Davis’s RubyInline or ZenOptimize, and other development environments. Again, I don’t find this compelling (and would prefer not having such a thing as a single distribution package because it would be unnecessarily large). It is a significant concept, though, because the one thing that the Microsoft tool chain has over MinGW/MSYS is that it is easier to install and get running, even though it’s significantly harder to build extensions with.
What Do We Need From Microsoft?
This question is the source of considerable debate and anger. There is a level of mistrust of Microsoft in the Ruby community (especially the Japanese community) based on missteps by the Microsoft developer community with regards to backwards compatibility. For this, I’m going to quote Usa Nakamura, who builds the Ruby binaries that Curt uses for the Ruby One-Click Installer for 1.8.4 and 1.8.5 with some clarification by myself and URABE Shyouhei. The original messages were provided in response to a post that I made on the 26th of June, 2006 if they are needed.
The problem of errno is not serious, I think. We will be able to avoid the problem with some simple code (for example, replace it with a function call by macro.) The real problems [arise from separate resource management per runtime DLL version, keeping independent file descriptor tables, memory management, etc., that are internally maintained and not open through any API].
The decision [by Microsoft to break] binary compatibility between versions of runtime […] is foolish. [Did they think] that passing file descriptors between DLLs [would not happen]? As time goes by, there will be some [need] to introduce such incompatibility, I know. […] VC7, VC7.1, and VC8 were shipped [in rapid succession], and they are mutually incompatible[…]. It’s crazy, to say the least.
I decided to [stay] with VC6 for the above-mentioned reasons. If MS keeps shipping incompatible versions at each upgrade, I will throw away VC and shift to MinGW. If MinGW also follows [MS’s binary incompatibility], I’ll shift to Cygwin or throw away Windows as development environment.
[I] hope [that] Microsoft [could provide a] wrapper DLL [for] MSVCRT80.DLL named MSVCRT.DLL. If MS prepares such a mechanism, [there will be] binary compatibility[…]. If so, [it wouldn’t matter whethera program links against MSVCRT80.DLL or MSVCRT.DLL].
It’s bad enough where we have the source but prefer using pre-compiled binaries either because of time, trustworthiness, or compilation complexity. Ara Howard notes that compiling the GNU Scientific Library (GSL) on Windows is so complex without Cygwin or MinGW that there is a company that charges $600 for a compiled version of the source. Others have noted similar issues with other code.
It’s worse when we have third-party binary DLLs compiled with an earlier or later version of Microsoft tools. Ruby is receiving a lot of attention now because of Rails, and I suspect that Microsoft would love to have people using SQL Server as their database underneath Rails, but if there are problems compiling the binary extensions to link against the SQL Server interface drivers, this is less likely to happen or be common. Oracle drivers present a similar problem. We don’t have the source to recompile with the tool that we prefer, and the binary DLL we must use may have been compiled with an earlier incompatible version of the compiler.
(Let me digress for a moment and make it clear that this is an overall problem on all platforms, not just Windows; I have experienced it most often on Linux in my day job where I do C/C++ development. It is far more apparent on Windows, however, because it is unusual for there to be more than one C/C++ runtime on a UNIX-like system. Not impossible, but unusual.)
Thus, the first thing that we feel that would be beneficial from Microsoft would be some sort of runtime shim or wrapper that would allow us to use programs built with VC6, VC7, VC7.1, VC8 or even MinGW compatibly.
The second thing that we feel that would be beneficial from Microsoft would be a better command-line tool chain that is preferably compatible with many of the UNIX-style build commands. A minimal start for this would be command-wrappers that allow you to use a gcc/g++ front-end that actually calls VS8’s cl.exe with the appropriate command-line parameters. Microsoft’s move away from command-line support and toward devenv as the primary build environment helps most Windows developers, but hurts those who do their primary development on platforms other than Windows.
The third thing reaches into core functionality problems that Nobu Nakada posted about in late July:
- more unified I/O
- nonblocking flag can be set on only sockets.
- WaitForMultipleObjects doesn’t work with sockets.
- APIs to tell how many bytes can be read immediately differfor ordinary file, console, socket and pipe.
- pseudo tty: pty feature is not supported at all now.
Charlie Savage had some interesting things to say in a lengthy discussion in July while I was in Europe with limited Internet access:
From my experience using both tool chains on Windows (for the ruby-prof extension and SWIG-based extensions for GEOS and GDAL).
- You can build Ruby extensions using MingW that run against Ruby built with VC++. I’ve done this with Ruby 1.8.2 / 1.8.4, various MingW releases and VC++ 2003 and VC++ 2005. This used to require changing a small bug in ruby.h for Ruby 1.8.2, but that bug has been fixed with 1.8.4.[…]
However, you cannot do this with MingW using VC++ built Ruby.
ruby extconf.rb make make installThe problem is that extconf is quite limited - it will assume you are building your extension with the same compiler that built Ruby (VC++). Python avoids this issue because disutils will recognize the compiler being used (MingW, VC++) for the extension and provide the correct command line parameters.
If mkrf can work like Python distutils, then it will become simple to use MingW to build extensions that work with VC++
When compiling with MingW do not link against the ruby *.lib files. Instead, just link directly against the DLL (msvcrt-ruby18.dll). It’s faster (links much faster) and works better.
So you need to manually compile your extension or create a makefile to do it. This actually turns out be the way GEOS and GDAL work - they have autoconf based build systems so extconf.rb wouldn’t fit in anyway.
The advantage of MingW is that it avoids the unmanaged assemblies that VC++8 uses, so its simpler to deal with[…]
VC++ has several large advantages on Windows.
- First, it lets you debug your extensions while GDB does not support this on Windows (or if it does, its never worked for me).
- Second, it compiles much faster
- Third, there is a lot more help available.
- [Fourth], its quite easy to build Ruby extensions.
Using MingW on Windows is a huge barrier to entry. Gettting MingW setup, along with msys, is a time consuming process that only experienced *Nix developers will understand and be able to do.
MingW on Windows is not very easy to use. It’s nice to think that you can download an open source project, type ./configure, make, make install and it will work. Alas, it doesn’t really work that way. There are myriad of issues you run into. First you’ll need msys. Then many projects have prequisites that you’ll have to download and compile. In addition, you often times have to change the CFLAGS and LDFLAGS to get successful compiles. Linking is a pain and requires hand-holding, and sometime just doesn’t work. Libtool is really flakey on Windows. For some projects, you’ll have to need to download/build/install the latest version of it. You also need to get autoconf/automake installed. Many projects require bison - something I’ve never been able to successfully compile on Windows. All in all - it literally took me weeks to figure out how to get everything to work together. The MingW/msys tool chain is quite complex on Windows, and most people won’t have the time or desire to put forth the effort to get it to work.
My recommendation:
- Use VC++ 2005 and get Microsoft to tell us how to properly use unmanaged assemblies so that we can avoid dll hell
- Make sure that mkrf supports building Ruby extensions “out-of-the-box” on Windows using MingW if you have it installed. I think this would be the best of both worlds - you support both tool chains. VC++ is the default one, but MingW should work fine for building extensions.
Hope this helps - I’d be glad to share more of my experiences if its helpful.
So, what can we do to move Ruby toward a highly usable environment that is based on modern Microsoft compiler technology yet remains backwards compatibility with the tools that for many reasons we cannot give up?
- 2014-09-29: Added an opening note about why this post is no longer relevant.[ back ]