On Sharing Code between Rails Apps
In the beginning we had a single rails app; life was good. Then we wrote a second one, and a third and a fourth and so the problem of how to share code amongst these apps came about.
The trigger for doing something about this was when we wrote a common authentication server (gatekeeper) responsible for holding users’ details, validating passwords etc… Clearly all our applications need to talk to gatekeeper and the code for doing so was going to be very similar. So how do you share this code?
First of all, what sort of thing were we looking for:
- Ease of development: It should be easy to change the shared code and have the changes take effect quickly, without messing around.
- Propagation of changes: We should not have to do much (ideally anything) in order for all apps to pick up changes.
- Ease of deployment: It shouldn’t complicate the process of deploying our apps. In particular, it should add as few steps as possible and it should play nicely with the way we do branching/tagging.
Candidate solutions
There are a several common solutions to this sort of thing:
- plugins
- gems
- use svn:externals to install plugins
1) We all love plugins, they do all sorts of things very nicely. However in their standard incarnation they aren’t quite right for us. If we checkout our plugin into all of our various rails apps then it’s a nuisance when we update the plugin; it’s hard to make sure those changes get propagated to all the apps.
2) Gems are the canonical ruby code sharing mechanism. With rails there are some issues, for example, we have some models we need to share and so everything needs to be in the rails load path, etc. etc. It also complicates deployment. It’s no longer just a case of updating the source everywhere. You need to be running a private gem_server, push the gems to it, update gems on the production machines and so on.
Plugems [1] looks like an interesting solution to a lot of this, but right now it is a little too new for us (and to be 100% honest we couldn’t get it to work). We remain very interested in seeing what is going to become of it, since on paper it has a lot of good things going for it.
3) This keeps things simple but makes branching and tagging a nightmare (if you branch a directory containing an svn:externals then the branch still contains an svn:externals pointing at the same thing). Of course you can then go and change all the svn:externals to point to an appropriate branch of the common code (possibly creating it) but it’s something you’re bound to screwup eventually, and well we hate screwups.
So none of these were quite right for us. Plugins do most of what we want, and using svn:externals on them is almost exactly what we want, with the exception of the way it interferes with the ability to branch off the code.
So, what to do?
Lets make things a little more concrete. We’ve got, say, 2 apps: app_a and app_b. We’ve also got a chunk of code we call gatekeeper_access that allows applications to access our authentication server and perform related tasks. By default, plugins are loaded from vendor/plugins, but it doesn’t have to be that way. You can set config.plugin_paths to be anything you want and rails will load plugins from those locations. So we can have the following layout on our servers:
app_a/
app
models
...
app_b/
app
models
...
shared_plugins/
gatekeeper_access/
init.rb
lib/
We put the following [2] in the initializer section of environment.rb for each app:
config.plugin_paths = [ "#{RAILS_ROOT}/vendor/plugins", "#{RAILS_ROOT}/../shared_plugins"]
And now we’re done: Rails will load the plugin exactly as if it had been in vendor/plugins. Branching, tagging and deploying are easy version control operations. We’ve not yet had the need to have apps depend on different versions of the shared plugins, however you could do this to an extent by tweaking the directory structure to look like:
apps_using_version_1/
app_a
app_b
shared_plugins (version1)
apps_using_version_2/
app_c
app_d
shared_plugins (version2)
However, our general philosophy in this area is not to do this; it would quickly become very complicated when different parts of the app all start depending on different versions of various components.
There is one slight annoyance: when running script/generator you don’t get access to generators in plugins not in vendor/plugins. The easiest way round this is to check out the generators into ~/.rails/generators.
Other than that (and to be fair, typically you don’t use generators very often) this setup has worked very well for us.
[2] In environment.rb/initializer.rb you’ll currently see a comment saying you can use config.plugin_paths = ["vendor/plugins", "../shared_plugins"]. This works as long as the current working directory is the root of your rails app, which you can’t always rely on (cron jobs for example). We prefer not having to remember to always ensure the working directory has any particular value.
July 8th, 2007 at 10:25 pm
Why don’t you just use symlinks?
have your /common directory,
and have a symlink /app_a/vendor/plugins/gatekeeper_access -> ../../../common/vendor/plugins/gatekeeper_access.
that’s how i’d do it,
that’s how i do do it.
July 9th, 2007 at 8:36 am
In a way it’s very similar, and I don’t see why it wouldn’t work however we’d rather not depend on symlinks (which don’t exist on all platforms, including windows (and we do have some windows development machines)). I also believe there’s something to be said for keeping as much of the cleverness as possible in ruby
September 3rd, 2007 at 12:42 pm
I found this to be an extremely helpful article, and I almost implemented the solution… but it didn’t quite work out for me, and I’ll explain why (and how I got around it with Subversion), because I think it’s a pretty novel alternative that might suit others’ needs.
We have several websites which use a common database, and for performance reasons and simpler database code we have certain ‘derived’ properties (such as the ‘quality’ of a user’s data - whether we have their phone number, address, etc) stored as columns and assigned in the model. If and when we change which fields are ‘required’, updates need to be made to the model code on all of the websites to reflect this fact, so it seemed easier to do it as a plugin.
Naturally, I wanted to avoid duplication, so I ran across this article and started putting it into practice. Unfortunately, the way our plugin unit tests are run, they need access to the host application’s environment (we’re using a process similar to the one described here.). There’s no way to tell, using a shared plugin directory, which website to run the tests against! Admittedly, we’re looking into creating a separate environment for each plugin (with its own test database etc.) but that hasn’t happened yet.
So I started digging in the Subversion documentation (we just started using it recently), and I ran across Externals. Turns out they were the key to solving this, and it adds some benefits, too.
Say you want to include the ‘auth’ plugin as an external. From the root of the application, issue
svn propedit svn:externals vendor/plugins
Then enter, for example,
auth http://auth.com/auth/trunk/
As far as I can tell, you’ll have to actually check the external out (if you’re adding it to an existing project), but from that point on it’s a “bolted-on” addition to the code, drawn from another repository.
Here’s the best part - you can modify and make changes to that plugin independently! Now, if you do a ’svn commit’ from the root of your application, it won’t commit anything in the plugin. But if you issue ’svn commit vendor/plugins/auth’, you can send stuff back for the plugin!
It’s kind of the best of both worlds - you can edit the plugin within the context of the Rails application (rather than as a completely separate ‘project’ in your IDE of choice) but still commit changes to shared code. And there’s no duplication of this plugin code across the various Rails apps that use it.
Also, you can ‘lock’ an External at a particular revision level, but learning why that’s so cool is an exercise left to the reader. =)
I hope this is helpful to someone!
September 3rd, 2007 at 12:59 pm
Oh dear, I’ve been a complete doof. You already mentioned svn:externals in your article as #3, heh. I came back after reading this article a couple weeks ago to post my wonderful new discovery (we’re new to SVN, heh)… and didn’t even bother to look over it again. It’s been a long night of coding. Um, feel free to delete these posts… I’d rather you do, only because I look like an idiot. =)
I see your concerns about keeping a SVN property (which isn’t immediately apparent when working on a project) properly synchronized. I think we’ll still give svn:externals a shot on our upcoming project, see how it pans out. If it gets too complicated, though, I find your approach here extremely simple and helpful, and will probably turn back to it! Either way, thanks for your help! And sorry about the dumb post, heh.
September 3rd, 2007 at 1:13 pm
Thanks for your feedback! If there’s a way to bend svn:externals to your will it would be cool to hear about it. I’m pretty sure that you could come up with some capistrano scripts that would do all the messing around with svn:externals at the point when you branch off a release.
There’s a session or 2 about this at railsconf europe in 2 weeks time, I’m hoping to be taught some magic that will improve on this!
January 1st, 2008 at 2:13 am
I was thinking about this issue earlier today… so far, svn:externals has been working okay as originally designed (although I do indeed have to update the svn:externals property on every referencing project when a new version comes out), but mainly because there aren’t too many dependent projects.
One of the benefits I’ve personally found is that I can make local changes to the shared code (since it’s just included under /vendor/plugins) and check them in or revert them as appropriate for everything else. Additionally, since the svn:externals properties that reference the common code repository include revision numbers… updating shared code doesn’t immediately affect other applications which may require testing or further work. [In fact, using fixed revisions on externals has, so far, seemed to prevent a number of branch/tag problems... I'm curious what your thoughts are on this.] Admittedly, simple bug fixes would be overwhelmingly annoying using svn:externals (because you’d have to catch every external reference and update it), but adding or modifying major functionality (which may cause breaking changes) seems safer to me when using externals. It’s probably only working well at the moment because the common code involves simple models without too much functionality, to which we add a new model periodically, or update references therein, and there are only five or six sites which reference the common repository via svn:externals. In other words, keeping the references updated is easy, albeit still a manual update-and-test process.
However, with Rails 2.0 formally released a few weeks ago, I’ve been thinking about an alternative approach using ActiveResource. Rather than worrying about how to integrate common model code as plugins into sites… instead just move the common functionality into a RESTful web service. Of course, this carries all the usual complications of web service functionality… changes in production affect all consuming sites immediately, testing is frustrated, breaking changes are extremely significant, etc. But it’s an idea I’ll be toying with over the next month.
January 1st, 2008 at 2:29 am
One last thought - it seems to me that, in situations where the common code changes infrequently (as in my case), svn:externals works relatively well and offers a few advantages - in-place editing/testing, locking to different revisions of the common code - for only a few disadvantages, e.g. frequent changes to common code require frequent updates to the locked revision in the external.
Additionally, so far, most of our common functionality is in the form of models, not /lib helper routines. That might be different for a lot of people, and explain why updates are infrequent and managable via svn:externals on our end.
Like you said in your post, the situation where it would get really nasty is if, in creating a branch of an application, you also wanted to create a branch of the common library. Craziness would ensue rapidly, I suspect. Fortunately this hasn’t been a concern as of yet - we try and keep as little functionality as possible in the common code base, and try to update it as infrequently as possible - but as time goes by things could get interesting.
I suspect ActiveResource-driven REST services might also be helpful as an _adjunct_ to these other techniques. Obviously performance can be a concern (less so if the service is on the same server), but it would be a great, plugin-style way of separating out model logic. Not so helpful for common ruby code, of course; nobody needs a web service that provides string formatting. =)