Fixed: Rubymine doesn’t show files in “library” directories

If you work on a medium or large-sized Rails project, there’s a good chance that you utilize the vendor/gems or vendor/engines directories to help keep your code organized into modules (as opposed to stuffing hundreds of models, controllers and views into a web of subdirectories in the app folder).

Unfortunately for us, the Powers that Be at Rubymine do not index any of these directories in file or symbol searches under the belief that any file in these directories must be third party library code, and therefore not shown unless one checks the “include non-project files” every time they want to look up their files. Even with this burdensome workaround, one would still find their methods/files listed below any partial matches that may occur in the main project.

The fix is to edit your Rubymine project’s iml file, it’s located at [project dir]/.idea/[project name].iml. You’ll need to find “” and add the following lines below it:

...
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/vendor/gems" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/vendor/engines" isTestSource="false" />
...

You’ll then need to close and re-open the project, and voila.

I have repeatedly and unsuccessfully lobbied Jetbrains to at least create an option for users to make their own decision about whether vendor/engines and vendor/gems should be treated as libraries, but they have denied the motion, telling me to file a bug for it. I did so, it lives here: https://youtrack.jetbrains.com/issue/RUBY-18315. If you care about this file, you should login to Youtrack and cast a vote for it. Or leave a comment on this blog.

Determine Ruby GC Garbage Collection Settings

Ever wish that you could determine what garbage collection settings your current running version of Ruby was using? Me too, but Google was not much help for it. There are two techniques for getting the current GC settings.

Technique 1: Use sh

In Rails console (or debugger), you can run shell commands by surrounding your command in the “ characters. So you can do:

`echo $RUBY_HEAP_MIN_SLOTS`
`echo $RUBY_HEAP_FREE_MIN`
`echo $RUBY_GC_MALLOC_LIMIT`

The pros of this approach is that it maps directly to the way that most people configure their GC settings: by setting environment variables. This is a means to be certain that the files you think are getting sourced when you start Ruby are being loaded correctly.

Technique 2: Use GC.stat

As of Ruby 1.9, running “GC.stat” can tell you all sorts of information about your current GC stats. The drawback of this approach is that there isn’t clear mapping between the environment variables you set and the data that GC.stat returns. Anecdotally, it appears that GC.stats[:heap_free_num] corresponds to RUBY_HEAP_MIN_SLOTS. I’m not sure yet how the other variables get reflected, but feel free to chime in with a comment if you know how the standard Ruby GC settings map to the hash keys returned by GC.stat

Rails 3.2 Performance: Another Step Slower

Having a large codebase means that we don’t upgrade our version of Rails very often (we’re averaging once every two years, with about 1-2 weeks of dev time per upgrade). Every time we do upgrade, though, one of the first things that I’m curious to inspect is the performance delta between versions.

For our previous upgrade, I documented our average action becoming about 2x slower when we moved from Rails 2.3 to Rails 3.0, with an action that had averaged 225ms climbing to 480ms. Luckily, in that episode we were able to pull out some tricks (GC tuning) such that we eventually got the same action down to 280ms. Still around 25% slower than Rails 2.3, even implementing fancy new tricks, but we could live with it.

When we finally decided we had to move from Rails 3.0 to 3.2 to remain compatible with newer gems, I was understandably anxious about what the performance drop was going to be based on our past experience. With the numbers now in hand, it looks like that apprehension was warranted. Here is the same action I profiled last time (our most common action – the one that displays an item), on Rails 3.0 before upgrade:

Most common action before upgrade, averaging 301 ms over 3 hours time window

 

And here it is now:

After upgrade, same time period as last week, averaging 423 ms over 3 hour time window

 

The problem with 3.2 is that, unlike last time, we don’t have any more tricks to pull out of our hat. We’ve already upgraded to the latest and greatest Ruby 2.0. We’ve already disabled GC during requests (thanks Passenger!). When we made these upgrades, they sped up our Rails 3.0 app around 25%. That performance improvement has now been overshadowed by the 40% slower controller and view rendering we endure in Rails 3.2, making us slower than we were in 3.0 before our Ruby optimizations.

Suffice it to say, if you have a big app on Rails, you have probably learned at this point to fear new versions of Rails. I fully empathize with those who are forking over bucks for Rails LTS. If we didn’t need compatibility with new gems, staying on 2.3 would have left us about 100% faster than Rails 3.0, which in turn is about 40% faster than Rails 3.2.

New Rails trumpets improvements like “ability to build single-page web apps” and “tighter security defaults” and “streamlining, simplifying” the constituent libraries. The closest we’ve seen to a performance improvement lately was that 3.2 made loading in development faster (1). This was certainly a fabulous improvement (took our average dev page load from 5+ seconds to 1-2), albeit one we already had in Rails 3.0 thanks to active_reload.

My sense is that performance has become the least of the concerns driving Rails development these days, which, if true, is a shame. If Rails put equal time into analyzing/improving performance as it does to “streamlining, simplifying,” it’s hard to believe that we would keep swallowing 40%-100% performance setbacks with each release. Maybe a partnership with New Relic could help the Rails team to see the real world impact of their decisions on the actual apps being built with their platform? If others’ experience is similar to ours, that would be a lot of pain felt by a lot of people.

I admit I’m a bit reluctant to make this post, because Rails has given so much to us as a platform, and our business is too small at this point to be directly involved in improving performance within Rails. We will, however, continue to post any salient optimizations that we discover to this blog and elsewhere.

My primary concern though, and the reason I am posting this, is that if Rails keeps slowing down at the rate it has, it makes me wonder if there will be a “point of no return” in the 4.x or 5.x series where it simply becomes too slow for us to be able to upgrade anymore. Each new release we’ve followed has been another step toward that possibility, even as we buy ever-faster servers and implement ever-more elaborate optimizations to the compiler.

Has anyone else out there upgraded a medium-to-large webapp from Rails 2 -> 3 -> 4? I’d be very curious to hear your experience? The lack of results when Googling for “Rails performance” has always left me wanting for more details on other developers upgrade experiences.

(1) New caching models may improve performance as well in some scenarios, as could the dynamic streaming when used with compatible web servers. For the purposes of this post I’m focusing on “performance” as it pertains to dynamic web apps that run on a server, which means stuff like interpreting requests, interacting with the database, and rendering responses.

Ruby slice to end of an array

It’s popular enough to be a Google-suggested search, but not popular enough to have a good result yet.

If you want to slice to the end of a Ruby array, and/or get the end of a Ruby array, what you want is

arr[1, -1] # -1 means all the rest of the array

Simplest AJAX upload with Rails Carrierwave and jQuery

The time has finally come for a follow-up to my post from a couple years ago on using jQuery, attachment_fu, and Rails 2.3 to upload an asset to my blog. I wanted to share the updated version of my attempt to determine the absolute minimal code necessary to implement AJAX uploads on Rails 3 with Carrierwave.  As was the case a few years ago, the Google results still tend to suck when searching for a simple means to accomplish an AJAX upload with Rails — the most popular result I could find this evening was a Stackoverflow post that detailed 9 (ick) steps, including adding a gem to the project and creating new middleware.  No thanks!

The Javascript from my previous example is essentially unchanged.  It uses jQuery and the jQuery-form plugin. The main challenge in getting a AJAX uploading working is that form_for :remote doesn’t understand multipart form submission, so it’s not going to send the file data Rails seeks back with the AJAX request. That’s where the jQuery form plugin comes into play. Following is the Rails code that goes in your html.erb. Remember that in my case I am creating an image that will be associated with a model “BlogPost” that resides in the BlogPostsController. Adapt for your models/controllers accordingly:

<%= form_for(:image_form, :url => {:controller => :blog_posts, :action => :create_asset}, :remote => true, :html => {:method => :post, :id => 'upload_form', :multipart => true}) do |f| %>
 Upload a file: <%= f.file_field :uploaded_data %>
<% end %>

Here’s the associated Javascript:

$('#upload_form input').change(function(){
 $(this).closest('form').ajaxSubmit({
  beforeSubmit: function(a,f,o) {
   o.dataType = 'json';
  },
  complete: function(XMLHttpRequest, textStatus) {
   // XMLHttpRequest.responseText will contain the URL of the uploaded image.
   // Put it in an image element you create, or do with it what you will.
   // For example, if you have an image elemtn with id "my_image", then
   //  $('#my_image').attr('src', XMLHttpRequest.responseText);
   // Will set that image tag to display the uploaded image.
  },
 });
});

Now, chances are you’re uploading this asset from a #new action, which means that the resource (here, the BlogPost) that will be associated with the image has yet to be created. That means we’re going to need a model that we can stick the AJAX-created image in until such time that the main resource has been created. We can do this if we create a migration for a new BlogImage model like so:

def self.up
  create_table :blog_images do |t|
    t.string :image
  end
  add_column :blog_posts, :blog_image_id, :integer # once created, we'll want to reference the BlogImage we created beforehand via AJAX
end

The corresponding BlogImage model would then be:

class BlogImage < ActiveRecord::Base
  mount_uploader :image, BlogImageUploader
end

Of course, if your resource already exists at the time the AJAX upload will happen, then you’re on easy street. In that case, you don’t have to create a separate model like BlogImage, you can just add a column to your main resource (BlogPost) and mount the uploader directly to BlogPost. In either case, the BlogImageUploader class would be setup with whatever options you want, per the Carrierwave documentation.

Continuing under the assumption that you will separate your model from the main resource (in this case, the BlogImage, which is separate from the BlogPost), we can create this image before the BlogPost exists, and stash the BlogPost id however you see fit. Thus, your controller’s #create_asset method will look like:

def create_asset
  blog_image = BlogImage.new
  blog_image.image = params[:image_form][:uploaded_data]
  blog_image.save!
 
  # TODO: store blog_image.id in session OR pass ID back to form for storage in a hidden field
  # OR if your main resource already exists, mount the uploader to it directly and go sip on a 
  # pina colada instead of worrying about this
 
  render :text => blog_image.image.url
end

And that’s it. No new gems, plus low-fat Javascript and controller additions.

Bonus section: How to embed this AJAX upload form in the form for its parent resource

One of the more common questions from my last post was how to display this AJAX image upload amongst the form for another resource. There are many ways to accomplish this (see comments from last post if you’ve got time to kill), but in keeping with the spirit of simplicity in this post, one fast hack I’ve used:

  1. After all the form fields for the main resource, close the form without a submit button
  2. Insert the AJAX form
  3. Add a link below the AJAX form that is styled to look like a button. Have this link call Javascript to submit your main form

Not going to win any beauty contests, but easy to setup and gets the job done.

Rails Exception Handling and Notification with Errbit

Bonanza has travelled a long road when it comes to trying out exception handling solutions. In the dark & early days we went simple with the Exception Notification plugin. The drawbacks of it were many, starting with the spam that it would spew forth when our site went into an error state and we’d end up with thousands of emails. There was also no tracking of exceptions, which made it very difficult to get a sense for which exception was happening how often.

Eventually we moved to HopToad (now Airbrake). It was better, but lacked key functionality like being able to close exceptions en masse or leave comments on an exception.

From there we moved to Exceptional, which we ended up using for the past year.  It was alright, when it worked.  The problem was, for us, it frequently didn’t work.  Most recently, we spent the last week having received two exceptions reported by Exceptional, when New Relic clearly showed that hundreds of exceptions had happened over that time period.  Also damning was the presentation of backtraces, which were hard to read (when present), as well as an error index page that made it difficult to discern what the errors were until they were clicked through.

Enter Errbit.  Jordan found this yesterday as we evaluated what to do about the lack of exceptions we were receiving from Exceptional.  Within a couple hours, he had gotten Errbit setup for us, and suddenly we were treated to hundreds of new exceptions that Exceptional had silently swallowed from our app over the past year.

But it’s not just that Errbit does what it is supposed to — it’s the bells and whistles it does it with.

Specifically, a handful of the features that make Errbit such a great solution:

  • Can set it up to email at intervals (e.g., 1st exception, 100th exception) so you hear about an exception when it first happens, and get reminded about it again later if it continues to be a repeat offender
  • Allows exceptions to be merged (or batch merged) when similar
  • Allows comments by developers on exceptions, and shows those comments from the main index page so you can quickly see if an exception is being worked on without needing to click through to it
  • Easy to read backtrace, plus automagic tie-in to Github, where you can actually click on the backtrace and see the offending code from within Github (holy jeez!)
  • Liberal use of spacing and HTML/CSS to make it much easier to read session, backtrace, etc relative to Exceptional and other solutions we’ve used
  • Open source, so you can add whatever functionality you desire rather than waiting for a third party to get around to it (a fact we’ve already made use of repeatedly in our first two days)
  • Open source, so the price is right (free)

Simply put, if you’re running a medium-to-large Rails app and you’re not using Errbit, you’re probably using the wrong solution.  Detailed installation instructions exist on the project’s Github home.

Fix: IRB pasting is super slow, typing in ruby debugger has lag

After numerous hours spent trying unsuccessfully to fix this problem by following the instructions outlined on a few StackOverflow posts, Jordan presented a recipe for fixing this problem (that actually fixed the problem) today.

The essence of the issue is that the readline package that gets installed with REE is by default some bastard version that lags, at least on our Ubuntu and Mint installations. Installing the rvm readline package did not fix it for either of us, nor did an assortment of experiments on compiling REE with different options. Here’s what did:

$> sudo apt-get remove libreadline6-dev
$> sudo apt-get install libreadline-gplv2-dev
$> rvm remove ree
$> rvm install ree

One problem you may encounter is that if you’re avoiding newer versions of Ubuntu until admit defeat about Unity, the “libreadline-gplv2-dev” package is by default only present in Oneiric and above. Here’s where I found the package versions that worked with Maverick: https://launchpad.net/~dns/+archive/test0/+sourcepub/2252776/+listing-archive-extra. After downloading the packages from this link, the install sequence became

$> sudo apt-get remove libreadline6-dev libreadline5
$> sudo dpkg -i libreadline5_5.2-9~maverick0_amd64.deb
$> sudo dpkg -i libreadline-gplv2-dev_5.2-9~maverick0_amd64.deb
$> rvm remove ree
$> rvm install ree

Create a Custom Thumbnail Processor in Carrierwave

The latest in my of “shouldn’t this be better covered in Google and docs if people really use CarrierWave?”-series. Creating a custom thumbnail processor in Carrierwave is pretty straightforward, but not from any search query I could construct. The gist:

class MyUploader &lt; CarrierWave::Uploader::Base
  version :custom_thumbnail do
    process :some_fancy_processing
  end
 
  def some_fancy_processing
    # Here our context is the CarrierWave::Uploader object, so we have its full
    # assortment of methods at our disposal.  Let's say we want to open an image with
    # openCV and smooth it out.  That would look something like:
    cv_image = OpenCV::CvMat.load *([ self.full_cache_path ]) # See past blog post on the origins of full_cache_path
    cv_image.smooth(5,5)
 
    # The key to telling CW what data this thumb should use is to save our output to
    # the current_path of the Uploader, a la
    cv_image.save_image(self.current_path)
  end
end

Just call #custom_thumbnail.url on your Uploader instance, and you should get the path to the custom result you created.

Using this framework you should be able to get CW to perform whatever sort of custom image processing you want. Thanks to these fellows for helping me decode the #current_path magic here.

Howto: Store a CarrierWave File Locally After Uploading to S3

Have now needed to do this twice, and both times have required about an hour of sifting through Google morass to figure out how to pull this off. The situation we’re assuming here is that you have some CarrierWave file that you’ve stored to a remote location, and you want to get a copy of that file locally to manipulate it. With promising method names like “retrieve_from_cache!” and “cache!” and “move_to_cache” you too may become entangled in the maze of what the hell you’re supposed to be calling to accomplish this. Here’s what.

Step 1: Retrieve from your store (the external place your file exists) to cache (your local machine).

my_cw_uploader.cache_stored_file!

After running that, if you call

my_cw_uploader.cached?

You should get a long string that is the filename for the file that got cached. Note that the cache status of a file is only saved on a per-object basis, so if you try something like

Image.find(1).my_cw_uploader.cache_stored_file!

… you will never again see the object that did the caching, and you will never be able to use that cache. Only call #cache_stored_file! on an object you are keeping around.

Step 2: Go access the file that you cached. Not quite as easy as it sounds. For this, I mixed in the following into my uploader class

module CarrierWave::Uploader::Cache
	def full_cache_path
		"#{::Rails.root}/public/#{cache_dir}/#{cache_name}"
	end
end

So now when you call

my_cw_uploader.full_cache_path

You’ll get the full pathname to the file you downloaded.

Hope this helps save someone else from an hour of Google hell.

Ruby debugger list local variables

Short and sweet. You want to list local variables in ruby-debug? Try

info locals

You can also get ruby-debug to list your stack, instance variables, and much more. Type in plain old “info” into debugger to see a full list of what ruby-debug is willing to reveal to you.