Relentless Simplicity :: The Bonanzle Tech Blog

I have found it useful to populate my local development database with data from our production server in order to be able to get good test coverage.  However, a perpetual problem I’ve had with this approach is that it introduces an environment where sometimes images are available and sometimes they aren’t (the database knows about all the images, but some were uploaded locally, some reside on our main servers, and some are on S3).

What I’ve found is that even though Rails doesn’t give exceptions when it finds missing images, it does start to get painfully slow.  Each missing image it has to process usually takes about 2 seconds.  On pages with 5-10 missing images, the wait could be quite painful.

So I finally got fed up yesterday and wrote a hacky patch to get around this problem.  Here it is:

def self.force_image_exists(image_location)
 default_image = "/images/dumpster.gif"
 if(image_location && (image_location.index("http") || File.exists?(RAILS_ROOT +  "/public" + image_location.gsub(/\?.*/, ''))))
  image_location
 else
  default_image
 end
end

This function is part of a utility class (named “UtilityGeneral”) that we use for various miscellaneous tasks.  I call this method from a simple mixin:

if RAILS_ENV == 'development'
 module ActionView 
  module Helpers #:nodoc: 
   module AssetTagHelper
   # replace image tag
   def path_to_image(source)
     original_tag = ImageTag.new(self, @controller, source).public_path
     UtilityGeneral.force_image_exists(original_tag)
    end
   end
  end
 end
end

If anyone else works locally with images that may or may not exist, this wee patch should come in handy to save you from load times of doom on pages that are missing images.  It just subs in an alternate image when the real image doesn’t exist locally.

P.S. When I grow up, I want a blog about coding that lets me paste code.
P.S.S. 4/10: I grew up!

Bill Rails rails slow load images missing

Or do you notice that every time you visit a website or a submit a form, and you get the indefinite spinner of doom, that the url always seems to end in .aspx?

Hmmmm!

Bill Uncategorized

We are on the cusp of having Passenger running, but I am paranoid, based on our Mongrel experiences, of Passenger instances leaking memory up the wazoo and eventually exhausting our system resources.  With Mongrel, we’ve used monit to ensure that memory usage remains intact with each Mongrel, but I hadn’t found a straightforward way to do the same with Phusion yet.  So I’m improvising:

kill $(passenger-memory-stats | grep '[56789]..\.. MB.*Rails’ | awk ‘{ print $1 }’)

This single line (run via crontab) ought to do what our thousand line monit config file used to do:  kill off Rails processes that exceed 500 MB.  From my testing so far, it seems to do the trick.

I have verified that it does indeed kill one or multiple Rails processes started by Passenger if their memory usage is reported as being a three digit number that starts with 5-9.  Obviously if a Rails instance were able to jump past the 500-999 MB range in less time than the frequency of our cron task, that would be a problem.

Will report back once I’ve witnessed it at work in the wild.

Update from the wild: Yes, it works.

Bill Rails memory, monit, passenger, Rails

We had been racking our brains on this one for a couple weeks.  We have monit looking over our Mongrels, which usually keeps everything on the up and up.  But every so often, our server would go bananas and the nginx error log would flood with the message:

939#0: accept() failed (24: Too many open files) while accepting new connection

Usually the problem automatically resolved itself, but last night it didn’t.  Taking the error at face value, our server guy started looking at the number of open files on our system and the maximum files that could be opened (it’s confusing…  “ulimit -a” reports one limit while “cat/proc/sys/fs/file-max” reports another.  I think that the former might be for actual file system files opened and the latter might be for file handles (which also includes open IP connections and such)).  But even after upping the limit and rebooting repeatedly, the problem persisted.

After server guy (literally) fell asleep on the keyboard around 2 AM, I figured out what had really been happening: any time a new visitor came to our site, we were geocoding their IP with a service that had gone AWOL.  About a week earlier I’d noticed a similar slowdown of about 1-2 seconds with actions that created sessions, but I assumed it was the session creation itself that was causing the slowdown, when in fact it was the geocoding that happened alongside the session creation that was responsible for the lag.

Long story short, when nginx gives this error, what it really seems to mean is that it is holding too many open connections, and usually that is happening because you are using round robin dispatching (bad, I know, but we have our reasons) and one or more of the Mongrels is stuck and forcing the Mongrel queue to skyrocket.

The other lesson here is an obvious one that I’ve read many times before but have been slow to actually act on:  making remote API calls without timeouts is asking for trouble.   Here is a fine article if you’re interested in solving that problem in your own site before it is your ruin.

Bill Rails