Hank Beaver

This example uses Ruby to telnet into Varnish and issue the “url.purge .*” command. The only gotcha is that the varnish telnet server does not issue a command prompt which causes Ruby telnet to timeout and get cranky. Well a little exception handling hacks past this. Enjoy.

require 'rubygems'

namespace "varnish" do

  desc "Purge ALL urls from Varnish"
  task :global_purge => :environment do

    #It WILL timeout, just accept it. Varnish does not have a command prompt.
    require 'net/telnet'
    @result = ""
    begin
      localhost = Net::Telnet::new("Host" => "localhost",
      "Port" => 6082,
      "Timeout" => 5)
      localhost.cmd("url.purge .*") { |c| @result = c}
    rescue Exception
      if @result.include? ("200 0")
        puts "varnish purged OK."
      else
        raise "Varnish not purged."
      end
    end
  end

end

Introduction

At my current gig with Primedia, we have some pretty high volume Rails sites. Since having highly available sites are essential for our core business, we have to be very attentive to production issues. Below is a list that I’ve compiled recently in my head and by working with some very bright folks. This doc is to help Ops/Dev remember some basics and tricks when debugging issues. So remember to try this stuff.

Non-Technical (READ FIRST)

1. If the issue is not a major impact on production, make sure all your troubleshooting is passive! You might introduce an outage if you try serious troubleshooting.
2. Confirm the issue with Ops/Infrastructure in your presence, this gets buy-in from with the folks who have official root access.
3. As a developer, you should not work alone on production systems except to gather information (if you get temporary access). All changes (in seeking issue mitigation) should be paired with Ops/Infra. This will avoid finger-pointing and headaches later.

Technical

Elimination + log files is my troubleshooting method of choice.

1. Skip the load balancer if possible and hit direct IP/Port of a single app server to eliminate load balancer in mix.
2. Enable DEBUG log level and restart app server(s). Preferably, take an app server out of load balancer pool and work on in isolation. Regardless, DEBUG can slow response signficantly so be aware. However, there are situations where DEBUG is the only way to get an indication from Rails what is going on.
3. Tail log file of single app IP/Port to squelch noise from other servers and hit just that app server.
4. Make sure that the PIDs are not hung,old versions. Look at time/date of PIDS. If the deploy is screwed up, you can be deploying new code and yet the processes running are code from a month ago.
5. To be thorough, look at the kernel logs too. In Linux, /var/log/messages etc. You could have something else that is going on.
6. The logs should give status regarding database(s), but especially look for the words “timeout” if you have any custom network libraries that fetch data (we do) or “ActiveRecord”.  Perhaps even, grep for it.
7. Use a non-browser client, like Curl or wget to eliminate possible browser issues. Case in point, we had an issue related to ActiveRecord sessions that were migrated to another datacenter. When accessed via Curl there was no issue (Curl does not save cookies by default).

Other

1. If you cannot duplicate issue in development, remember to change your configs to look like production mode and run your development workstation in production mode. Rails behaves different in its modes.

Motivation:
As an avid Ruby/Rails programmer and longtime CMS coder and user the following is yet another effort to keep a list of current Ruby/Rails CMS’s. That is the goal of this post. I will use comments from others to update the list and hopefully keep and up-to-date list for all (including myself) to use. Heck, I would love to have others help review and keep list up to date.

External Resources:

http://www.widgetfinger.com/
http://www.ajaxlines.com/ajax/stuff/article/top_ruby_cms.php

http://webscripts.softpedia.com/downloadTag/ruby+cms

I want to move all files in this svn dir:

svn list http://svn.atld1/svn-prodops/sysadmin

to:

svn list http://svn.atld1/svn-prodops/sysadmin/scripts

Doing svn mv would see obvious and using xargs or -exec might work. I prefer not to use any brain energy on a shell script. I just do it in ruby in 5 minutes:

svn list http://svn.atld1/svn-prodops/sysadmin > /tmp/script.list

Write this to a ruby file:

f = File.open("/tmp/script.list")

f.each_line {|line|

cmd = "svn mv http://svn.atld1/svn-prodops/sysadmin/#{line.strip}
http://svn.atld1/svn-prodops/sysadmin/scripts/ -m
\"moving #{line} to scripts \" "

puts "running: #{cmd}"
`#{cmd}"

}

Now execute the file above after double checking. Viola done even with relevant comments for each move!

When instantiating a model class this error occurs:

instance = MyModel.new

NoMethodError: You have a nil object when you didn't expect it!

The error occurred while evaluating nil.has_key?

This is rectified by placing a super within model class.

def initialize

super

logger.debug("initialize VirtualMta: #{self.class}")

end

This all came up b/c I wanted to write a shared mixin module to have attachment_fu ready actions for all my controllers that have attachments. No need to put that crap in every controller. So in a mixin module when you want to do some fun reflection/introspection like so:

…

    @this_obj = self.controller_name.classify.constantize.find_safe
(id,logged_in_user.id)

…

In Dev this worked great. Tests were good, my mostly view/template developer partner, approved. Deploy to test and WHAMMO!

…

    "NameError ("Blogs | ListAll" is not a valid constant name!):

…”

What gives? Mmm, seems when you include a mixin, that is included in a controller. The self.controller_name will report incorrectly in Test and Prod. So I had to adjust the code a little using a new method:

        @this_obj = self.this_controller_name.classify.constantize.find_safe(
id,logged_in_user.id)

Below is the method(which is basically David’s code for controller_name).

###############################

#This method is here b/c it is apparent that Rails
#ActionController.controller_name

# is unreliable in Test and Production for some reason.

# attachment_symbol_name: Blogs | List_all_attachments

# attachment_model ERROR: "Blogs | ListAllAttachment"
#is not a valid constant name

###############################

def this_controller_name

    self_class = self.class.to_s.sub(/Controller$/, '').underscore

end

Soon the rest of this blog will deal with a basic overview of has_many associations and a practical use. For now look at this simple tag that uses has_many of a Blog model to show the Interests (or topics) for a given blog. You would use this helper_tag like so in your view, given that you have a single instance of a blog object called ‘blog’

<%= more_ugly_interests_tag(blog) %>

I hope this small article might help others:

#########################################
#Tag to show usage of has_many for a model.
#Ruby on Rails
#*this_model* is the actual instance of a blog or photo
#which has a list of interests.
#
#Call tag like this:
# <%= more_ugly_interests_tag(blog) %>
#the syntax below uses what Ruby calls a 'block'
#this is the format:
# some_variable_that_has_a_list.each { |list_item
# #do something with each item
# puts list_item + " !!!!"
# }
#########################################
def more_ugly_interests_tag(this_model)

to_return = "INTERESTS"

this_model.interests.each { |each_interest|

to_return =  << each_interest << "
"

}

#now we have end the whole function/method with what we want to return/print
to_return

end

All the Agile folks would say if you run/create your tests you might not even get this far. However, that’s not always reality for me.

1. ‘tail -f /development.log’

If you are on OSX, the above is easy.
On Windows, I’d suggest looking at Cygwin and getting tail working. It’s worth it.

a) open terminal.
b) Clear the screen before you reattempt what is broken. (Use Apple + K on OSX)
c) Stop tail right after getting the error (CTRL + C)
d) Look for the first line that is not normal Rails and debug output. See next area.

2. Skip all the fluff and go for Ruby errors. You will learn to see “stacktraces” and errors in Ruby/Rails after a little practice.

They will look something akin to this:

1)An error
2)The file that the error occurred in…
3)the line number/and problem code…
4)A sample of the code at and around the error.
…
5) A bunch of code that was run prior to this happening.

3. Did you recently change the file in question? If so, revert what you did?

4. Use logger.debug to print out something right before the error to see if your code is making it that far.

There are several ways out there to install RMagick: via ports, package installer, etc. My advice: before you even start working on these.

1. Install ALL updates from Apple for your OSX.

2. Update XCode to the latest downloadable version.

Doing the above will save you a whole weekend and avoid weird errors like this:
.libs/libMagick.dylib must have a ‘:’ between its file names
H

This is DEPRECATED, because Gavin Stark wrote a native ruby class last month. I will leave this up if someone wants a C implementation for OSX, but after that Gavin did the work in Ruby and I encourage looking there:

 http://hasmanyquestions.wordpress.com/2007/11/23/perfect-paper-passwords-in-ruby/

Ruby Extension for GRC’s Perfect Paper Passwords V3.
UPDATE 12/11/2007, current version only supports Mac OSX, I will be creating a Linux version shortly. Not sure if I’ll wrap a Windows version.
Ruby Extension for GRC’s Perfect Paper Passwords/PPP C (http://www.grc.com/ppp/) implementation by John Graham-Cumming.
Create with SWIG.
Version .01
CODE/LIB: http://www.1000fires.com/ppp3-Ruby-extension.zip
C Ref: http://www.jgc.org/blog/2007/11/steve-gibsons-ppp-new-version-3-in-java.html
NOTE: Because of the way C works, the first argument must be a space (not a null char).

Entry point to ppp3 is “Ppp3.main” Ruby call after include with a Ruby array as argument.

USAGE in ruby:

require ‘ppp3′ #ppp3.bundle#Usage 1, just get a random sequence key.
Ppp3.main([” “,” “])

# Usage 2, create an array and call with arguments.
# call the Ppp3.main call with this array:

a = Array[”",”efbda242bd1db23fd343b516a2a10d0eed08250e542a789811307f2011a92df6″,”0″,”2″]
Ppp3.main(a)

#Usage 3, call with implied array string.
Ppp3.main([”",”efbda242bd1db23fd343b516a2a10d0eed08250e542a789811307f2011a92df6″,”0″,”70″])

TODO. Add a Ruby convenience library to do some parsing and pretty printing as well as prepare for Rails integration.