Today I encountered an issue that passenger forks too many workers than what we set (6) on qa servers. I used strace, the passenger worker is blocked by failed to writing to a socket, like

select(15, [], [13], [], [58, 915000])

fd 13 is a socket.

I also tried netstat and found the status for some redis socket connections are CLOSE_WAIT.

So I judged this is the problem the ruby redis clients are not closed correctly. This reminds me that passenger fork() nature, I checked our source codes, unfortunately, we didn't do anything special for passenger fork.

This is the link tells you how to close the redis connection after passenger forks a worker. After deploy the new codes to qa servers, passenger never forks more workers than we expected. But the workers still hang up according strace result, that means some workers keep inactive status, they won't be able to handle any requests. Wooops...

I looked through the redis-rb source codes, we used redis 2.0.5, it didn't handle TIMEOUT error and always retry writing to redis. Fortunately, the latest redis version is 2.2.2 and it already fixed this issue, retry 3 times, if still failed, the release the connection.

Now it works fine, no unexpected additional passenger workers and no unexpected inactive workers.

I made a mistake when merging branch last week, I forgot to remove a conflict syntax "<<<<<< HEAD" and push it to remote repository. It breaks other one's development. So stupid to make such mistake.

To avoid making such mistake anymore, I write a git hook .git/hooks/pre-commit to check conflict syntax "<<<<<<" and ">>>>>>"

#!/usr/bin/env ruby

`git diff-index --name-status HEAD`.split("\n").each do
|status_with_filename|
  status, filename = status_with_filename.split(/\s+/)
  next if status == 'D'
  File.open(filename) do |file|
    while line = file.gets
      if line.include?("<<<<<<<") || line.include?(">>>>>>>")
        puts "ERROR: #{filename} is conflict"
        exit(1)
      end
    end
  end
end

It will prevent you from committing conflicts.

We are using RabbitMQ as our message queue system, ruby client is workling. This week we encountered a strange issue, we create a notification, and define an after_create callback to ask workling to find that notification and then push the notification to twitter or facebook, it works fine except that sometimes it will raise an error said "can't find the notification with the specified ID"

class Notification < ActiveRecord::Base
  after_create :asyns_send_notification
  ......
  def async_send_notification
    NotificationWorker.async_send_notification({:notification_id => id})
  end
end

class NotificationWorker < Workling::Base
  def send_notification(params)
    notification = Notification.find(params[:notification_id])
    ......
  end
end

It's wierd the notification_id is passed to NotificationWorker, that means the notification is already created, the notification is supposed to be existed.

After talking with MySQL DBA, we find the problem is the find sql is executed before insert transaction is committed.

Let me describe it step by step.

1. Notification sends "Transaction Begin" command
2. Notification sends "INSERT" command
3. Notification gets "next sequence value" as new object id
4. Notification sends "new object id" to NotificationWorker
5. NotificationWorker sends "SELECT" command to find notification object
6. Notification sends "Transaction Commit" command

As you seen, at step 5, the new notification is not existed in the mysql database yet, so the error "Not found" will be raised.

To solve this issue, we can use after_commit callback.

In rails 2.x, we should install after_commit gem, in rails 3.x, after_commit callback is supported by default.

class Notification < ActiveRecord::Base
  after_commit_on_create :asyns_send_notification
  ......
  def async_send_notification
    NotificationWorker.async_send_notification({:notification_id => id})
  end
end

So Notification asks NotificationWorker to run only after the transaction is committed.

I thought reset_counter method is to reset a counter_cache column to be 0, but it is not. After trying several times, I finally realize that reset_counter is to update the value of counter_cache column to the exact count of associations. The usecase of reset_counter is when you add the counter_cache in migration and update the counter_cache value, like

def self.up
  add_column :posts, :comments_count
  Post.all.each do |post|
    Post.reset_counter(post.id, :comments)
  end
end

it will add comments_count column to posts table, and calculate the comments count for each post, and set it to posts' comments_count column.

I didn't find a method to reset the counter_cache column to be 0, why? Because counter_cache is used to cache the association count, it will be incremented and decremeneted automatically, you should never reset it 0. If you find you need to reset counter_cache to 0, that means it's a wrong usage of counter_cache.

Rspec 2 introduce a very efficient way to test only one test or one test suit, it's filter_run.

You should first add filter_run in rspec/spec_helper.rb

config.filter_run :focus => true

Then you can tell rspec to test only one test you are focused by

it "should focus now", :focus => true do
  ...
end

rspec will only test this spec, :focus => true can be applied on describe/context as well.

One problem is that if there is no :focus => true on your tests, rspec will do nothing, but most of time we are expecting to test all specs if no focus is true, so you should add a line to spec_helper as well.

config.run_all_when_everything_filtered = true

As the name implies, rspec will test all specs if no focus filter.

Another you may interest that you can also define filter_run_excluding

config.filter_run_excluding :slow => true

rspec will run all specs except what specs are marked as slow.