Unixmonkey.net

I’ve got some data in an excel file that I need to put in the database and its far too much to do by hand, what will I do?

Lets throw some ruby at the problem!

First, excel it too darn complicated and proprietary a format to even mess with unless you are creating something really worth it, so lets open that .xls with Excel or OpenOffice and do a File -> Save As -> .csv (comma separated values) to get a file that is easier to work with.

Now, we could write our own CSV parser since its such a simple format, but why futz with it when someone else has already put out a good library for that will likely be more error tolerant? Lets use FasterCSV, as its pretty well-known.

Install by issuing:

sudo gem install fastercsv

Now you can just fire up script/console of your Rails app and type in the below, or just put this in a database migration to slurp up all that good spreadsheet data.

The below assumes you have a ‘users’ table with fields name, address, and email that are also rows in your excel file. Adjust as necessary.

require 'fastercsv'
FasterCSV.foreach("#{RAILS_ROOT}/myfile.csv") do |row|
  record = User.new(
    :name    => row[0], # first column of csv file
    :address => row[1], # second column
    :email   => row[2]  # third
  )
  record.save
end

That’s pretty awesome; now how can I export that stuff in the database back out to Excel again?

Lets use Ruport, the Ruby report gem!

sudo gem install acts_as_reportable

Toss the require statement somewhere obvious (like environment.rb or above the model you want to export), and put ‘acts_as_reportable’ in your model declaration.

require 'ruport'
class User < ActiveRecord::Base
  acts_as_reportable
end

Now I can do this kind of stuff to export to a csv file (again with script/console, but a migration should work equally well):

content = User.report_table.as(:csv) # convert your model table to CSV

or

content = User.report_table_by_sql("SELECT name, address, email FROM users").as(:csv)

Then write that to a file like so:

file = File.open("#{RAILS_ROOT}/report.csv", "w") # open file
file.print(content) # print that csv content to the open file
file.close          # close the file

Open that CSV file with Excel and amaze the sales team, your boss, or whoever.

This is a narrow view of what we can do with FasterCSV and Ruport, but I’m sure you can see how you could build out a format.csv in a respond_to block in a Rails controller, or have a setter in your model that sucks in an uploaded CSV to create some records.

These are some pretty great libraries, and I’m very glad they were able to help me load, combine, query and output some data I’d been working with in a pinch.

I hope this post serves to help someone else in a similar situation.

If you’ve ever wanted to keep track of revisions to document files or images in your Rails app, you are likely to want to use Acts_as_versioned, which is the authority on versioning database records, and Attachment_fu, which is the authority on uploading files with Rails.

The problem is that they don’t know about each other and will step on each other’s toes without some changes. This article serves as a quick introduction to each, and shows how to make the two plugins get along like best friends.

Acts_as_versioned was written by Rails Core Team member Rick Olsen (who also wrote attachment_fu and Restful_authentication among others) that essentially makes a mirror table of the one you want to version, and keeps every version of the record you are updating.

Say I have a document table with fields like this:

Acts_as_versioned will add a column “version”, and a separate table “document_versions”.

The document_versions table will look a bit like this

Setting up acts_as_versioned is pretty simple, I got most of my introduction to it from urbanhonking.com

Now every time you update the original document, the changes are saved in your main documents table, and the version column is incremented by 1.

After a few edits of the document, you’ll see the versioning information in the Document_versions table add up.

Great! We can now use some of acts_as_versioned’s built-in methods for determining if there are older versions, and be able to view or even revert to them.

Now lets add the ability to upload a file to attach to a document record with attachment_fu.

Attachment_fu is another plugin that makes uploading files and keeping track of them in the database relatively simple.

A good intro to attachment_fu can be found on Mike Clark’s blog

Attachment_fu would require a few changes to our documents table:

Don’t forget to add the same fields to your documents_versions table, too.

Once we’ve added the right file fields to the new and edit forms, and image_tag or download link on the show view, we’ve got working file uploads. Nice.

Try to edit a record by attaching a new file, the new file is displayed and the record is preserved as an older version in the versioned table. But if you try to view the old version…wait a minute? Where did my version 1 file go!

That’s right, attachment_fu deletes the old file when you add a new one (as it should if you aren’t versioning your data). Attachment_fu’s rename_file method is the one responsible for deleting (or renaming) the old file when a new one is added, so lets monkeypatch that in our model to not do anything.

def rename_file
end

Now, it will only overwrite the file if the filename is the same. Lets store each version in its own folder to keep them from clobbering each other by monkey-patching the path files get written to in our model also:

def attachment_path_id
  "/#{id}/v#{version}/"
end
def partitioned_path(*args)
  attachment_path_id + args.to_s
end

This changes the public path from /0000/0001/rep08.jpg to /1/v1/rep08.jpg

Now, if we want to display the image, we cannot use the ‘public_filename’ method, because it is only given to the Document model, and not the Document_Version model.

That’s okay, because with our new path arrangement, we can reliably predict where the old versions of the files will be kept. You can show them with some code similar to this in your views:

<% for version in @document.versions %>
 
  Version <%= version.version %>
  <%= image_tag("/documents/#{@document.id}/v#{version.version/" + version.filename) %>
  <hr />
 
<% end %>

Now, when we delete a record, attachment_fu only knows about the current document, and will leave behind orphaned files and folders from the old versions. Lets fix that by having it get rid of the document id folder.

Rails reserves some special methods (callbacks) for performing actions before or after other major actions, lets tap into that by defining a method that will magically get called every time we delete a record.

def after_destroy
  FileUtils.rm_rf(RAILS_ROOT + "/public/documents/#{id}/")
end

This translates into the shell command rm -rf and deletes our ID directory and everything inside it.

Hooray!

As a wrap up, lets look at our complete Document model:

class Document < ActiveRecord::Base
  acts_as_versioned
  has_attachment :storage => :file_system
 
  def rename_file
  end
 
  def attachment_path_id
    "/#{id}/v#{version}/"
  end
 
  def partitioned_path(*args)
    attachment_path_id + args.to_s
  end
 
  def after_destroy
    FileUtils.rm_rf(RAILS_ROOT + "/public/documents/#{id}/") if id
  end
 
end

I’ve whipped up a sample Rails app demonstrating the points and code in this article. It uses Rails 2.0.2 with the sqlite3 database.

Download it here: Attachments_versioned (240kb .zip)

I hope this saves some work for someone who wants to leverage these two excellent plugins by Rick Olsen (technoweenie) on the same model without having them fight too much.