Development Blog: Viget Labs

Ben Scofield, Development Director, January 15, 2008 15

Over the past few months, I’ve been talking here and there about a project I’ve been working on - Sandstone. The general idea is that, while Rails makes it easy to build an application quickly, it’s unreasonably difficult to integrate two distinct Rails projects - but if you have an e-commerce site, you may have a need for static content that doesn’t justify the effort of running an entirely distinct Rails application like Radiant CMS. Sandstone meets this need by providing a drop-in, customizable CMS as a plugin, ready to be installed on any given Rails application.

There are a couple of really interesting things about this project. The first is that it’s an instance of what we at Viget are calling “resourceful plugins” - that is, plugins that provide a complete set of resources and functionality around them in a specific domain. These plugins are intended to be easy to integrate into any existing Rails application, and to allow for extensive modification and customization without losing the ability to upgrade the core plugin code.

That last bit is especially important, since that’s where a lot of alternatives fall down - sure, there are plugins out there that add CMS functionality, but once you invest the time in customizing them to your particular project you’re wed to the particular version you installed. If a new version comes out with some compelling features, you have to do a lot of work to upgrade. With a resourceful plugin, upgrades are relatively painless. When you install Sandstone, for instance, it copies a series of controllers, models, and views into your app folder so you can tweak them to your heart’s desire. The bulk of the core functionality, however, resides in modules defined in the plugin (which are included in the copied files). To upgrade, then, you just update the plugin and leave the copied files alone. The only problems you’ll run into are ones where the new version of the plugin adds some significant new functionality (with new controllers, models, and views), but we have a strategy for dealing with that easily, too.

The other aspect of the Sandstone project (and our other resourceful plugins) that excites me is the fact that we’re open sourcing it. If you’re interested, you can grab the current version of Sandstone (so new it doesn’t even have a version number!) from http://svn.extendviget.com/lab/sandstone/trunk. If you’d like to contribute, we have Trac set up, so you can open bugs and feature requests and otherwise participate in the process. We’re hoping that Sandstone can help meet the needs of a lot of developers, so we welcome your feedback.

Update: Changed Trac URL to reflect the project’s new location

image taken by Ozyman; from http://www.flickr.com/photos/ozyman/211749949/

Brian Wynne Williams, CEO & Co-Founder, June 22, 2007 2

“Can Ruby on Rails scale when my web traffic takes off like a rocket ship?”

This is a common question we get (or some variation).  I have a simple answer:

“Yes, it can.”

As with any technology or framework, it’s easy to build something that won’t scale.  Furthermore, there are applications where Ruby/Rails isn’t the right solution.  However, we believe that Rails will scale fine if you make smart architecture decisions.  Heavily-used apps like Basecamp and large-scale sites like Revolution Health show this; but, lately people like to talk about Twitter‘s problems.

Silicon Valley-based Powerset just announced that they’re building their front-end in Ruby (though not necessarily Rails).  They addressed the question of scaling and Twitter’s situation with some first-hand insight:

We’re not worried about scaling.
So, inevitably, whenever we talk about Ruby or Rails scaling these days, someone brings up Twitter and its scaling problems in the past. Twitter is right down the block from our offices and I know several of the devs personally, so before we made a final decision I arranged a sit down with Twitter’s lead developer, Blaine Cook, to talk about the situation. Blaine was kind enough to let me bring along our Search Architect (and former search architect at Yahoo!) Chad Walters, our Head of Product Scott Prevost, and our COO Steve Newcomb, to poke and prod and get their questions answered. The simple fact is that Ruby wasn’t the source of Twitter’s woes. As it often happens with rapidly-growing sites, they ran into architectural problems. Some design decisions don’t hurt until they reach a massive scale and, at that point, you have to rethink your approach. In an email he writes:

For us, it’s really about scaling horizontally - to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement; but, thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10,000% faster than it was in January.

---

Twitter’s explosion of growth in March certainly created pain points for them (gotta love slide 7 here) and a lot of lessons learned; but, it’s not an indication that the framework inherently won’t scale.

Our belief is that when building a web app, agility is more important that built-in scalability.  “Premature optimization” causes more problems than it prevents.  Focus on building a site that people will really use, then refactor and optimize over time as your audience grows.  Rails is made for agility.  You always hear about development speed, but test-driven development is easier and faster in Rails as well.  Both factors mean that making architecture improvements to an already-deployed site is more practical.

Regardless of technology, build with scaling in mind, not necessarily built-in.  Friendster didn’t scale, but that seemed to be a management decision.  Had Twitter obsessed about trying to build for the number of users it’s supporting today, they probably never would have made it out of the gates.  And then how would I possibly know that Darth Vader just ate a burrito?

Patrick Reagan, Development Director, June 05, 2007 1

We had a great time at RailsConf this year. Ben’s presentation went well and got a great response from the crowd that attended. I had a chance to chat with some people (both from our local group and elsewhere) about the content of the presentation, and the responses were pretty consistent.

For the developers who had faced the same issue of building a dynamic site with a large amount of static content, the solution met their needs. I ran into a lot of people who had just been creating a single controller (e.g. GeneralController) to handle all the static content for their sites. This approach works for a while, but as the static content grows – which was the case when we built the ACVA site – things quickly become cumbersome.

I also talked with others who were misled by the title and wondered “why would you use Rails to build a static site?” The answer to that is… “you don’t.” If you’re building a truly static site, Apache, Lighttpd, or Tux are your friends and will serve up static files faster than Rails can. In fact, Ben’s solution relies on a front-end web server to serve up the cached pages generated from the static site framework.

So for those that missed out, here’s what the framework allows you to do (implemented as a plugin-app):

• Upload or create content files – In typical CMS CRUD fashion you can add content directly to the ‘pages’ table through a web front-end either using a textarea or a standard file upload control. From there, the content is then immediately available in your application.


• Generate site maps – Because the page content is stored in the database and the Page model uses acts_as_tree, we’re able to traverse from the root and generate a site map for display. This design decision, along with providing modification timestamps, also helps when you need to generate the XML for use with Google Webmaster Tools and other search engines that use this site map standard.


• Provide customized page-specific variables – Again, because of acts_as_tree in the Page model, we can provide a hierarchical inheritance model for page-specific variables. Setting a variable at the parent level makes it available to all the children with the ability for each child to override the parent’s default setting.


• Search for terms in your “static” content – The framework plays well with your current search solution. If you’re using Ferret, Solr, or Hyper Estraier you can search both your existing CRUDified content and your new static content.

If you want to test it out, you can download the plugin and view the slides from the Viget site. In the near future, we’ll be pulling the code into a public SVN repository – stay tuned.

Patrick Reagan, Development Director, May 20, 2007 6

Things have been pretty fast-paced at RailsConf, so I’m just now getting around to talking about DHH’s keynote.  After talking a bit about the popularity of Rails since last year’s conference in Chicago, he gave a quick run-down of the top 9 new features that will be part of Rails 2.0.  Some of this will look pretty familiar if you’ve been following development on edge:

1. Breakpoints – After a Ruby bugfix, the breakpointer in Rails stopped working.  Rails 2.0 adds this back in with the ‘debugger’ keyword added to your controller action (previously, you could use ‘breakpoint’).  Similar to the old breakpoint functionality, the debugger will drop you into an IRb console so that you can inspect local variables.  In addition to inspection, you can now traverse the stack, print backtraces, and go back and forth between an IRb session.


2. HTTP Performance – The performance of a web application suffers as included JavaScript and stylesheets increase in number since they too must be downloaded.  Rails solves this problem by modifying the helpers so that these files are compressed for transfer and cached on the client:
   

   <%= javascript_include_tag :all, :cache => true %>
   
   
   <%= stylesheet_link_tag :all, :cache => true %>
   
   
   

   Due to the way that browsers operate, they will only make a limited number of connections to a single host when downloading assets referenced on a page.  The new use of dynamic asset servers will allow browsers to make more simultaneous downloads from the same host.  This is accomplished with a simple configuration change:

   config.action_controller.asset_host = 'assets%d.example.com'

   The sweet spot is cycling between 4 hosts to serve up all your assets; you’ll have to add the approprate CNAME records to match the configuration.

3. Query Cache – Rails is now smart enough to re-use data that it has previously retrieved from the database.  When ActiveRecord sees a query for data that it has already loaded, it will hit the cache instead of re-querying the database.  Since this happens at the application layer, you don’t have to worry about the byte sensitivity of the MySQL query cache.  You also don’t have to worry about expiring the cache since it gets invalidated each time a change is made to the database (e.g. on insert, update, or delete)
4. ActionTemplate Renderer – There are new conventions for templates that give greater detail about what each template will render.  Views now use the new .erb suffix and indicate their type as part of the template name (e.g. index.html.erb).  The same applies with XML files that are created with Builder (e.g. index.rss.builder).
5. Environment Configuration – As your environment configuration gets complicated, it becomes more useful to break up the configuration into separate files. Rails will now make this easier and allow you to share your more generalized configurations with your other applications.
6. Sexy Migrations – The migration DSL is getting a facelift and things are getting swapped around.  Where we used to have this:

   create_table :users do |t|
   t.column :first_name, :string, :null => false
   t.column :last_name, :string, :null => false
   t.column :account_id, :integer
   t.column :description, :text
   t.column :created_at, :datetime
   t.column :updated_at, :datetime
   end
   

   
   We now have this:

   create_table :users do |t|
   
   
   t.integer :account_id
   
   
   t.string :first_name, :last_name, :null => false
   
   
   t.text :description
   
   
   t.timestamps
   
   
   end
   
   
   

7. HTTP Authentication – While HTTP authentication isn’t attractive in HTML-based applications, it’s very useful for services that consume data.  This works great for authentication-aware applications like feed-readers.
8. The MIT Assumption – Due to licensing confusion for people using plugins, the new plugin generator now creates a license file by default.  The assumption is that you are distributing under the MIT license otherwise you have to modify the file to meet your needs.
9. Spring Cleaning – In addition to deprecating some features, non-core components like the in-place editing macros have been pulled out into plugins. This is also where you can find useful tools like resource_feeder and open_id_authentication.

If you want to check out some of these features (not everything is out there right now), grab the latest code from SVN or use piston to freeze your application against edge.

Patrick Reagan, Development Director, May 10, 2007 4

As Mark mentioned in his latest post, most of our previous applications have been built with PHP (including Squidoo).  Despite all our years of collective PHP experience, we made the decision to switch new development to Rails less than a year ago.  In that time, we’ve hosted a training session, attended a conference, connected with other developers in the community, and even had one of our top developers selected to speak at this year’s RailsConf. 

Admittedly, the switch wasn’t altogether painless; but, as I look back on the past 7 months and how far we’ve come, I’ve made notes about everything that Rails provides that has made our lives easier.  This isn’t an exhaustive list (what did you expect?); but, it does cover the top 5 from across the Development Lab here at Viget:

1. ActiveRecord – Obviously a critical component of the Rails framework and something that the PHP world still seems to be lacking.  It’s clear which ORM library you’ll use when developing in Rails; but, there isn’t a clear choice when developing in PHP.  While the myriad PHP frameworks provide a multitude of choices, nothing is as developer-friendly as ActiveRecord.
2. Test::Unit – Though we were new to Rails, we had been using Test-driven development practices even in our PHP applications.  The framework that we developed in house pulled in SimpleTest to handle all our model and controller tests; but, the integration wasn’t as nice as we would have liked.  With Test::Unit and its integration into Rails, writing and running tests became both easier and faster.  Through the use of fixtures and database transactions, we no longer felt the pain of writing mundane (and sometimes complex) set up and teardown methods for managing our test data.
3. Mocking Libraries – When working in PHP, we found ourselves changing the behavior of our classes just to make them easier to test.  Though SimpleTest did provide mocking capabilities, libraries like Mocha and FlexMock, combined with the power of Ruby, let us mock out method calls deep inside our code – something that just wasn’t possible for us before the switch.
4. Filters – I recently found myself maintaining an existing PHP application where I needed to add some complex permissions checking.  While the task took quite a while, it would have been easily solved in Rails through the use of filters.  The ability to remove duplication with before filters and clean up exception handling with around filters has been a huge productivity boost for us.
5. Ruby – Last on the list – but it’s truly the cornerstone of the framework.  We did have some of the hurdles involved with learning a new language; but, the time we spent ramping up has been well worth it.  The syntax of Ruby is much more elegant and consistent – something that PHP has been sorely lacking for a long time (I hear it’s coming in PHP 6).

As we continue to push for using Rails for web application development, I feel confident that we’re giving our clients the best solution possible while decreasing the time it takes to bring their products to market.