Development Blog: Viget Labs

Ben Scofield, Technology Director, October 24, 2006 3

The standard framework for testing in Ruby on Rails is a thing of beauty; there are folders to drop your unit and functional tests into right off the bat, fixtures and mock objects are expected and encouraged, and when you use scaffolding you get stubs for the common CRUD tests for free. That helps make testing less onerous for developers and, as a result, applications are better tested.

Even Rails isn’t perfect, however, and there are some situations where the testing set-up created out-of-the-box just doesn’t work so well—and, unfortunately, one of those problem situations involves database transactions.

Put simply, transactions are a way to keep your data in a valid state. Say you have a banking application that allows people to transfer money from one account to another. The code might look like this:

andy_account.withdraw(300.00)
ben_account.deposit(300.00)


If all goes well, Andy’s account is $300 lighter, while I’m $300 richer—but, what if the system crashed in the middle, after the withdrawal but before the deposit? When the system comes back up, we’d see that the $300 withdrawal has been lost to the whims of the application. Transactions represent one possible solution to this problem. They run on the database server and make sure that a set of operations either all succeed or all fail. If our current example used transactions, the failure of either operation would cause both to be rolled back, leaving both accounts as they were before the ill-fated transfer attempt.

Clearly, then, transactions can be useful. It turns out, though, that when you use transactions in your code in Rails, the tests don’t always show what you’d expect them to. The problem is that, by default, all tests in Rails are already using transactions behind the scenes (to make loading and unloading test data fixtures as fast as possible). There’s a setting in the TestHelper class that controls this: use_transactional_fixtures. You could just set that to false to fix your transaction tests, but then you’d slow down the entire test suite significantly. A better alternative is to change the setting for the cases that hold transaction tests:


class TransferTest
self.use_transaction_fixtures = false
...
end


If you have transactional and non-transactional tests in a single case, it can make sense to control this setting at an even more granular level, by separating those tests in different test cases:


class TransferTest
...
end
class TransactionalTransferTest
self.use_transaction_fixtures = false
...
end

Happy testing!

Kevin McFadden, Former Staffer, October 23, 2006 0

I spent Thursday and Friday (Oct 19-20) at the DC PHP Conference with my co-director Patrick Reagan.  Overall, the conference was well done, though some of the talks lacked technical depth, possibly because the intended audience was a grab bag of developers and managers from the public and private sector.

For most of the conference, Patrick and I attended separate sessions.  He should be
summarizing his soon; but, for now, here is my take on the sessions that impacted me the most.

A nice surprise was how cool and down-to-earth Rasmus Lerdorf is in person.  His talk was entitled “Get Rich with PHP”; but, the main focus was demonstrating how to optimize your PHP scripts.  He has given this talk before, so I’ll just point you to a good summary.  Forget about premature optimization, the tricks here are simple enough to use for every page, assuming you have access to the tools.

Another pleasant surprise was Eli White’s excellent talk about “High Volume PHP and MySQL Techniques” validating what I already knew about scaling PHP and MySQL.  You can find his slideshow on his website in OpenOffice format.

Mike Ho’s talk was entitled “WebServices Integrating with other Agencies”; but, it was really a demonstration of how easy it was to create and integrate SOAP web services using his Qcodo framework and an MS Visual .Net application. Qcodo is still in beta, but looks very promising, especially after this week’s beta 3 release.  Qcodo will automatically generate the WSDL file, and Visual .Net leverages it to make hooking into service a snap.

Chris Shiflett spoke twice about PHP Security Testing, specifically about XSS and CSRF.  Both talks were very good and provided very useful and eye-opening information about the dangers of unfiltered user input.  Rasmus references PECL::Filter as a good tool to help prevent these problems.

This was my first conference in quite a while, and both the conference and the people were great.  Congratulations to the DC PHP Developers’ Group!

Patrick Reagan, Development Director, October 06, 2006 0

When it comes to the topic of messaging in a web application, clients will often ask if they can send targeted emails to their site’s users with their preferred email client (usually Microsoft Outlook). This question starts to hint at the real functionality that a client wants.  From their perspective, they need a solution that:

1. Aligns with their current skill sets when it comes to messaging (e.g., using Outlook as the preferred email application).


2. Allows them to deliver customized messages to every subset of their user base.


3. Provides dynamic capabilities so that each time a new user is added, he or she will start receiving the appropriate messages.

Let’s look at the pros and cons of the traditional ways developers have solved these problems.

Approach #1: Email Export to Outlook

Most of the time, web applications simply store and display data.  A typical reporting feature in these applications is the ability to generate an email list based on certain criteria.  Once you have access to this data in the format that you want, you can use this in the “Bcc” field in your desktop email client.

This satisfies all three conditions above; but, there are a few key areas where this falls apart:

• Instead of easily firing off an email from their address book, clients are now forced to log into the web application, run a report, and grab the contact list – not a simple proposition.


• If the list of emails is large enough, the risk of exceeding the maximum number of recipients increases – not something we want our clients to have to suffer through.


• Since this is a manual process, any human error introduces the possiblity of exposing our distribution list, compromising the privacy of our user base.

If we want to reduce the possibility for human error, we can rely on our application to handle the delivery of messages.

Approach #2: Custom Messaging

Assuming our application has the reporting mechanism discussed in the first approach, it’s quite simple to tie in some basic messaging functionality.  Once the user has generated the email list, we can provide a simple interface to customize the content of the message. 

While this removes the hassle of switching back-and-forth between applications, we are still faced with some issues:

• Any advanced features (e.g., attachments, alternate content) that the client is familiar with in their existing email application will have to be custom-built into our new web-based system.


• Though not necessarily difficult (depending on the implementation), the client now has to re-learn how to send a message from the new web application.

If we can both leverage the client’s existing email skills and provide some dynamic capabilities, we will have a solution that addresses all three objectives.

Approach #3: The Email Reflector

When I attended the Flickr Workshop in NYC early this year, Cal discussed how Flickr gives users the ability to email photos from their existing mail clients into their Flickr account.  This got me thinking about what other features were possible with this approach.

As it turns out, this gives us the ability to send email to a dynamic list of recipients using a single pre-configured email address.

The Technical Details

At a high level, there are only three things necessary to make this work:

1. Set up the MTA to push raw email content to a script.


2. Gather a list of email addresses.


3. Redirect the raw message to the list of recipients.

Let’s look at these steps in more detail.  The examples provided assume that you’re working with my sample code (provided at the end of the post).

Step #1: Set Up the MTA

In my testing, I used the installation of Qmail that we had installed on one of our Gentoo development servers.  This will work just as well with a Sendmail installation – in fact, Cal discussed using Sendmail in his example.  We basically just need to configure the email for a single user to redirect to a script.

For Qmail or Sendmail, you only need to add a single line to the .qmail or .forward file, respectively (make sure the script is at least executable by the user):

|/path/to/demo/reflector.php

This handles the messages as they come in; but, we still need to decide who will ultimately receive the message.

Step #2: Gather the Email Addresses

This is pretty simple: we just need to get a list of addresses based on certain reporting criteria. Once we have this list, we can send the original message on its way, stripping the original “To” address and any unnecessary “Received” headers before we send.

Step #3: Send the Message

Using the provided EmailReflector class, we can push off all the heavy-lifting to it with just a few lines of code:

$reflector = new EmailReflector(’smtp-host’);
$reflector->setInputSource(’php://stdin’);
$reflector->redirect($recipientList);


The message is now Bcc’d to the specified list of recipients with all the original contents (including attachments) in place.

Enhancements

Because this is still “proof-of-concept” code, there are some issues that still need to be addressed:

• Security - If someone can guess the target email address for the reflector, he or she has the ability to spam your site’s users.  By making the target email harder to guess (good) or having the sender provide a security token (better), we can minimize this risk.


• Batching - As a site’s user base grows, the number of emails that will be sent through this sytem increases.  By providing some basic batching features, we can handle a larger volume.


• Catch-all addresses - A reporting feature that I see as useful would give users the ability to generate a report and save it with an associated email address.  Any time a message is sent to this address, it will regenerate the report and send the email.  Setting up a catch-all email address for a subdomain would make this possible.

The Code

Download the source (.tar.gz format)
Download the source (.zip format)