Skip to main content

Make Remote Files Local with Ruby Tempfile

Lawson Kurtz|

We live in the age of remote resources. It’s pretty rare to store uploaded files on the same machine as your server process. File storage these days is almost completely remote, and for very good reasons.

Using file storage services like S3 is awesome, but not having your files accessible locally can complicate the performance of file-oriented operations. In these cases, use Ruby’s Tempfile to create local files that live just long enough to satisfy your processing needs.

Anatomy of Tempfile#new

# What you want the tempfile filename to <span class="operator"><span class="keyword">start</span> <span class="keyword">with</span>
name_start = <span class="string">'my_special_file'</span>

# What you want the tempfile filename <span class="keyword">to</span> <span class="keyword">end</span> <span class="keyword">with</span> (probably including an extension)
# <span class="keyword">by</span> <span class="keyword">default</span>, tempfilenames carry <span class="keyword">no</span> extension
name_end = <span class="string">'.gif'</span>

# <span class="keyword">Where</span> you want the tempfile <span class="keyword">to</span> live
location = <span class="string">'/path/to/some/dir'</span>

# Options <span class="keyword">for</span> the tempfile (e.g. which encoding <span class="keyword">to</span> use)
options = { encoding: Encoding::UTF_8 }

# Will <span class="keyword">create</span> a tempfile
# <span class="keyword">at</span> /path/<span class="keyword">to</span>/<span class="keyword">some</span>/dir/my_special_file_20140224-<span class="number">1234</span>-abcd123.gif
# (<span class="keyword">where</span> <span class="string">'20140224-1234-abcd123'</span> represents <span class="keyword">some</span> <span class="keyword">unique</span> <span class="keyword">timestamp</span> & token)
# <span class="keyword">with</span> a UTF-<span class="number">8</span> encoding
# <span class="keyword">with</span> the contents <span class="string">'Hello, tempfile!'</span>
Tempfile.new([name_start, name_end], location, options) <span class="keyword">do</span> |file|
 file.<span class="keyword">write</span>(<span class="string">'Hello, tempfile!'</span>)
<span class="keyword">end</span></span>

Example Application

URL to Tempfile: Remote File Processing

We have a service that takes a URL and processes the file it represents using a Java command-line utility. Our command-line utility expects a filepath argument, so we must create a local file from the remote resource before processing.

<span class="class"><span class="keyword">class</span> <span class="title">LocalResource</span></span>
 attr_reader <span class="symbol">:uri</span>

 <span class="function"><span class="keyword">def</span> <span class="title">initialize</span><span class="params">(uri)</span></span>
 <span class="variable">@uri</span> = uri
 <span class="keyword">end</span>

 <span class="function"><span class="keyword">def</span> <span class="title">file</span></span>
 <span class="variable">@file</span> ||= <span class="constant">Tempfile</span>.new(tmp_filename, tmp_folder, <span class="symbol">encoding:</span> encoding).tap <span class="keyword">do</span> |f|
 io.rewind
 f.write(io.read)
 f.close
 <span class="keyword">end</span>
 <span class="keyword">end</span>

 <span class="function"><span class="keyword">def</span> <span class="title">io</span></span>
 <span class="variable">@io</span> ||= uri.open
 <span class="keyword">end</span>

 <span class="function"><span class="keyword">def</span> <span class="title">encoding</span></span>
 io.rewind
 io.read.encoding
 <span class="keyword">end</span>

 <span class="function"><span class="keyword">def</span> <span class="title">tmp_filename</span></span>
 [
 <span class="constant">Pathname</span>.new(uri.path).basename,
 <span class="constant">Pathname</span>.new(uri.path).extname
 ]
 <span class="keyword">end</span>

 <span class="function"><span class="keyword">def</span> <span class="title">tmp_folder</span></span>
 <span class="comment"># If we're using Rails:</span>
 <span class="constant">Rails</span>.root.join(<span class="string">'tmp'</span>)
 <span class="comment"># Otherwise:</span>
 <span class="comment"># '/wherever/you/want'</span>
 <span class="keyword">end</span>
<span class="keyword">end</span>

<span class="function"><span class="keyword">def</span> <span class="title">local_resource_from_url</span><span class="params">(url)</span></span>
 <span class="constant">LocalResource</span>.new(<span class="constant">URI</span>.parse(url))
<span class="keyword">end</span>

<span class="comment"># URL is provided as input</span>
url = <span class="string">'https://s3.amazonaws.com/your-bucket/file.gif'</span>

<span class="keyword">begin</span>
 <span class="comment"># We create a local representation of the remote resource</span>
 local_resource = local_resource_from_url(url)

 <span class="comment"># We have a copy of the remote file for processing</span>
 local_copy_of_remote_file = local_resource.file

 <span class="comment"># Do your processing with the local file</span>
 `some-command-line-utility <span class="comment">#{local_copy_of_remote_file.path}`</span>
<span class="keyword">ensure</span>
 <span class="comment"># It's good idea to explicitly close your tempfiles</span>
 local_copy_of_remote_file.close
 local_copy_of_remote_file.unlink
<span class="keyword">end</span>

Tempfiles vs Files

Ruby Tempfile objects act almost identically to regular File objects, but have a couple of advantages for transient processing or uploading tasks:

  • Tempfiles’ filenames are unique, so you can put them in a shared tmp directory without worrying about name collision.
  • Tempfiles’ files are deleted when the Tempfile object is garbage collected. This prevents a bunch of extra files from accidentally accumulating on your machine. (But you should of course still explicity close Tempfiles after working with them.)

Common Snags

 

Rewind

Certain IO operations (like reading contents to determine an encoding) move the file pointer away from the start of the IO object. In these cases, you will run into trouble when you attempt to perform subsequent operations (like reading the contents to write to a tempfile). Move the pointer back to the beginning of the IO object using #rewind.

io_object = StringIO.<span class="keyword">new</span>(<span class="string">"I'm an IO!"</span>)
encoding = io_object.read.encoding

<span class="preprocessor"># The pointer is now at the end of 'io_object'.</span>
<span class="preprocessor"># When we read it again, the return is an empty string.</span>
io_object.read
<span class="preprocessor"># => ""</span>

<span class="preprocessor"># But if we rewind first, we can then read the contents.</span>
io_object.rewind
io_object.read
<span class="preprocessor"># => "I'm an IO!"</span>
 

Encoding

Often you’ll need to ensure the proper encoding of your tempfiles. You can provide your desired encoding during Tempfile initialization as demonstrated below.

encoding = Encoding::UTF_8

Tempfile.new('<span class="keyword">some</span>-filename', '/<span class="keyword">some</span>/tmp/dir', encoding: encoding).tap do |<span class="type">file</span>|
 <span class="comment"># Your code here...</span>
<span class="keyword">end</span>

Obviously your desired encoding won’t always be the same for every file. You can find your desired encoding on the fly by sending #encoding to your file contents string. Or if you’re using an IO-object, you can call io.object.read.encoding.

encoding = file_contents_string.encoding
<span class="comment"># or</span>
<span class="comment"># encoding = io_object.read.encoding</span>

Tempfile.new('<span class="keyword">some</span>-filename', '/<span class="keyword">some</span>/tmp/dir', encoding: encoding).tap do |<span class="type">file</span>|
 <span class="comment"># Your code here...</span>
<span class="keyword">end</span>

Read more about Ruby encoding.

 

Extensions

By default, files created with Tempfile.new will not carry an extension. This can pose problems for applications or tools (like Carrierwave and soffice) that rely on a file’s extension to perform their operations.

In these cases, you can pass an extension to the Tempfile initialization as demonstrated above in Anatomy of Tempfile#new.

<span class="preprocessor"># A quick refresher</span>
Tempfile.<span class="keyword">new</span>([<span class="string">'file_name_prefix'</span>, <span class="string">'.extension'</span>], <span class="string">'/tmp'</span>)

If you need to dynamically determine your file’s extension, you can usually grab it from the URL or file path you are reading into your Tempfile:

uri = URI.parse('https://example.com/<span class="keyword">some</span>/path/<span class="keyword">to</span>/<span class="type">file</span>.gif')
path = '/<span class="keyword">some</span>/path/<span class="keyword">to</span>/<span class="type">file</span>.gif'

Pathname.new(uri.path).extname
<span class="comment"># => '.gif'</span>

Pathname.new(path).extname
<span class="comment"># => '.gif'</span>

Local Development (Paths vs URLs)

Many developers use local file storage for their development environment. In these cases, local file paths often appear in methods that are expecting URLs. Not fun.

OpenURI to the Rescue

If you need to write code that supports reading files from both file paths and URLs, OpenURI is your saviour.

OpenURI is accessible via the Kernel function open, which provides a file-like API for both local and remote resources.

open('/path/<span class="keyword">to</span>/your/<span class="keyword">local</span>/<span class="type">file</span>.gif') do |<span class="type">file</span>|
 <span class="comment"># Your code here...</span>
<span class="keyword">end</span>
<span class="keyword">open</span>(<span class="attribute">'https</span>://s3.amazonaws.com/your-bucket/<span class="keyword">file</span>.gif') do |<span class="keyword">file</span>|
 # Your code here...
<span class="keyword">end</span>

We like Ruby Tempfiles for performing file-oriented operations on remote resources. What do you use?

Thanks to Ryan Foster for his contributions to the sample code.

← Drag →