Double Shot #1894

Active Storage Samples

Active Storage is still moving along at a fast clip, so I figured it would be good to update my test code. You can find all the examples here in my repo at https://github.com/ffmike/activestorage_sample

A Note On Code Stability

It isn't. Active Storage has been merged to the main Rails project now, but it's not yet merged to master. Expect further changes and improvements, as well as potentially broken things.

That said, I'll try to keep this article & the associated samples up-to-date. No promises though.

Change History

  • 8 August 2017

    • Switch to master branch of Rails
    • Update name of Azure service
  • 2 August 2017

    • Switch to Rails with merged Active Storage, remove separate Active Storage gem.
    • Switched to using `url_for(document.url)` in show. See https://github.com/rails/activestorage/commit/d0e90b4a9dc1accd4f1044fde0dd9a347cd0afcf and future plans at https://github.com/rails/activestorage/issues/77 .
    • Added Microsoft Azure sample.
    • Added direct upload sample.
  • 24 July 2017

    • Updated database schema. If you used an earlier version of the samples, you'll need to drop & recreate the database.
    • Replaced .url with .service_url.

A New Edge Rails Application

Start by spinning up a simple application with edge rails:

1. Create a new directory and use whatever you like (rvm, rbenv, chruby) to give it a fresh Ruby environment. I used Ruby 2.4.1 in my testing.

2. Install bundler:

  gem install bundler

3. Create a simple Gemfile in your new directory:

  source 'https://rubygems.org'
  gem 'rails', git: 'https://github.com/rails/rails.git'

4. Install Rails and everything it drags in:

  bundle install

5. Create a new Rails application in the current directory, using the version of Rails you just installed and overwriting the Gemfile:

  bundle exec rails new . --dev --force

6. Create and start the simplest possible application:

  bundle exec rails g scaffold user name:string
  bundle exec rails db:create
  bundle exec rails db:migrate
  bundle exec rails s

You should now be able to go to http://localhost:3000/users to create and update users. Yay!

Active Storage with Local File Storage

Let's start by making sure that Active Storage is working without getting the cloud involved:

1. Create a new branch of code:

  git checkout -b local

2. Install Active Storage to your application:

bundle exec rails activestorage:install

This will create the storage and tmp/storage directories in your application, copy a default configuration file to config/storage.yml, and create a new migration. The migration builds the active_storage_blobs and active_storage_attachments tables in your database.

3. Update the database schema:

bundle exec rails db:migrate

4. Set up the development environment to use local storage by adding a line to your development.rb file:

config.active_storage.service = :local

5. Tell your User model that it has some attached files:

class User < ApplicationRecord
  has_one_attached :avatar
  has_many_attached :documents
  ...
end

6. Comment out the amazon, google, microsoft, and mirror sections from the config/storage.yml file. Otherwise, your server won't start, because it will be looking for keys and files that don't exist.

7. Add input fields to app/views/users/_form.html.erb:

  <div class="field">
    <%= form.label :avatar %>
    <%= form.file_field :avatar %>
  </div>

  <div class="field">
    <%= form.label :documents %>
    <%= form.file_field :documents, multiple: true %>
  </div>

8. Add controls to app/views/users/show.html.erb to display the data

<p>
  <%= image_tag(url_for(@user.avatar)) %>
<p>

<p>
  <strong>Documents:</strong>
  <ul>
    <% @user.documents.each do |document| %>
      <li><%= link_to document.blob.filename, url_for(document) %></li>
    <% end %>
  </ul>
<p>

9. Update your users controller to attach the files:

  # POST /users
  # POST /users.json
  def create
    @user = User.new(user_params)
    avatar = params[:user][:avatar]
    documents = params[:user][:documents]

    respond_to do |format|
      if @user.save
        if avatar
          @user.avatar.attach(avatar)
        end
        if documents
          @user.documents.attach(documents)
        end
        format.html { redirect_to @user, notice: 'User was successfully created.' }
        format.json { render :show, status: :created, location: @user }
      else
        format.html { render :new }
        format.json { render json: @user.errors, status: :unprocessable_entity }
      end
    end
  end

  # PATCH/PUT /users/1
  # PATCH/PUT /users/1.json
  def update
    avatar = params[:user][:avatar]
    documents = params[:user][:documents]

    respond_to do |format|
      if @user.update(user_params)
        if avatar
          @user.avatar.attach(avatar)
        end
        if documents
          @user.documents.attach(documents)
        end
        format.html { redirect_to @user, notice: 'User was successfully updated.' }
        format.json { render :show, status: :ok, location: @user }
      else
        format.html { render :edit }
        format.json { render json: @user.errors, status: :unprocessable_entity }
      end
    end
  end

10. Restart your application. You should now be able to add avatars and documents to users, and retrieve them via the show view.

Active Storage with Amazon Web Services Storage

Moving along, here's how to move your Active Storage files over to Amazon S3:

1. Create a new branch of code, starting from the local branch so that you already have the basics:

  git checkout -b aws

2. Sign in to your AWS account and go to your S3 management console (the URL will be something like https://console.aws.amazon.com/s3/home?region=us-west-2# depending on your region).

3. Create a new bucket (for this tutorial, I'll use gstroop-production). Grant public read access to the bucket.

4. Retrieve the Access Key ID and Secret Access Key for your AWS account. Better yet, create a new pair just for this application.

5. Update your config/storage.yml file with a configuration stanza for Amazon:

amazon:
  service: S3
  access_key_id: ****************
  secret_access_key: *********************************************
  region: us-west-2
  bucket: gstroop-production

NOTE: Obviously, you need to keep those keys confidential. Don't check them into a public repository, for example. You can use the Rails secrets file to store them, or whatever other mechanism you prefer for production secrets.

6. Update your development.rb file to use the Amazon storage:

config.active_storage.service = :amazon

7. Add the AWS SDK gem to your Gemfile:

gem 'aws-sdk'

8. Install the gem:

bundle install

9. Restart the application. You should now be able to add user avatars and documents, and have then stored in the S3 bucket that you configured.

Active Storage with Google Cloud Platform Storage

You can also store your Active Storage files on Google Cloud Platform:

1. Create a new branch of code, starting from the local branch so that you already have the basics:

  git checkout -b gcs

2. Sign in to your Google Cloud Platform account and go to your console (the URL will be something like https://console.cloud.google.com/home/dashboard).

3. You need to create a new project, and then a storage bucket inside of the project. Record the names for both so you can add them to your storage.yml file.

4. You'll need to use GCS's API Manager to create credentials for your new bucket. Create a set of credentials that use the Google Cloud Datastore API. Download your credentials as JSON and store a copy of the file in your project at config/gcs.json.

NOTE: Obviously, you need to keep the keys in this file confidential. Don't check them into a public repository, for example. Manage it the same way you manage your database.yml or other files containing confidential information.

5. Update your storage.yml file with a configuration stanza for Google:

google:
  service: GCS
  project: **********-******
  keyfile: <%= Rails.root.join("config/gcs.json") %>
  bucket: ****-*******-****

6. Update your development.rb file to use the Google storage:

config.active_storage.service = :google

7. Add the Google Cloud Storage gem to your Gemfile:

gem 'google-cloud-storage'

8. Install the gem:

bundle install

9. Restart the application. You should now be able to add user avatars and documents, and have then stored in the Google bucket that you configured.

Active Storage with Microsoft Azure

You can also store your Active Storage files on Microsoft Azure:

1. Create a new branch of code, starting from the local branch so that you already have the basics:

  git checkout -b azure

2. Sign in to your Microsoft Azure account and go to your dashboard (https://portal.azure.com).

3. You need to create a new storage account, and then a container inside of the project. Record the names for both so you can add them to your storage.yml file.

4. You'll also need to navigate to "Access Keys" in the Azure portal and record one of your storage account access keys.

NOTE: Obviously, you need to keep the keys in this file confidential. Don't check them into a public repository, for example. Manage it the same way you manage your database.yml or other files containing confidential information.

5. Update your storage.yml file with a configuration stanza for Azure:

microsoft:
  service: AzureStorage
  # Note that the path must NOT have a trailing slash
  path: https://**********.blob.core.windows.net
  storage_account_name: **********
  storage_access_key: ******************************************
  container: *************

NOTE: Obviously, you need to keep those keys confidential. Don't check them into a public repository, for example. You can use the Rails secrets file to store them, or whatever other mechanism you prefer for production secrets.

6. Update your development.rb file to use the Microsoft Azure storage:

config.active_storage.service = :microsoft

7. Add the Azure gem to your Gemfile:

gem 'azure-core'

8. Install the gem:

bundle install

9. Restart the application. You should now be able to add user avatars and documents, and have then stored in the Azure container that you configured.

NOTE: There is currently a bug preventing attached files being properly retrieved from Azure. See https://github.com/rails/rails/pull/30135

Active Storage Mirroring

1. Create a new branch of code, starting from the local branch so that you already have the basics:

  git checkout -b mirror

2. Create amazon and google stanzas in your config/storage.yml file, following the instructions given above.

3. Update your storage.yml file with a configuration stanza for mirroring:

mirror:
  service: Mirror
  primary: local
  mirrors: [ amazon, google ]

4. Update your development.rb file to use the mirrored storage:<

config.active_storage.service = :mirror

5. Add both the AWS and Google Cloud Storage gems to your Gemfile:

gem 'aws'
gem 'google-cloud-storage'

6. Install the gems:

bundle install

7. Restart the application. You should now be able to add user avatars and documents.

What does mirroring do? It gives you built-in redundancy against cloud service failures. In the case of the configuration above:

  • Files are uploaded to Amazon and Google, and stored locally.
  • Files are served from local storage.

Should AWS go down for an entire region, you'd only need to change the mirror configuration in your config file, restart your server, and you'd be up and running again.

Active Storage Variants

Active Storage also includes built-in support for applying arbitrary transforms to images with MiniMagick. To force all uploaded avatars to 128x128 pixels, follow these steps:

1. Create a new branch of code, starting from the local branch so that you already have the basics:

  git checkout -b variant

2. Update app/views/users/show.html.erb:

<p>
  <%= image_tag(url_for(@user.avatar.variant(resize: "128x128"))) %>
<p>

3. Restart the application. Regardless of the size of avatar you upload, it should be displayed at 128x128 pixels.

Active Storage uses a lazy strategy to create variants. No variants are created at upload time. Rather, the first time you try to access a variant it is created and stored, and then the URL gets it from the new storage location.

Direct Uploads

If you're using cloud storage, Active Storage includes a JavaScript library that can bypass your server entirely, uploading files directly from the browser to the cloud. To use direct upload with AWS, follow these steps:

1. Create a new branch of code, starting from the aws branch so that you already have the basics, including a working AWS configuration:

  git checkout -b direct_upload

2. Add the Active Storage JavaScript to your app/assets/javascripts/application.js file:

//= require activestorage

3. Modify app/views/users/_form.html.erb with the `direct_upload` option:

  <div class="field">
    <%= form.label :avatar %>
    <%= form.file_field :avatar, direct_upload: true %>
  </div>

  <div class="field">
    <%= form.label :documents %>
    <%= form.file_field :documents, multiple: true, direct_upload: true %>
  </div>

4. Restart the application. You should be able to add user avatars and documents.

5. Update the CORS configuration on your Amazon S3 bucket to allow incoming requests:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

NOTE: This is a super-open CORS policy. In a production application, you'll want to lock things down.

NOTE: At the moment this is working for me in Chrome but not in Firefox. There is some sort of request signing issue that I haven't sussed out. Advice welcome.

Double Shot #1893

  • Automating Rails Security with Brakeman Pro - I've seen too many build processes that didn't consider security at all. Here's one way to fix that.
  • Octobox - Management console for GitHub notifications. There's a hosted versionl, or you can self-host and get a bit more functionality.
  • Evaluating persistent, replicated message queues - A comparison of many entrants in the message queuing field, with hard numbers and a decent discussion of methodology and results. (But first: do you really need a message queue?)
  • Spreadsheet Architect - Gem to create spreadsheets from Active Record relations, POROs, or just data in Ruby.

Double Shot #1892

Double Shot #1891

  • GraphQL Ruby - Getting more popular by leaps and bounds. I guess it's time to figure out what it brings to the API table. Besides more syntax.
  • On being a remote...manager - Lots of good advice here. I've been working remote for decades, and it's really only in the past few years that being a remote part of the team has really been posible.
  • A Week with a Rails Security Strategy - Building some habits (and checking some common sore spots) to improve the security of your application. Like most devs, I could tell horror stories.
  • OmniPascal 0.14.0 - Mac and Linux Support - Brings back memories. I wonder if I still have my copy of SOFTWARE TOOLS IN PASCAL around here somewhere.
  • Skylight - "Smart profiler for Ruby on Rails applications", a performance monitoring solution with stack-specific insights.
  • Splitting APIs, Servers, and Implementations in Elixir - Dave Thomas noodles around with architecture. It's still early days for Elixir, so conventions aren't frozen yet.
  • Alibaba Cloud - Worth a look if you're chasing mainland China business, simply because they'll help get you through the licensing. Otherwise I'd be scared to touch it.
  • Crystal [ANN] - If you're interested in the Crystal language, this looks like the blog to follow.
  • Introducing Moon: A blazing fast UI library - Another entrant in the JavaScript front-end race, with an API similar to Vue but a focus on speed.
  • Effective Pivotal Tracker: Tips and Tricks - A few ways to make your PT use more useful.
  • Scout Roadmap - The folks behind the Scout monitoring service are experimenting with sharing their roadmap publicly, using issues in a GitHub repo to track future work.

Double Shot #1890

Double Shot #1889

Double Shot #1888

  • Devflow - Independently-provisioned continuous delivery test URLs for every branch in your source code control. Currently in beta for PHP hosted at GitHub, but with ambitious plans for the future.
  • How to GraphQL - "The Fullstack Tutorial for GraphQL". Lots of technology stuff here that I should probably learn as long as I'm on the bench.
  • Seashells - Experimental service to pipe output from a command-line program direct to a temporary URL on the web.
  • Two-factor authentication is a mess - Yes, yes it is. There are some recommendations here along with a look at how we got into this mess.
  • Choosing a frontend framework in 2017 - A discussion of Ember, Angular, React, and Polymer, set against the history of the past 10 years of JavaScript development.
  • Makara - Generic master/slave proxy that handles things like choosing and blacklisting connections. Comes with an ActiveRecord database adapter.
  • Sidekiq::Logstash - Turn your Sidekiq log into JSON syntax, ready to go to a logstash server.
  • Elixir v1.5.0-rc.1 released - I lack the expertise to pinpoint the significant changes here, but with that version number I'm sure there are some.
  • DogWatch - Ruby DSL for creating DataDog monitors.

Double Shot #1887

  • System Tester - Tool that works with a Chrome extension to automate the production of system tests for Rails 5.1+. It's nice to see Rails testing start to move beyond just writing everything by hand.
  • Double-check Bitcoin addresses when pasting - Analysis of another charming bit of malware. Always new ways to have your money stolen.
  • Serverless Stack - One of the better tutorials I've seen, this one covers hooking up Serverless and React and AWS Lambda. Also has lots of little real-world touches that make it more than just a toy.
  • Take the Journey: Build Your First Serverless Web Application - Amazon has their own tutorial out on the subject as well.
  • Tech companies: these are the perks (and benefits) I want. - Tara Hackley nails it. I would LOVE to see more salary transparency and fewer kegs. If your company offers these things, I'm still looking.
  • Maintainer.io - Open source project maintanence as a service. They promise to make your project work more smoothly, help triage issues, and defuse conflicts.
  • Kap - Free & open source screen recorder built on top of WebRTC, macOS only. Hat tip to Syntax for this one.

Double Shot #1886

Playing the LinkedIn game these days, just in case anyone feels like adding more connections.

Double Shot #1885

  • smith - Command line utility from Oracle designed to build containers with only a single process & its dependencies, reading and writing from standard locations.
  • Announcing Graylog v2.3.0-rc.1 - Notably with Elasticsearch 5 compatibility.
  • Syntax. - New podcast from Wes Bos and Scott Tolinski, kicking off with an episode about React tools.
  • Exclude Some Stylesheets in Rails 4 - Rails helpfully provides different stylesheets for different controllers - and then insists on concatenating them all together. Here are some thoughts on a more modular setup.
  • Redux isn't slow, you're just doing it wrong - An optimization guide - Some fairly detailed guidance, much of it revolving around how to avoid unnecessary rendering.
  • GitNotifier - Get email notifications when someone stars or forks one of your repos on GitHub, or when they follow or unfollow you.
  • Bitcoin, Ethereum, Blockchain, Tokens, ICOs: Why should anyone care? - Despite the title, this is actually a clear explanation of a lot of the basic cryptocurrency concepts and ideas floating around. It is not however in the least skeptical about them.
  • Ship - Native macOS client for GitHub, focused on code review and issue tracking.
  • Ruby on Rails Code Audits: 8 Steps to Review Your App - Quick ways to judge the quality of a codebase that you just stepped in. I'd add that you should look at routes.rb early on; if the routing is craptacular the rest of the app won't be any better.
  • Wildcard Certificates Coming January 2018 - From Let's Encrypt, that is. Free of charge from one of their automated endpoints.

Double Shot #1884

  • Why Serverless? - An introduction to the idea behind things like OpenWhisk, Amazon Lambda, and Google Cloud Functions. I need to dig in a bit here.
  • ES2017's async/await is the best thing to ever happen to JavaScript - There are days when I feel like I'll never catch up with innovation in JavaScript. Here's the latest way to handle async code.
  • Elvish - Another option for a shell on Linux, macOS, or BSD. It features nicer control structures and pipelines that can handle more than plain text.
  • Project Guidelines - For JavaScript projects, from Hive. I'm not always happy with formal guidelines, but I welcome thinking about the tradeoffs. Lots of good stuff here.
  • Running feature specs with Capybara and Chrome Headless - How to set up the latest refinements in Rails system tests.
  • Suicide Linux - The distro equivalent of "do you feel lucky, punk?" Now available as a Docker image.
  • The future of querying json in PostgreSQL - Looks yummy, though certainly not the SQL I learned lo these many years ago. SELECT * FROM my_table WHERE JSON_EXISTS(ranges, '$[*] ? (@.min < $x && $x <= @.max)' PASSING number AS x);
  • Announcing Gatsby 1.0.0 - A major milestone for this React-based static site generator. Lots of spiffy advanced features are built in, too.

Double Shot #1883

Introduction to Active Storage

Today DHH announced Active Storage: a new gem that makes it easy to attach Amazon S3 or Google Cloud Storage files to Active Record model instances. Looks like this is slated to become part of Rails in 5.2. I happen to have a Rails 5.2 application with a user model around, so here's a quick look at adding an avatar to that model with Active Storage.

NOTE: This is early days for this library, so don't count on this post to be perfectly accurate if you're reading it months from now.

1. Add the gem to your Gemfile and run bundle install

gem 'activestorage', github: 'rails/activestorage'

2. Add require "active_storage" to config/application.rb.

3. Run rails activestorage:install. This will create the storage and tmp/storage directories in your application, copy a default configuration file to config/storage_services.yml, and create a new migration. The migration will build the active_storage_blobs and active_storage_attachments tables in your database.

4. Run rails db:migrate

NOTE: I had to change the class name in the migration from ActiveStorage::CreateTables to ActiveStorageCreateTables to get it to run without errors.

5. Add config.active_storage.service = :local to your development.rb file. This will use the preconfigured local-file storage when you're in the development environment.

6. Tell your User model that it has an attached file:

class User < ApplicationRecord
  has_one_attached :avatar
  ...
end

7. Make a copy of your config/storage_services.yml file to config/storage_services_backup.yml. Then delete the amazon, google, and mirror sections from the original file. Otherwise, your server won't start, because it will be looking for keys and files that don't exist.

8. Add a field to your user edit form to input an avatar file:

<%= f.file_field :avatar, class: "form-control" %>

9. Add a control to your user show view to display the avatar:

<%= image_tag(@user.avatar.url) %>

10. Update your users controller to attach the file:

  def update
    @user = User.find(params[:id])
    avatar = params[:user][:avatar]
    if @user.update(user_params)
      if avatar
        @user.avatar.attach(avatar)
      end
      redirect_to @user, notice: "Your profile has been updated"
    else
      render 'edit'
    end
  end

11. Restart your application and navigate to /users/<id>edit to edit an existing user. Select a file and save the change. You should see the avatar image displayed on the edited user.

Of course, in production you'll want to use an actual cloud service. If you check your backed-up configuration file you'll see how to set up the keys for either S3 or AWS. Then it should just be a matter of setting the appropriate configuration in production.rb (note that I haven't tried this yet!)

Introduction to Active Storage, Part 4: Mirroring

If you've been following along:

Then you're ready for one of the cool features of Active Storage: mirroring. This one is really easy to set up:

1. Ensure that you have created amazon, local, and google sections in your storage_services.yml file, and that you have tested both of them.

2. Update your storage_services.yml file with a configuration stanza for mirroring:

mirror:
  service: Mirror
  services: [ amazon, local, google ]

3. Edit your production.rb file:

config.active_storage.service = :mirror

4. Deploy the application. You should still be able to add user avatars.

What does mirroring do? It gives you built-in redundancy against cloud service failures. In the case of the configuration above:

  • Files are uploaded to Amazon and Google, and stored locally.
  • Files are served from Amazon (the first member of the services array)

Should AWS go down for an entire region, you'd only need to change the services array in your config file, restart your server, and you'd be up and running again.

BUT: Note the "should" in step 4. As I write this, this doesn't actually work with the code in the Active Storage repo, because the MirrorService doesn't get configured properly. See issue 9, which includes a quick and hacky patch for the Active Storage internals to get things working.

Introduction to Active Storage Part 3: Google Cloud Storage

And to complete the trifecta, here's how you can update the Active Storage code to use Google Cloud Storage:

1. Sign in to your Google Cloud Platform account and go to your console (the URL will be something like https://console.cloud.google.com/home/dashboard).

2. You need to create a new project, and then a storage bucket inside of the project. Record the names for both so you can add them to your storage_services.yml file.

3. You'll need to use GCS's API Manager to create credentials for your new bucket. Create a set of credentials that use the Google Cloud Datastore API. Download your credentials as JSON and store a copy of the file in your project at config/gcs.json.

NOTE: Obviously, you need to keep the keys in this file confidential. Don't check them into a public repository, for example. Manage it the same way you manage your database.yml or other files containing confidential information.

4. Update your storage_services.yml file with a configuration stanza for Google:

google:
  service: GCS
  project: gstroop-******
  keyfile:
  bucket: ****-*******-9999 

5. Add a line to your production.rb file:

config.active_storage.service = :google

6. Add the google-cloud-storage gem to your Gemfile and run bundle install.

7. Deploy the application. You should now be able to add user avatars and have then stored in the GCS bucket that you configured.

Introduction to Active Storage Part 2: Amazon S3

Moving along, here's how to move your Active Storage files over to Amazon S3:

1. Sign in to your AWS account and go to your S3 management console (the URL will be something like https://console.aws.amazon.com/s3/home?region=us-west-2# depending on your region).

2. Create a new bucket (for this tutorial, I'll use gstroop-production). Grant public read access to the bucket.

3. Retrieve the Access Key ID and Secret Access Key for your AWS account. Better yet, create a new pair just for this application.

4. Update your storage_services.yml file with a configuration stanza for Amazon:

amazon:
  service: S3
  access_key_id: AKIA************
  secret_access_key: *********************************************
  region: us-west-2
  bucket: gstroop-production

NOTE: Obviously, you need to keep those keys confidential. Don't check them into a public repository, for example. You can use the Rails secrets file to store them, or whatever other mechanism you prefer for production secrets.

5. Add a line to your production.rb file:

config.active_storage.service = :amazon

6. Add the aws-sdk gem to your Gemfile and run bundle install.

7. Deploy the application. You should now be able to add user avatars and have then stored in the S3 bucket that you configured.

Double Shot #1882

Double Shot #1881

Double Shot #1880

subscribe via RSS