Jekyll2024-02-20T08:30:00+00:00https://hansschnedlitz.com/feed.xmlHans SchnedlitzHans Schnedlitz is a Ruby on Rails Engineer from Vienna.Hans SchnedlitzUsing Jekyll with Esbuild2024-02-09T09:00:00+00:002024-02-09T09:00:00+00:00https://hansschnedlitz.com/2024/02/09/using-jekyll-with-esbuild<p>I know. What an unholy union. Why would anyone do this? Why would <em>anyone</em> want that?</p>
<p>Well, first for science. Obviously.</p>
<p>Second, believe it or not, there are <em>actual</em> good reasons for combining Esbuild with Jekyll. I like Jekyll. It’s mature, has many plugins, and a vibrant ecosystem. Also, it’s built on Ruby, and Ruby is fantastic. Most importantly, it’s simple.</p>
<p>But.</p>
<p>Sometimes, you want that extra bit of visual oomph. Sometimes, you want to do weird or cool things, and a bundler might be necessary under those circumstances. Esbuild is modern and efficient. If you’re working with Ruby on Rails, you might already be used to it. Coincidentally, it’s also simple. Relatively speaking.</p>
<p>If you look at it like that, maybe Jekyll and Esbuild are meant to work together after all?</p>
<p class="notice--info">Why not Hugo? Or Bridgetown? Or any other static site generator that’s not Jekyll? Look, I don’t know what else to say. For science. I like Jekyll. It works for me 🙃</p>
<h2 id="what-well-do">What we’ll do</h2>
<p>The idea is this. We want to use Esbuild for bundling JS and CSS and let Jekyll take care of the rest. We’ll also add some plugins to Esbuild (<a href="https://github.com/postcss/autoprefixer">autoprefixer</a>, <a href="https://www.npmjs.com/package/esbuild-sass-plugin">build-sass-plugin</a>, <a href="https://postcss.org/">postcss</a>). First, so we can keep using the SCSS that Jekyll already provides, second for demonstration purposes.</p>
<p>We’ll also make sure we end up with a pleasant developer experience. Jekyll and Esbuild might not want to play nice, but we’ll make them 😈</p>
<h2 id="getting-set-up">Getting set up</h2>
<p class="notice--info">Heads up! This guide assumes you have Ruby, Bundler, and Node set up on your machine. Also, Jekyll should already be installed. If that’s not the case, take care of that before you continue.</p>
<p>We’ll forego any plugins and start with an empty Jekyll scaffold to keep things simple.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jekyll new jekyll-esbuild --blank && cd jekyll-esbuild
</code></pre></div></div>
<p>Let’s create an empty Javascript file for demonstration purposes. Let’s also install Esbuild and the plugins we are going to use.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir assets/javascript && touch assets/javascript/main.js
npm i -D --save-exact esbuild
npm i -D autoprefixer esbuild-sass-plugin postcss
</code></pre></div></div>
<p>When we run <code class="language-plaintext highlighter-rouge">jekyll serve --watch</code> we should see <em>something</em> on <a href="http://localhost:4000">localhost:4000</a>.</p>
<p><img src="https://hansschnedlitz.com/assets/images/posts/2024-jekyll-esbuild/jekyll.webp" alt="A Jekyll site" class="align-center" /></p>
<h2 id="bundling-assets-with-esbuild">Bundling Assets with Esbuild</h2>
<p>To use Esbuild with our CSS plugins, using the command line won’t do. We’ll have to create a small build script.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// scripts/build.mjs</span>
<span class="k">import</span> <span class="nx">esbuild</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">esbuild</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">sassPlugin</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">esbuild-sass-plugin</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">postcss</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">postcss</span><span class="dl">"</span><span class="p">;</span>
<span class="k">import</span> <span class="nx">autoprefixer</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">autoprefixer</span><span class="dl">"</span><span class="p">;</span>
<span class="k">await</span> <span class="nx">esbuild</span><span class="p">.</span><span class="nx">build</span><span class="p">({</span>
<span class="na">entryPoints</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">assets/css/main.scss</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">assets/javascript/main.js</span><span class="dl">"</span><span class="p">],</span>
<span class="na">outdir</span><span class="p">:</span> <span class="dl">"</span><span class="s2">_site/assets</span><span class="dl">"</span><span class="p">,</span>
<span class="na">bundle</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">plugins</span><span class="p">:</span> <span class="p">[</span>
<span class="nx">sassPlugin</span><span class="p">({</span>
<span class="k">async</span> <span class="nx">transform</span><span class="p">(</span><span class="nx">source</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">css</span> <span class="p">}</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">postcss</span><span class="p">([</span><span class="nx">autoprefixer</span><span class="p">]).</span><span class="nx">process</span><span class="p">(</span><span class="nx">source</span><span class="p">,</span> <span class="p">{</span>
<span class="na">from</span><span class="p">:</span> <span class="kc">undefined</span><span class="p">,</span>
<span class="p">});</span>
<span class="k">return</span> <span class="nx">css</span><span class="p">;</span>
<span class="p">},</span>
<span class="na">loadPaths</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">_sass</span><span class="dl">"</span><span class="p">],</span>
<span class="p">}),</span>
<span class="p">],</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Once we update our <code class="language-plaintext highlighter-rouge">package.json,</code> we can build our assets.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"devDependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"scripts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"build"</span><span class="p">:</span><span class="w"> </span><span class="s2">"node scripts/build.mjs"</span><span class="p">,</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>If we were to run <code class="language-plaintext highlighter-rouge">npm run build</code> now, we’d get an error. Our <code class="language-plaintext highlighter-rouge">assets/main.scss</code> still contains YML markup that we need to remove.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--- <<< DELETE ME
--- <<<
@import "base";
</code></pre></div></div>
<p>After that, we should be able to build assets without any issues.</p>
<h3 id="configuring-jekyll">Configuring Jekyll</h3>
<p>Now, if you keep a close eye on the <code class="language-plaintext highlighter-rouge">_site</code> folder and make some changes to any watched files, you’ll notice that your built files will be changed. Makes sense because, as things stand, Jekyll still feels responsible for assets, thus overwriting or deleting the files created by Esbuild.</p>
<p>Let’s fix that. The simplest way to tell Jekyll to not worry about CSS and JS assets is to exclude the respective folders by updating <code class="language-plaintext highlighter-rouge">_config.yml</code>.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">exclude</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">_sass</span> <span class="c1"># Let build handle CSS</span>
<span class="pi">-</span> <span class="s">assets/css</span> <span class="c1"># Let build handle CSS</span>
<span class="pi">-</span> <span class="s">assets/javascript</span> <span class="c1"># Let esuild handle JS</span>
<span class="pi">-</span> <span class="s">scripts</span>
<span class="pi">-</span> <span class="s">package.json</span>
<span class="pi">-</span> <span class="s">package-lock.json</span>
<span class="na">keep_files</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">assets/css</span> <span class="c1"># Let build handle CSS</span>
<span class="pi">-</span> <span class="s">assets/javascript</span> <span class="c1"># Let esuild handle JS</span>
</code></pre></div></div>
<p>Notice that we also told Jekyll to not delete CSS and JS assets when rebuilding. We also excluded some additional files that our Jekyll site doesn’t need. See <a href="https://jekyllrb.com/docs/configuration/options/">configuration options</a> if you need more details.</p>
<p class="notice--warning">This won’t do if you’re using a theme. By excluding asset folders, theme styles won’t be processed appropriately. There are ways to fix that, but it’s a bit much for this blog post.</p>
<p>Let’s bundle and serve again to make sure everything is still working.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm run build
jekyll serve --watch
</code></pre></div></div>
<p>Changing anything (e.g. CSS) will no longer overwrite files created by Esbuild. Success! Unfortunately, our changes won’t be reflected on your site. Let’s change that by adding watch mode to Esbuild.</p>
<h3 id="improving-developer-experience">Improving Developer Experience</h3>
<p>Until the beginning of 2023, adding watch mode was a simple matter of adding <code class="language-plaintext highlighter-rouge">watch: true</code> to the arguments of Esbuild. Now, it’s a bit different. I opted to change the script behavior based on input arguments, but you can also create a separate script.</p>
<p>The updated <code class="language-plaintext highlighter-rouge">scripts/build.mjs</code> looks something like this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">args</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">argv</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">watch</span> <span class="o">=</span> <span class="nx">args</span><span class="p">.</span><span class="nx">includes</span><span class="p">(</span><span class="dl">"</span><span class="s2">--watch</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">context</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">esbuild</span><span class="p">.</span><span class="nx">context</span><span class="p">({</span>
<span class="c1">// ...</span>
<span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">watch</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">watch</span><span class="p">();</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">Watching!</span><span class="dl">"</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">rebuild</span><span class="p">();</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">dispose</span><span class="p">();</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="dl">"</span><span class="s2">Build done!</span><span class="dl">"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>After updating <code class="language-plaintext highlighter-rouge">package.json</code> once again, we can start watching our file changes by running <code class="language-plaintext highlighter-rouge">npm run watch</code>.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"devDependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"scripts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"watch"</span><span class="p">:</span><span class="w"> </span><span class="s2">"node scripts/build.mjs --watch"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>We’re already using Ruby anyway, so we might as well use <a href="https://github.com/ddollar/foreman">Foreman</a> to run everything we need with a simple command. Note that I also added <a href="https://browsersync.io/">browser-sync</a> for live reloading to the <code class="language-plaintext highlighter-rouge">Procfile</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jekyll: jekyll serve --watch
build: npm run watch
browser: browser-sync start --proxy localhost:4000 --files "**/*"
</code></pre></div></div>
<p>Run <code class="language-plaintext highlighter-rouge">foreman start</code> and your site should reload whenever you change your CSS, JS, or content. Mission accomplished!</p>Hans SchnedlitzI know. What an unholy union. Why would anyone do this? Why would anyone want that?Continuous Deployment with GitHub Actions and Kamal2024-01-07T15:00:00+00:002024-01-07T15:00:00+00:00https://hansschnedlitz.com/2024/01/07/continuous-deployment-with-github-actions-and-kamal<p><a href="https://kamal-deploy.org/">Kamal</a> is a wonderfully simple way to deploy your applications anywhere. It will also be <a href="https://github.com/rails/rails/issues/50441">included by default in Rails 8</a>. Kamal is trivial, but I don’t recommend using it on your development machine.</p>
<p>From experience working on an oldish laptop, I can tell you that building Docker images locally is not fun. Also, why would you, when GitHub Actions are for free!</p>
<p>In this post, I’ll show you how to build a simple CI pipeline with Kamal. We’ll create an application image and deploy it on every push. We’ll also add some simple image caching to speed up the workflow.</p>
<h3 id="the-complete-workflow">The Complete Workflow</h3>
<p>This is what we’ll end up with at the end of this post.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Deploy</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">push</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">main</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">Deploy</span><span class="pi">:</span>
<span class="na">if</span><span class="pi">:</span> <span class="s">${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">DOCKER_BUILDKIT</span><span class="pi">:</span> <span class="m">1</span>
<span class="na">RAILS_ENV</span><span class="pi">:</span> <span class="s">production</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout code</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v3</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Docker Buildx</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">buildx</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/setup-buildx-action@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Login to Docker Hub</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/login-action@v3</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">username</span><span class="pi">:</span> <span class="s">hschne</span>
<span class="na">password</span><span class="pi">:</span> <span class="s">${{ secrets.DOCKER_REGISTRY_KEY }}</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set Tag</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">tag</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">echo "tag=$(git rev-parse "$GITHUB_SHA")" >> $GITHUB_OUTPUT</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build image</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/build-push-action@v5</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">context</span><span class="pi">:</span> <span class="s">.</span>
<span class="na">builder</span><span class="pi">:</span> <span class="s">${{ steps.buildx.outputs.name }}</span>
<span class="na">push</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">labels</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">"service=anonymous-location"</span>
<span class="na">tags</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">"hschne/anonymous-location:latest"</span>
<span class="s">"hschne/anonymous-location:${{ steps.tag.outputs.tag }}"</span>
<span class="na">cache-from</span><span class="pi">:</span> <span class="s">type=gha</span>
<span class="na">cache-to</span><span class="pi">:</span> <span class="s">type=gha,mode=max</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">webfactory/ssh-agent@v0.7.0</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">ssh-private-key</span><span class="pi">:</span> <span class="s">${{ secrets.SSH_PRIVATE_KEY }}</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Ruby</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy command</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">bundle exec kamal deploy --skip-push</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">RAILS_MASTER_KEY</span><span class="pi">:</span> <span class="s">${{ secrets.RAILS_MASTER_KEY }}</span>
<span class="na">KAMAL_REGISTRY_PASSWORD</span><span class="pi">:</span> <span class="s">${{ secrets.DOCKER_REGISTRY_KEY }}</span>
</code></pre></div></div>
<p>That’s something. Well, nobody ever said GitHub actions are succinct. Let’s look at what’s going on here step by step.</p>
<h2 id="step-by-step">Step By Step</h2>
<p>The workflow above is based on <a href="https://dev.to/haukot/how-to-cache-mrsk-deployments-in-ci-52h9">the one in this post</a>. Unfortunately, that post is so old that it still refers to Kamal as Mrsk. I had to make some adjustments to how the image built and deployed.</p>
<p class="notice--info">I won’t go into details on GitHub Actions specifics. If you’re new to GitHub Actions or unfamiliar with one particular piece of syntax, I recommend you check out <a href="https://docs.github.com/en/actions">the documentation</a>.</p>
<p>To build and deploy our Rails application, we need to provide GitHub actions with three <a href="https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions">secrets</a>:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">RAILS_MASTER_KEY</code>: The master key to your Rails application credentials.</li>
<li><code class="language-plaintext highlighter-rouge">DOCKER_REGISTRY_KEY</code>: The API token for pushing and pulling from your container registry.</li>
<li><code class="language-plaintext highlighter-rouge">SSH_PRVIATE_KEY</code>: The SSH key for accessing the server where your app is deployed.</li>
</ul>
<p>We’ll also need to set some environment variables. We are building a production image. We’ll also need to instruct the Docker build step to use Docker Buildkit, as that is one of the requirements of Kamal.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">env</span><span class="pi">:</span>
<span class="na">DOCKER_BUILDKIT</span><span class="pi">:</span> <span class="m">1</span>
<span class="na">RAILS_ENV</span><span class="pi">:</span> <span class="s">production</span>
</code></pre></div></div>
<p>Our deployment workflow needs to do a couple of things. First, we need to check out the application source code. Next, we’ll log into Docker Hub to push our image.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout code</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v3</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Docker Buildx</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">buildx</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/setup-buildx-action@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Login to Docker Hub</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/login-action@v3</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">username</span><span class="pi">:</span> <span class="s">hschne</span>
<span class="na">password</span><span class="pi">:</span> <span class="s">${{ secrets.DOCKER_REGISTRY_KEY }}</span>
</code></pre></div></div>
<p>Kamal uses the git hash of the latest commit to determine which image to deploy, so image tags must match git commit hashes. We define this tag with a separate workflow step.</p>
<p>We use the <a href="https://github.com/docker/build-push-action">docker/build-push-action</a> to build the application image. In addition to setting the correct tag, the image build step must also provide a label matching your service name. Because the image should be pushed to your container registry, we set <code class="language-plaintext highlighter-rouge">push: true</code>, and because we want ludicrous build speed we instruct the build step to utilize the GitHub Actions cache.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set Tag</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">tag</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">echo "tag=$(git rev-parse "$GITHUB_SHA")" >> $GITHUB_OUTPUT</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build image</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">docker/build-push-action@v5</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">context</span><span class="pi">:</span> <span class="s">.</span>
<span class="na">builder</span><span class="pi">:</span> <span class="s">${{ steps.buildx.outputs.name }}</span>
<span class="na">push</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">labels</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">"service=service-name"</span>
<span class="na">tags</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">"user/image-name:latest"</span>
<span class="s">"user/image-name:${{ steps.tag.outputs.tag }}"</span>
<span class="na">cache-from</span><span class="pi">:</span> <span class="s">type=gha</span>
<span class="na">cache-to</span><span class="pi">:</span> <span class="s">type=gha,mode=max</span>
</code></pre></div></div>
<p>Once the image has been built and pushed, you only need to trigger the deployment using Kamal. We use the <a href="https://github.com/webfactory/ssh-agent">webfactory/ssh-agent</a> to establish a connection to our production server. After installing the required Ruby dependencies, it’s only a matter of running Kamal. As the image is already built and pushed, we use the <code class="language-plaintext highlighter-rouge">--skip-push</code> flag.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">webfactory/ssh-agent@v0.7.0</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">ssh-private-key</span><span class="pi">:</span> <span class="s">${{ secrets.SSH_PRIVATE_KEY }}</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Ruby</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy command</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">bundle exec kamal deploy --skip-push</span>
</code></pre></div></div>
<p>And that’s it! If you’ve enjoyed this post or have any other tips on how to use Kamal together with GitHub Actions, let me know! 🙂</p>Hans SchnedlitzKamal is a wonderfully simple way to deploy your applications anywhere. It will also be included by default in Rails 8. Kamal is trivial, but I don’t recommend using it on your development machine.The Simplest Static Site Generator2023-11-21T12:04:00+00:002023-11-21T12:04:00+00:00https://hansschnedlitz.com/2023/11/21/the-simplest-static-site-generator<p>Sometimes, you want to build a simple HTML page and populate it with some data. You may have some JSON lying around and want to make a simple website to visualize its contents. Or perhaps you want to show off <a href="https://www.hansschnedlitz.com/bookshelf/">how many books you read</a> in the last few years.</p>
<p>In any case, we’re talking about a small set of data and a <em>single</em> HTML page here. Would you use a static site generator for that? Which one? Jekyll? Hugo, Gatsby? Isn’t that overkill for what we’re trying to do?</p>
<p>I was pondering questions like this when I came across <a href="https://pandoc.org/MANUAL.html#templates">Pandoc Templates</a>. Turns out there is a way to generate static sites that is much simpler than most other options out there.</p>
<p class="notice--info">Alright, this might not be the absolutely simplest way to generate a static site. You got me. I bet someone, somewhere builds their static sites using a one-line Perl script. Jokes aside, let me know if you find an even simpler approach to building static sites.</p>
<h2 id="enter-the-template">Enter The Template</h2>
<p>A Pandoc template is not much to look at. It’s just an HTML file filled with some special syntax sprinkled in. Syntax that allows you to write <a href="https://pandoc.org/MANUAL.html#conditionals">conditionals</a> or <a href="https://pandoc.org/MANUAL.html#for-loops">loops</a> as well as do some other stuff. Let’s say we want to list authors and their respective books to simplify things. To do so, let’s create <code class="language-plaintext highlighter-rouge">template.html</code>:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><html</span> <span class="na">lang=</span><span class="s">"en"</span><span class="nt">></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">charset=</span><span class="s">"utf-8"</span> <span class="nt">/></span>
<span class="nt"><meta</span> <span class="na">name=</span><span class="s">"date"</span> <span class="na">content=</span><span class="s">"$date-meta$"</span> <span class="nt">/></span>
<span class="nt"><title></span>$title$<span class="nt"></title></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><section></span>
$for(authors)$
<span class="nt"><h2</span> <span class="na">class=</span><span class="s">"author"</span><span class="nt">></span>$authors.name$<span class="nt"></h2></span>
<span class="nt"><ul></span>
$for(authors.books)$
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"book"</span><span class="nt">></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"$authors.books.link$"</span><span class="nt">></span>$authors.books.name$<span class="nt"></a></span>
<span class="nt"></li></span>
$endfor$
<span class="nt"></ul></span>
$endfor$
<span class="nt"></section></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p>Now that we have a template, we must populate it with some data. I’ve found the most uncomplicated way to do so is to create a markdown file that contains the data as <a href="https://pandoc.org/chunkedhtml-demo/8.10-metadata-blocks.html#extension-yaml_metadata_block">metadata</a>. We keep our data in <code class="language-plaintext highlighter-rouge">authors.md</code>.</p>
<div class="language-md highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="na">title</span><span class="pi">:</span> <span class="s">Authors and their Books</span>
<span class="na">authors</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">James S.A. Corey</span>
<span class="na">books</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Leviathan Wakes</span>
<span class="na">link</span><span class="pi">:</span> <span class="s">https://www.goodreads.com/book/show/8855321-leviathan-wakes</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Leviathan Falls</span>
<span class="na">link</span><span class="pi">:</span> <span class="s">https://www.goodreads.com/book/show/28335699-leviathan-falls</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Margaret Atwood</span>
<span class="na">books</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">The Handmaid's Tale</span>
<span class="na">link</span><span class="pi">:</span> <span class="s">https://www.goodreads.com/book/show/38447.The_Handmaid_s_Tale</span>
<span class="nn">---</span>
</code></pre></div></div>
<p class="notice--info">I found Markdown documents with embedded metadata easy enough to work with. If you prefer specific JSON or YML files, you can use the <code class="language-plaintext highlighter-rouge">--metadata-file</code> flag to pass them to your template. See also the other <a href="https://pandoc.org/MANUAL.html#reader-options">reader options</a>.</p>
<p>Then, run <a href="https://pandoc.org/">Pandoc</a> to generate your static site. And that’s it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pandoc <span class="nt">--standalone</span> <span class="nt">--template</span> template.html authors.md <span class="nt">-o</span> index.html
</code></pre></div></div>
<p>I’ll take Pandoc over other options every day of the week for this particular use case. What about you?</p>
<h2 id="i-dont-want-to-install-pandoc">I Don’t Want to Install Pandoc!</h2>
<p>I know, right? I don’t want to either. Let’s use Docker instead. Put this into your <code class="language-plaintext highlighter-rouge">.aliases</code> and use <code class="language-plaintext highlighter-rouge">pandock</code> rather than <code class="language-plaintext highlighter-rouge">pandoc</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">alias </span><span class="nv">pandock</span><span class="o">=</span><span class="s1">'docker run --rm -v "$(pwd):/data" -u $(id -u):$(id -g) pandoc/latex'</span>
</code></pre></div></div>Hans SchnedlitzSometimes, you want to build a simple HTML page and populate it with some data. You may have some JSON lying around and want to make a simple website to visualize its contents. Or perhaps you want to show off how many books you read in the last few years.(Ab)Using Single Table Inheritance to Refactor Fat Models2021-07-24T17:00:00+00:002021-07-24T17:00:00+00:00https://hansschnedlitz.com/2021/07/24/ab-using-single-table-inheritance-to-simplify-your-legacy-models<p>How to deal with a model that tries to do too much? Consider something like this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="n">validates</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">presence: </span><span class="kp">true</span>
<span class="n">validates</span> <span class="ss">:color</span><span class="p">,</span> <span class="ss">inclusion: </span><span class="p">{</span> <span class="ss">in: </span><span class="p">[</span><span class="s1">'green'</span><span class="p">]</span> <span class="p">},</span> <span class="ss">if: </span><span class="o">-></span> <span class="p">{</span> <span class="nb">name</span> <span class="o">==</span> <span class="s1">'AVOCADO'</span><span class="p">}</span>
<span class="n">validates</span> <span class="ss">:color</span><span class="p">,</span> <span class="ss">inclusion: </span><span class="p">{</span> <span class="ss">in: </span><span class="p">[</span><span class="s1">'yellow'</span><span class="p">]</span> <span class="p">},</span> <span class="ss">if: </span><span class="o">-></span> <span class="p">{</span> <span class="nb">name</span> <span class="o">==</span> <span class="s1">'POTATO'</span><span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>It makes sense to split this up into three classes: <code class="language-plaintext highlighter-rouge">Vegetable</code>, <code class="language-plaintext highlighter-rouge">Avocado</code> and <code class="language-plaintext highlighter-rouge">Potato</code>. Alright, this particular class isn’t so bad, but imagine <code class="language-plaintext highlighter-rouge">Vegetable</code> being hundreds of lines long and containing dozens of validations like that. Yikes.</p>
<figure>
<img src="https://hansschnedlitz.com/assets/images/posts/2021-07-25/potato.jpg" alt="A big Potato" class="align-center" />
<figcaption>Quite the enormous potato!</figcaption>
</figure>
<h2 id="cutting-up-our-fat-model">Cutting Up our Fat Model</h2>
<p>Rails gives us a straightforward way to refactor <code class="language-plaintext highlighter-rouge">Vegetable</code>: <a href="https://api.rubyonrails.org/classes/ActiveRecord/Inheritance.html">Single Table Inheritance</a>. We can create some submodels to split up <code class="language-plaintext highlighter-rouge">Vegetable</code> and improve our code’s cohesion.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ApplicationRecord</span>
<span class="n">validates</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">presence: </span><span class="kp">true</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">Avocado</span> <span class="o"><</span> <span class="no">Vegetable</span>
<span class="n">validates</span> <span class="ss">:color</span><span class="p">,</span> <span class="ss">inclusion: </span><span class="p">{</span> <span class="ss">in: </span><span class="p">[</span><span class="s1">'green'</span><span class="p">]</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">Potato</span> <span class="o"><</span> <span class="no">Vegetable</span>
<span class="n">validates</span> <span class="ss">:color</span><span class="p">,</span> <span class="ss">inclusion: </span><span class="p">{</span> <span class="ss">in: </span><span class="p">[</span><span class="s1">'yellow'</span><span class="p">]</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>If you create a new model from scratch, this approach will just work™. But if you are working with existing code, as in our case, things tend not to be so simple. Rails makes two assumptions when you use single table inheritance:</p>
<ul>
<li>The subtype of your model is designated by a column <code class="language-plaintext highlighter-rouge">type</code>.</li>
<li>The <code class="language-plaintext highlighter-rouge">type</code> column contains the literal name of your subtypes, e.g. <code class="language-plaintext highlighter-rouge">Avocado</code>, <code class="language-plaintext highlighter-rouge">Potato</code>.</li>
</ul>
<p>Our models don’t adhere to these requirements. Our database looks like this:</p>
<table>
<thead>
<tr>
<th>id</th>
<th>name</th>
<th>color</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>VEGETABLE</td>
<td><code class="language-plaintext highlighter-rouge">nil</code></td>
</tr>
<tr>
<td>2</td>
<td>AVOCADO</td>
<td>green</td>
</tr>
<tr>
<td>3</td>
<td>POTATO</td>
<td>yellow</td>
</tr>
</tbody>
</table>
<p>The value that distinguishes the types of vegetables lives in the <code class="language-plaintext highlighter-rouge">name</code> column rather than the <code class="language-plaintext highlighter-rouge">type</code> column. Also, the names themselves are uppercase versions of our subclass-names: <code class="language-plaintext highlighter-rouge">AVOCADO</code> rather than <code class="language-plaintext highlighter-rouge">Avocado</code> and so on. To solve these issues you <em>could</em> migrate your data - and if you can, you definitely should! But sometimes that just isn’t an option.</p>
<p>Luckily, there are ways to shoehorn single table inheritance into models like these.</p>
<h2 id="adapting-single-table-inheritance">Adapting Single Table Inheritance</h2>
<p>After splitting up your models, you may try to run and run a query to get all potatoes:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Potato</span><span class="p">.</span><span class="nf">all</span>
</code></pre></div></div>
<p>Surprise. Instead of returning only a single record, all vegetables are returned. This is not at all what we wanted! We need to tell Rails about our non-standard inheritance column <code class="language-plaintext highlighter-rouge">name</code>. To do so, we can update the parent model <code class="language-plaintext highlighter-rouge">Vegetable</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ApplicationRecord</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">inheritance_column</span><span class="o">=</span><span class="s1">'name'</span>
<span class="o">...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Unfortunately, doing so will not only make our queries <em>still</em> return nonsense - <code class="language-plaintext highlighter-rouge">Potato.all</code> now returns no records at all - but also break a bunch of other things. Even creating new vegetables now fails:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Raises ActiveRecord::SubclassNotFound (The single-table inheritance mechanism failed to locate the subclass: 'POTATO'...</span>
<span class="no">Vegetable</span><span class="p">.</span><span class="nf">create</span><span class="p">(</span><span class="ss">name: </span><span class="s1">'POTATO'</span><span class="p">)</span>
</code></pre></div></div>
<p>Rails expects the inheritance column to contain the class name of the specific sub-type, <code class="language-plaintext highlighter-rouge">Potato</code> rather than <code class="language-plaintext highlighter-rouge">POTATO</code>. Under the hood, it executes <code class="language-plaintext highlighter-rouge">POTATO.constantize</code>, which of course doesn’t work. We have to change how Rails locates the types used to instantiate STI records. But how?</p>
<p>Enter <code class="language-plaintext highlighter-rouge">sti_class_for</code>. By overwriting this method, we can customize which types are used for instantiation:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">inheritance_column</span> <span class="o">=</span> <span class="s2">"name"</span>
<span class="k">class</span> <span class="o"><<</span> <span class="nb">self</span>
<span class="k">def</span> <span class="nf">sti_class_for</span><span class="p">(</span><span class="n">type_name</span><span class="p">)</span>
<span class="k">super</span><span class="p">(</span><span class="n">type_name</span><span class="p">.</span><span class="nf">dowcase</span><span class="p">.</span><span class="nf">camelize</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p class="notice--warning"><strong>Warning</strong>: <code class="language-plaintext highlighter-rouge">sti_class_for</code> was added in Rails 6.1. If you are stuck with an earlier version of Rails, you can use <code class="language-plaintext highlighter-rouge">find_sti_class</code> instead. It does pretty much the same thing but is <em>private</em>. You can still overwrite it all the same of course, just be careful.</p>
<p>That fixes querying for vegetables. However, querying our subclasses and creating new sub-records <em>still</em> does not work like we want it to:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Potato</span><span class="p">.</span><span class="nf">all</span>
<span class="o">=></span> <span class="c1">#<ActiveRecord::Relation []></span>
<span class="no">Potato</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">color: </span><span class="s1">'yellow'</span><span class="p">)</span>
<span class="o">=></span> <span class="c1">#<Potato id: 6, name: "Potato", color: "yellow",</span>
</code></pre></div></div>
<p>Although we overwrite <code class="language-plaintext highlighter-rouge">sti_class_for</code>, Rails uses the wrong <code class="language-plaintext highlighter-rouge">name</code> values. We have to ask ourselves: How does Rails know which values to put into the inheritance column when instantiating child records? It uses <code class="language-plaintext highlighter-rouge">sti_name</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">sti_name</span>
<span class="n">store_full_sti_class</span> <span class="p">?</span> <span class="nb">name</span> <span class="p">:</span> <span class="nb">name</span><span class="p">.</span><span class="nf">demodulize</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You probably know where this is going. Let’s overwrite <code class="language-plaintext highlighter-rouge">sti_name</code> as well:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Vegetable</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">inheritance_column</span> <span class="o">=</span> <span class="s2">"name"</span>
<span class="k">class</span> <span class="o"><<</span> <span class="nb">self</span>
<span class="k">def</span> <span class="nf">sti_class_for</span><span class="p">(</span><span class="n">type_name</span><span class="p">)</span>
<span class="k">super</span><span class="p">(</span><span class="n">type_name</span><span class="p">.</span><span class="nf">lower</span><span class="p">.</span><span class="nf">camelize</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">sti_name</span>
<span class="nb">name</span><span class="p">.</span><span class="nf">upcase</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Success! We have refactored the chonky <code class="language-plaintext highlighter-rouge">Vegetable</code>, and we can work with our subclasses just like we would expect:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Potato</span><span class="p">.</span><span class="nf">all</span>
<span class="o">=></span> <span class="c1">#<ActiveRecord::Relation [#<Potato id: 3, name: "POTATO", color: "yellow", created_at: "2021-07-25 14:41:00.032041000 +0000", updated_at: "2021-07-25 14:41:00.032041000 +0000">]></span>
<span class="no">Potato</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">color: </span><span class="s1">'yellow'</span><span class="p">)</span>
<span class="o">=></span> <span class="c1">#<Potato id: nil, name: "POTATO", color: "yellow", created_at: nil, updated_at: nil></span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Single Table Inheritance can be useful to re-organize existing models that have grown too large. Ideally, you’d never have to reach for this approach, but when are things ever try ideal? If you got any use out of this short guide let me know on <a href="https://twitter.com/hschnedlitz">twitter</a> 🤗</p>Hans SchnedlitzHow to deal with a model that tries to do too much? Consider something like this:Real-Time Command Line Applications with Action Cable and Thor2021-04-04T17:00:00+00:002021-04-04T17:00:00+00:00https://hansschnedlitz.com/2021/04/04/build-real-time-clis-with-actioncable<p>If you build a Rails application that has any kind of real-time feature, chances are you use <a href="https://guides.rubyonrails.org/action_cable_overview.html">Action Cable</a>.</p>
<p>Action Cable allows you to build nice things such as feeds that automatically refresh as new content is published, or editors that display a list of users currently working on a document. Under the hood, it uses Websockets to stream changes to clients as they happen.</p>
<p>The most commonly used client is, of course, the web browser. But that doesn’t mean you can’t leverage Action Cable when using other kinds of clients - such as command line applications.</p>
<p>Imagine a command line client that triggers some long-running job on the server. Wouldn’t it be nice to give users live updates on how that job is advancing?</p>
<p>In this guide, I’ll show you how to build exactly that. We’ll create a command line app that connects to an Action Cable server, triggers a lengthy background job, and then displays live updates about its progress.</p>
<p>A (moving?) picture tells more than a thousand words.</p>
<p><a href="https://hansschnedlitz.com/assets/images/posts/2021-04-05/demo.gif"><img src="https://hansschnedlitz.com/assets/images/posts/2021-04-05/demo.gif" alt="demo" /></a></p>
<p class="notice--info">This is a long post. If you have no patience for words, you can find the source code of the result on <a href="https://github.com/hschne/actioncable-cli">GitHub</a>.</p>
<h2 id="the-server">The Server</h2>
<p>To start things off let’s create a new Rails application. We don’t need most of Rails’ functionality in this guide, so we can skip a lot of things.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails new actioncable-cli \
--skip-action-mailer \
--skip-action-mailbox \
--skip-action-text \
--skip-active-job \
--skip-active-record \
--skip-active-storage \
--skip-javascript \
--skip-jbuilder \
--skip-spring \
--skip-test \
--skip-system-test \
--skip-webpack-install \
--skip-turbolinks
</code></pre></div></div>
<p>We will create our command line app later. First, we have to make some changes to the Action cable connection. Usually, clients provide information about the currently logged-in user, for example through session cookies, which then serves as a connection identifier. See the <a href="https://guides.rubyonrails.org/action_cable_overview.html#connection-setup">official connection docs</a>.</p>
<p>Our command line app offers no such thing. We could add some sort of authentication mechanism, but to keep things simple we won’t. We will use a simple UUID to identify connections.</p>
<p>Open and modify <code class="language-plaintext highlighter-rouge">app/channels/application_cable/connection.rb</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">ApplicationCable</span>
<span class="k">class</span> <span class="nc">Connection</span> <span class="o"><</span> <span class="no">ActionCable</span><span class="o">::</span><span class="no">Connection</span><span class="o">::</span><span class="no">Base</span>
<span class="n">identified_by</span> <span class="ss">:client_id</span>
<span class="k">def</span> <span class="nf">connect</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">client_id</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="nf">params</span><span class="p">[</span><span class="ss">:client_id</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Next, create a worker channel, through which we’ll later publish updates. Create a new file <code class="language-plaintext highlighter-rouge">app/channels/worker_channel.rb</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">WorkerChannel</span> <span class="o"><</span> <span class="no">ApplicationCable</span><span class="o">::</span><span class="no">Channel</span>
<span class="k">def</span> <span class="nf">subscribed</span>
<span class="n">stream_for</span> <span class="s2">"client_</span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">unsubscribed</span>
<span class="n">stop_all_streams</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Because we’ll be connecting from the command line, we’ll have to disable some security measures Rails enables by default. Uncomment this line in <code class="language-plaintext highlighter-rouge">development.rb</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>config.action_cable.disable_request_forgery_protection = true
</code></pre></div></div>
<p>Now that we have the connection and channel set up, let’s create a background job. We’ll be using <a href="https://github.com/mperham/sidekiq">Sidekiq</a>, so add this to your Gemfile:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'sidekiq'</span><span class="p">,</span> <span class="s1">'~> 6.1'</span>
</code></pre></div></div>
<p>We must also make sure that <a href="https://redis.io/">Redis</a> is up and running because Sidekiq relies on that for managing background workers. If you use <code class="language-plaintext highlighter-rouge">docker-compose</code>, add the following to <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3"</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">actioncable-cli-redis</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">redis</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">redis</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">6379:6379</span>
</code></pre></div></div>
<p>Next, create a new worker in <code class="language-plaintext highlighter-rouge">app/workers</code>. It won’t be doing any actual work, mostly it will be taking a nap.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Worker</span>
<span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span>
<span class="k">def</span> <span class="nf">perform</span>
<span class="n">steps</span> <span class="o">=</span> <span class="mi">5</span>
<span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="n">steps</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">progress</span><span class="o">|</span>
<span class="nb">sleep</span><span class="p">(</span><span class="nb">rand</span><span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="mi">3</span><span class="p">))</span>
<span class="no">Sidekiq</span><span class="p">.</span><span class="nf">logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="s2">"Step </span><span class="si">#{</span><span class="n">progress</span><span class="si">}</span><span class="s2"> for client"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>To offer a way to start the background job we just created, add a new controller with the following contents:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">WorkersController</span> <span class="o"><</span> <span class="no">ApplicationController</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="no">Worker</span><span class="p">.</span><span class="nf">perform_async</span>
<span class="n">head</span><span class="p">(</span><span class="ss">:ok</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Don’t forget to also add a new route to your <code class="language-plaintext highlighter-rouge">routes.rb</code>!</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">get</span> <span class="s1">'/workers/start'</span><span class="p">,</span> <span class="ss">to: </span><span class="s1">'workers#start'</span>
</code></pre></div></div>
<p>This is a good point to stop and check how badly broken everything is :crossed_fingers:</p>
<p>Start your Rails app, Sidekiq, and start the worker. If all is in order, you should see your worker writing to the Sidekiq logs.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails start
bundle <span class="nb">exec </span>sidekiq
<span class="c"># Send a request to trigger the worker</span>
curl <span class="s2">"http://localhost:3000/workers/start"</span>
</code></pre></div></div>
<p>Success? Then on to the next part.</p>
<h2 id="the-command-line-app">The Command Line App</h2>
<p>Our command line application will offer just a single command - one that starts the worker. <a href="https://github.com/erikhuda/thor">Thor</a> is a simple way to create command line apps, and it’s bundled with Rails, so we’ll be using that to implement that command.</p>
<p>Create <code class="language-plaintext highlighter-rouge">worker.thor</code> in your <code class="language-plaintext highlighter-rouge">lib/tasks</code> directory:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'thor'</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="kp">include</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Actions</span>
<span class="n">desc</span> <span class="s1">'start'</span><span class="p">,</span> <span class="s1">'Start a worker process'</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="nb">puts</span> <span class="s1">'Hello there!'</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You can test your command using <code class="language-plaintext highlighter-rouge">bundle exec thor worker:start</code>.</p>
<p>To receive live updates using Websockets we’ll need a Websocket client. I used <a href="https://socketry.github.io/async-websocket/">async-websocket</a>. Add it to your Gemfile:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'async-websocket'</span><span class="p">,</span> <span class="s1">'~> 0.17'</span>
</code></pre></div></div>
<p>Then update your command to connect to the server. Note that we generate a UUID to identify the connection. Remember that we adapted the Action Cable connection to make use of this <code class="language-plaintext highlighter-rouge">client_id</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'thor'</span>
<span class="nb">require</span> <span class="s1">'securerandom'</span>
<span class="nb">require</span> <span class="s1">'async'</span>
<span class="nb">require</span> <span class="s1">'async/http/endpoint'</span>
<span class="nb">require</span> <span class="s1">'async/websocket/client'</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="kp">include</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Actions</span>
<span class="n">desc</span> <span class="s1">'start'</span><span class="p">,</span> <span class="s1">'Start a worker process'</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="vi">@client_id</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">uuid</span>
<span class="n">url</span> <span class="o">=</span> <span class="s2">"ws://localhost:3000/cable?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Async</span> <span class="k">do</span> <span class="o">|</span><span class="n">_</span><span class="o">|</span>
<span class="n">endpoint</span> <span class="o">=</span> <span class="no">Async</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Endpoint</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="no">Async</span><span class="o">::</span><span class="no">WebSocket</span><span class="o">::</span><span class="no">Client</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">endpoint</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">connection</span><span class="o">|</span>
<span class="k">while</span> <span class="p">(</span><span class="n">message</span> <span class="o">=</span> <span class="n">connection</span><span class="p">.</span><span class="nf">read</span><span class="p">)</span>
<span class="nb">puts</span> <span class="n">message</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Run the command and check the server logs. You should see that a connection has been established, and should start receiving ping messages on the command line.</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bundle exec thor worker:start
{:type=>"welcome"}
{:type=>"ping", :message=>1617639988}
...
</code></pre></div></div>
<p>Now we need to subscribe to the worker channel. As soon as the subscription was confirmed, we are ready to receive messages. We can then start the worker.</p>
<p>Adapt the Thor command as follows:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'thor'</span>
<span class="nb">require</span> <span class="s1">'securerandom'</span>
<span class="nb">require</span> <span class="s1">'net/http'</span>
<span class="nb">require</span> <span class="s1">'async'</span>
<span class="nb">require</span> <span class="s1">'async/http/endpoint'</span>
<span class="nb">require</span> <span class="s1">'async/websocket/client'</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="kp">include</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Actions</span>
<span class="n">desc</span> <span class="s1">'start'</span><span class="p">,</span> <span class="s1">'Start a worker process'</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="vi">@client_id</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">uuid</span>
<span class="n">url</span> <span class="o">=</span> <span class="s2">"ws://localhost:3000/cable?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Async</span> <span class="k">do</span> <span class="o">|</span><span class="n">_</span><span class="o">|</span>
<span class="n">endpoint</span> <span class="o">=</span> <span class="no">Async</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Endpoint</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="no">Async</span><span class="o">::</span><span class="no">WebSocket</span><span class="o">::</span><span class="no">Client</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">endpoint</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">connection</span><span class="o">|</span>
<span class="k">while</span> <span class="p">(</span><span class="n">message</span> <span class="o">=</span> <span class="n">connection</span><span class="p">.</span><span class="nf">read</span><span class="p">)</span>
<span class="n">on_receive</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">on_receive</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">handle_connection_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">handle_connection_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">type</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:type</span><span class="p">]</span>
<span class="k">case</span> <span class="n">type</span>
<span class="k">when</span> <span class="s1">'welcome'</span>
<span class="n">on_connected</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">'confirm_subscription'</span>
<span class="n">on_subscribed</span>
<span class="k">else</span>
<span class="nb">puts</span> <span class="n">message</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">on_connected</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">command: </span><span class="s1">'subscribe'</span><span class="p">,</span> <span class="ss">identifier: </span><span class="p">{</span> <span class="ss">channel: </span><span class="s1">'WorkerChannel'</span> <span class="p">}.</span><span class="nf">to_json</span> <span class="p">}</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">flush</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">on_subscribed</span>
<span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">start</span><span class="p">(</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">3000</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">http</span><span class="o">|</span>
<span class="n">http</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="s2">"/workers/start?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>All that is missing is to stream updates from the worker to the connected clients. We’ll have to make some small changes to our worker and worker controller:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Worker</span>
<span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span>
<span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="n">client_id</span><span class="p">)</span>
<span class="n">steps</span> <span class="o">=</span> <span class="mi">5</span>
<span class="no">WorkerChannel</span><span class="p">.</span><span class="nf">broadcast_to</span><span class="p">(</span><span class="s2">"client_</span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="ss">type: :worker_started</span><span class="p">,</span> <span class="ss">total: </span><span class="n">steps</span><span class="p">)</span>
<span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="n">steps</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">progress</span><span class="o">|</span>
<span class="nb">sleep</span><span class="p">(</span><span class="nb">rand</span><span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="mi">3</span><span class="p">))</span>
<span class="no">Sidekiq</span><span class="p">.</span><span class="nf">logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="s2">"Step </span><span class="si">#{</span><span class="n">progress</span><span class="si">}</span><span class="s2"> for client </span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="no">WorkerChannel</span><span class="p">.</span><span class="nf">broadcast_to</span><span class="p">(</span><span class="s2">"client_</span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="ss">type: :worker_progress</span><span class="p">,</span> <span class="ss">progress: </span><span class="n">progress</span><span class="p">)</span>
<span class="k">end</span>
<span class="no">WorkerChannel</span><span class="p">.</span><span class="nf">broadcast_to</span><span class="p">(</span><span class="s2">"client_</span><span class="si">#{</span><span class="n">client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="ss">type: :worker_done</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">WorkersController</span> <span class="o"><</span> <span class="no">ApplicationController</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="no">Worker</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span><span class="n">params</span><span class="p">[</span><span class="ss">:client_id</span><span class="p">])</span>
<span class="n">head</span><span class="p">(</span><span class="ss">:ok</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Note that the worker uses the <code class="language-plaintext highlighter-rouge">client_id</code> to publish messages to the correct clients. We publish messages when the worker has started, when there is progress, and when the worker has finished.</p>
<p>We’ll update the command line app to handle these messages. Let’s also add <code class="language-plaintext highlighter-rouge">ruby_progressbar</code> so we can display the progress to the user.</p>
<p>Add this to your Gemfile.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'ruby-progressbar'</span><span class="p">,</span> <span class="s1">'~> 1.11'</span>
</code></pre></div></div>
<p>Then update the Thor command once again. In the end, it should look like this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'thor'</span>
<span class="nb">require</span> <span class="s1">'securerandom'</span>
<span class="nb">require</span> <span class="s1">'net/http'</span>
<span class="nb">require</span> <span class="s1">'async'</span>
<span class="nb">require</span> <span class="s1">'async/io/stream'</span>
<span class="nb">require</span> <span class="s1">'async/http/endpoint'</span>
<span class="nb">require</span> <span class="s1">'async/websocket/client'</span>
<span class="nb">require</span> <span class="s1">'ruby-progressbar'</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="kp">include</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Actions</span>
<span class="n">desc</span> <span class="s1">'start'</span><span class="p">,</span> <span class="s1">'Start a worker process'</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="vi">@client_id</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">uuid</span>
<span class="n">url</span> <span class="o">=</span> <span class="s2">"ws://localhost:3000/cable?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Async</span> <span class="k">do</span> <span class="o">|</span><span class="n">_</span><span class="o">|</span>
<span class="n">endpoint</span> <span class="o">=</span> <span class="no">Async</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Endpoint</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="no">Async</span><span class="o">::</span><span class="no">WebSocket</span><span class="o">::</span><span class="no">Client</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">endpoint</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">connection</span><span class="o">|</span>
<span class="k">while</span> <span class="p">(</span><span class="n">message</span> <span class="o">=</span> <span class="n">connection</span><span class="p">.</span><span class="nf">read</span><span class="p">)</span>
<span class="n">on_receive</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">on_receive</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">if</span> <span class="n">message</span><span class="p">[</span><span class="ss">:type</span><span class="p">]</span>
<span class="n">handle_connection_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">else</span>
<span class="n">handle_channel_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">handle_connection_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">type</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:type</span><span class="p">]</span>
<span class="k">case</span> <span class="n">type</span>
<span class="k">when</span> <span class="s1">'welcome'</span>
<span class="n">on_connected</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">'confirm_subscription'</span>
<span class="n">on_subscribed</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">handle_channel_message</span><span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:message</span><span class="p">]</span>
<span class="n">type</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:type</span><span class="p">]</span>
<span class="k">case</span> <span class="n">type</span>
<span class="k">when</span> <span class="s1">'worker_started'</span>
<span class="n">total</span> <span class="o">=</span> <span class="n">message</span><span class="p">[</span><span class="ss">:total</span><span class="p">]</span>
<span class="vi">@bar</span> <span class="o">=</span> <span class="no">ProgressBar</span><span class="p">.</span><span class="nf">create</span><span class="p">(</span><span class="ss">title: </span><span class="s1">'Worker Progress'</span><span class="p">,</span> <span class="ss">total: </span><span class="n">total</span><span class="p">,</span> <span class="ss">format: </span><span class="s1">'%t %B %c/%C %P%%'</span><span class="p">)</span>
<span class="k">when</span> <span class="s1">'worker_progress'</span>
<span class="vi">@bar</span><span class="p">.</span><span class="nf">increment</span>
<span class="k">when</span> <span class="s1">'worker_done'</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">close</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">on_connected</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">command: </span><span class="s1">'subscribe'</span><span class="p">,</span> <span class="ss">identifier: </span><span class="p">{</span> <span class="ss">channel: </span><span class="s1">'WorkerChannel'</span> <span class="p">}.</span><span class="nf">to_json</span> <span class="p">}</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">flush</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">on_subscribed</span>
<span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">start</span><span class="p">(</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">3000</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">http</span><span class="o">|</span>
<span class="n">http</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="s2">"/workers/start?client_id=</span><span class="si">#{</span><span class="vi">@client_id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The most important change here is the addition of <code class="language-plaintext highlighter-rouge">handle_channel_message</code> where we handle the messages we receive from the worker to create and update the progress bar.</p>
<p>Before wrapping up, we need to make one final change. Update <code class="language-plaintext highlighter-rouge">cable.yml</code> to use Redis in development. We need to do this so that our Sidekiq process knows about subscriptions made using the main process.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">development</span><span class="pi">:</span>
<span class="na">adapter</span><span class="pi">:</span> <span class="s">redis</span>
<span class="na">url</span><span class="pi">:</span> <span class="s"><%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/0" } %></span>
</code></pre></div></div>
<p class="notice--info">The default mechanism for managing Action Cable connections in development is <code class="language-plaintext highlighter-rouge">async</code>, which uses in-memory structures. These are accessible only by the current process. That is no good when multiple processes need to utilize the same connections.</p>
<p>Restart your Rails server and, for good measure, Sidekiq process if you haven’t already and run the worker command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle exec thor worker:start
</code></pre></div></div>
<h2 id="just-the-beginning">Just the beginning…</h2>
<p>This guide is done, but the story of ActionCable and command line applications isn’t. Updating a progress bar is nice and all, but it is only scratching the surface.</p>
<p>There is much more to explore. How about streaming process logs live to clients? Or what about streaming user inputs directly to the server?</p>
<p>Anything is possible - you just have to try it! :woman_scientist:</p>Hans SchnedlitzIf you build a Rails application that has any kind of real-time feature, chances are you use Action Cable.Processing images with ActiveStorage and Imgproxy2021-03-19T09:11:00+00:002021-03-19T09:11:00+00:00https://hansschnedlitz.com/2021/03/19/user-avatars-with-imgproxy-and-activestorage<p><a href="https://edgeguides.rubyonrails.org/active_storage_overview.html">ActiveStorage</a> has a nifty feature that allows you to serve variants of uploaded images. Think of a user uploading a profile image. You won’t need the full-sized original most of the time. Instead, you can serve smaller versions of that image, which, of course, consume less bandwidth and load faster.</p>
<p>ActiveStorage uses the <a href="https://github.com/janko/image_processing">image_processing</a> gem under the hood to accomplish this. Here is how you could render downsized version of a user avatar:</p>
<div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><%=</span> <span class="n">image_tag</span> <span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">.</span><span class="nf">variant</span><span class="p">(</span><span class="ss">resize: </span><span class="s2">"100x100!"</span><span class="p">)</span> <span class="cp">%></span>
</code></pre></div></div>
<p>I’ve found ActiveStorage’s variant mechanism very easy to use, but it does have its downsides. Underlying libraries, such as <code class="language-plaintext highlighter-rouge">libvips</code> or <code class="language-plaintext highlighter-rouge">minimagick</code> have to be installed and kept up to date. Additionally, transforming images is CPU- and memory-intensive work, and doing this on your app servers may cause issues as your app grows.</p>
<p><a href="https://github.com/imgproxy/imgproxy">Imgproxy</a> is an alternative way to process images. A small service written in Go, it does one thing, and one thing only: processing images. As such, it allows us to offload the job of processing images to specialized machines, alleviating some of the problems described above.</p>
<p>To show you how that works we’ll create a small application that allows users to upload profile images. We’ll then use <code class="language-plaintext highlighter-rouge">image_processing</code> and later Imgproxy to serve downsized versions of those same images.</p>
<p class="notice--info">Words are nice and all, but code is pretty cool too, right? You can find the source code for this guide <a href="https://github.com/hschne/avatars">on GitHub</a> if you prefer that.</p>
<h2 id="setting-up-the-app">Setting up the App</h2>
<p>The simplest way to get a user model and user profile page is by using <a href="https://github.com/heartcombo/devise">Devise</a>.</p>
<p>I set up a demo application called ‘Avatars’ using <a href="https://github.com/hschne/schienenzeppelin">Schienenzeppelin</a>, my own Rails template that is configured with Devise, Tailwind, and other libraries.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sz avatars
</code></pre></div></div>
<p>If you are starting with a fresh app, you’ll have to install and configure Devise yourself. I recommend you follow the instructions <a href="https://github.com/heartcombo/devise#getting-started">here</a>.</p>
<p>Once your app is up and running, install ActiveStorage:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails active_storage:install
</code></pre></div></div>
<p>Next, update your user model to allow attaching images to it.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">has_one_attached</span> <span class="ss">:avatar</span>
</code></pre></div></div>
<p>We’ll now update some views and controllers to enable users to upload profile images. There are <a href="https://github.com/heartcombo/devise#strong-parameters"> several ways </a> to go about this, I prefer ejecting Devises controllers and modifying them as necessary.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails generate devise:controllers <span class="nb">users</span>
</code></pre></div></div>
<p>We’ll need to update only the <code class="language-plaintext highlighter-rouge">registrations_controller.rb</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Users::RegistrationsController</span> <span class="o"><</span> <span class="no">Devise</span><span class="o">::</span><span class="no">RegistrationsController</span>
<span class="n">before_action</span> <span class="ss">:configure_account_update_params</span><span class="p">,</span> <span class="ss">only: </span><span class="p">[</span><span class="ss">:update</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">configure_account_update_params</span>
<span class="n">devise_parameter_sanitizer</span><span class="p">.</span><span class="nf">permit</span><span class="p">(</span><span class="ss">:account_update</span><span class="p">,</span> <span class="ss">keys: </span><span class="sx">%i[name avatar]</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>To tell Devise to use this controller, update <code class="language-plaintext highlighter-rouge">routes.rb</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">devise_for</span> <span class="ss">:users</span><span class="p">,</span> <span class="ss">controllers: </span><span class="p">{</span> <span class="ss">registrations: </span><span class="s1">'users/registrations'</span> <span class="p">}</span>
</code></pre></div></div>
<p>We are now ready to update the view where users can update their profile. Open <code class="language-plaintext highlighter-rouge">app/views/devise/registrations/edit.html.erb</code>. If that file is not present you have to run <code class="language-plaintext highlighter-rouge">rails generate devise:views users</code> first.</p>
<p>The code below allows users to upload a new profile image, displays it if it is present, and shows a fallback icon using <a href="https://github.com/jamesmartin/inline_svg">inline_svg</a> otherwise.</p>
<div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">class=</span><span class="s">"input-group"</span><span class="nt">></span>
<span class="cp"><%</span> <span class="k">if</span> <span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">.</span><span class="nf">attached?</span> <span class="cp">%></span>
<span class="cp"><%=</span> <span class="n">image_tag</span> <span class="n">url_for</span><span class="p">(</span><span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">),</span> <span class="ss">class: </span><span class="s2">"rounded"</span> <span class="cp">%></span>
<span class="cp"><%</span> <span class="k">else</span> <span class="cp">%></span>
<span class="cp"><%=</span> <span class="n">inline_svg_pack_tag</span><span class="p">(</span><span class="s1">'media/images/user.svg'</span><span class="p">,</span> <span class="ss">class: </span><span class="s2">"rounded"</span><span class="p">,</span> <span class="ss">size: </span><span class="s2">"5rem * 5rem"</span><span class="p">)</span> <span class="cp">%></span>
<span class="cp"><%</span> <span class="k">end</span> <span class="cp">%></span>
<span class="cp"><%=</span> <span class="n">f</span><span class="p">.</span><span class="nf">file_field</span> <span class="ss">:avatar</span> <span class="cp">%></span>
<span class="nt"></div></span>
</code></pre></div></div>
<p>With these changes, you should be able to upload user profile images and display them.</p>
<p><img src="https://hansschnedlitz.com/assets/images/posts/2021-03-19/profile.png" alt="Profile" class="align-center" /></p>
<h2 id="processing-images">Processing Images</h2>
<p>Our current implementation renders exactly the image that the user uploaded. That’s wasteful. What if they upload a 15 megapixel, high-res picture of their face? To serve processed images - variants - we’ll need to install the <code class="language-plaintext highlighter-rouge">image_processing</code> gem.</p>
<p>Add this to your <code class="language-plaintext highlighter-rouge">Gemfile</code> and run <code class="language-plaintext highlighter-rouge">bundle install</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'image_processing'</span>
</code></pre></div></div>
<p>Now replace <code class="language-plaintext highlighter-rouge">url_for(resource.avatar)</code> with <code class="language-plaintext highlighter-rouge">resource.avatar.variant</code> to serve a resized image.</p>
<div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><%=</span> <span class="n">image_tag</span> <span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">.</span><span class="nf">variant</span><span class="p">(</span><span class="ss">resize: </span><span class="s2">"100x100!"</span><span class="p">),</span> <span class="ss">class: </span><span class="s2">"rounded"</span> <span class="cp">%></span>
</code></pre></div></div>
<p>You can find additional info on variants in the <a href="https://edgeguides.rubyonrails.org/active_storage_overview.html#transforming-images">official documentation</a>.</p>
<h2 id="adding-imageproxy">Adding Imageproxy</h2>
<p>So far so good. Image upload and processing using <code class="language-plaintext highlighter-rouge">image_processing</code> works, now let’s use Imgproxy. Add the <a href="https://github.com/imgproxy/imgproxy.rb">Imgproxy client</a>, and once again run <code class="language-plaintext highlighter-rouge">bundle install</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'imgproxy'</span>
</code></pre></div></div>
<p>There are several ways to configure the Imgproxy client. I prefer using an initializer, <code class="language-plaintext highlighter-rouge">initializers/imgproxy.rb</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Imgproxy</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">endpoint</span> <span class="o">=</span> <span class="s1">'http://localhost:8080'</span>
<span class="n">config</span><span class="p">.</span><span class="nf">key</span> <span class="o">=</span> <span class="s1">'696d6770726f7879'</span> <span class="c1"># imgproxy</span>
<span class="n">config</span><span class="p">.</span><span class="nf">salt</span> <span class="o">=</span> <span class="s1">'73616c74'</span> <span class="c1"># salt</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Key and salt are not strictly required but enable URL signing, which is a good security practice. You can find more information about URL signing in the <a href="https://docs.imgproxy.net/#/signing_the_url">Imgproxy documentation</a>.</p>
<p>To replace <code class="language-plaintext highlighter-rouge">image_processing</code> with <code class="language-plaintext highlighter-rouge">imgproxy</code> update the view and replace the line where we retrieve the variant with:</p>
<div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><%=</span> <span class="n">image_tag</span> <span class="n">resource</span><span class="p">.</span><span class="nf">avatar</span><span class="p">.</span><span class="nf">imgproxy_url</span><span class="p">(</span><span class="ss">width: </span><span class="mi">100</span><span class="p">,</span> <span class="ss">height: </span><span class="mi">100</span><span class="p">,</span> <span class="ss">format: </span><span class="s1">'jpg'</span><span class="p">),</span> <span class="ss">class: </span><span class="s2">"rounded"</span> <span class="cp">%></span>
</code></pre></div></div>
<p class="notice--info">In my testing I faced an issue where <code class="language-plaintext highlighter-rouge">imgproxy_url</code> failed. If that is the case for you as well, you may need to add <code class="language-plaintext highlighter-rouge">Rails.application.routes.default_url_options[:host] = 'localhost:3000'</code> to your <code class="language-plaintext highlighter-rouge">development.rb</code>.</p>
<p>Our Rails app is good-to-go. The last thing remaining is to start the Imgproxy service. If you use <code class="language-plaintext highlighter-rouge">docker-compose</code> add the following to your compose file and run <code class="language-plaintext highlighter-rouge">docker-compose up</code>.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">imgproxy</span><span class="pi">:</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">imgproxy</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">darthsim/imgproxy:latest</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">IMGPROXY_KEY=696d6770726f7879</span> <span class="c1"># imgproxy</span>
<span class="pi">-</span> <span class="s">IMGPROXY_SALT=73616c74</span> <span class="c1"># salt</span>
<span class="pi">-</span> <span class="s">IMGPROXY_LOCAL_FILESYSTEM_ROOT=/storage</span>
<span class="na">network_mode</span><span class="pi">:</span> <span class="s">host</span>
</code></pre></div></div>
<p>Note that <code class="language-plaintext highlighter-rouge">network_mode: host</code> is required, as Imgproxy needs to connect to your Rails application to retrieve original images. For alternative ways to run Imgproxy consult the <a href="https://docs.imgproxy.net/#/installation">documentation</a>.</p>
<p>If you reload your profile page, you should see something like this in your Docker logs, which of course means that Imgproxy is doing its job.</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>imgproxy | INFO [2021-03-19T07:57:06Z] Completed in 61.071755ms /../s:100:100/plain/http://localhost:3000/rails/active_storage/blobs/redirect/../avatar.png@jpg request_id=7l4aJfTZ6kdjq0Ul1ITx9 method=GET status=200 image_url="http://localhost:3000/rails/active_storage/blobs/redirect/.../Hans.png" processing_options="Width: 100; Height: 100; Format: jpeg"
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Imgproxy can alleviate some of the pains associated with running ActiveStorage as your app grows - even if it requires additional infrastructure. In the context of this post, we didn’t see much of that.</p>
<p>A word of warning: Don’t get all hyped up. Using Imgproxy is probably not worth it unless your app is large enough. But still: It’s good to know what’s out there and what is possible, right?</p>
<p>I hope you learned a thing or two - let me know if you found this useful!</p>Hans SchnedlitzActiveStorage has a nifty feature that allows you to serve variants of uploaded images. Think of a user uploading a profile image. You won’t need the full-sized original most of the time. Instead, you can serve smaller versions of that image, which, of course, consume less bandwidth and load faster.Load testing GraphQL with WRK2021-03-09T17:00:00+00:002021-03-09T17:00:00+00:00https://hansschnedlitz.com/2021/03/09/load-testing-graphql-with-wrk<p>For performance testing <a href="https://github.com/wg/wrk">wrk</a> is one of my favorite tools. Whether you are trying to get a quick benchmark or building a performance test suite - it is fairly simple using <code class="language-plaintext highlighter-rouge">wrk</code>.</p>
<p>That is, as long as your requests are fairly simple. To benchmark a GET request against a classic REST API you could run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk -c <user-count> -t <cpu-core-count> -d 10 --latency https://your-server/api/books
</code></pre></div></div>
<p>Easy as pie, right? But what about <em>less simple</em> requests? Specifically, does <code class="language-plaintext highlighter-rouge">wrk</code> work for, say, <a href="https://graphql.org/">GraphQL</a> APIs?</p>
<p>While are other tools that you could use (for example <a href="https://k6.io/">k6</a>), <code class="language-plaintext highlighter-rouge">wrk</code> does the job well enough - if you aren’t scared of a little <a href="http://www.lua.org/">Lua</a> scripting.</p>
<p class="notice--warning">What you are seeing in this post is pretty much the entirety of all the Lua I’ve written in my entire life. So. Now you know. Take the Lua parts with a grain of salt.</p>
<h2 id="using-lua-to-perform-graphql-queries">Using Lua to perform GraphQL Queries</h2>
<p>You may pass a Lua script to <code class="language-plaintext highlighter-rouge">wrk</code> using the <code class="language-plaintext highlighter-rouge">-s</code> option.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk <span class="nt">-s</span> script.lua https://your-server/api/graphql
</code></pre></div></div>
<p>To perform a GraphQL query the script could look something like this:</p>
<div class="language-lua highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrk</span><span class="p">.</span><span class="n">method</span> <span class="o">=</span> <span class="s2">"POST"</span>
<span class="n">wrk</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s2">"Content-Type"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"application/json"</span>
<span class="n">wrk</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s2">"Accept"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"application/json"</span>
<span class="n">query</span> <span class="o">=</span> <span class="s">[[
query books($ids: [Integer]) {
books(ids: $ids) {
id
name
price
}
}
}
]]</span>
<span class="n">variables</span> <span class="o">=</span> <span class="s">[[
"ids": [1,2,3,4]
]]</span>
<span class="n">wrk</span><span class="p">.</span><span class="n">body</span> <span class="o">=</span><span class="s1">'{"query": "'</span> <span class="o">..</span> <span class="nb">string.gsub</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span> <span class="o">..</span> <span class="s1">'", "variables": {'</span> <span class="o">..</span> <span class="nb">string.gsub</span><span class="p">(</span><span class="n">variables</span><span class="p">,</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span> <span class="o">..</span> <span class="s1">'} }'</span>
</code></pre></div></div>
<p>Even if you have no clue about Lua you can most likely infer what is going on here. A quick explanation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk.method = "POST"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Accept"] = "application/json"
</code></pre></div></div>
<p>Most GraphQL servers require specific headers and HTTP verbs when processing requests. If your API has some specific requirements you will need to change this accordingly.</p>
<div class="language-lua highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">query</span> <span class="o">=</span> <span class="s">[[
query books($ids: [Integer]) {
books(ids: $ids) {
id
name
price
}
}
}
]]</span>
<span class="n">variables</span> <span class="o">=</span> <span class="s">[[
"ids": [1,2,3,4]
]]</span>
</code></pre></div></div>
<p>Double-brackets (<code class="language-plaintext highlighter-rouge">[[...]]</code>) denote multi-line strings in Lua, which allow us to specify our query and variables in a readable way.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk.body ='{"query": "' .. string.gsub(query, '\n', '') .. '", "variables": {' .. string.gsub(variables, '\n', '') .. '} }'
</code></pre></div></div>
<p>This sets the request body - as you might expect. Because the request body must contain valid JSON, we remove any line breaks from our <code class="language-plaintext highlighter-rouge">query</code> and <code class="language-plaintext highlighter-rouge">variables</code> variables and use string interpolation (<code class="language-plaintext highlighter-rouge">..</code>) to insert them into the final payload.</p>
<p>And that’s it. Have fun performance testing your GraphQL APIs! :rocket:</p>Hans SchnedlitzFor performance testing wrk is one of my favorite tools. Whether you are trying to get a quick benchmark or building a performance test suite - it is fairly simple using wrk.CLI OAuth in Ruby2021-02-26T17:00:00+00:002021-02-26T17:00:00+00:00https://hansschnedlitz.com/2021/02/26/cli-oauth-in-ruby<p>Have you ever used a command-line application that triggered an <a href="https://oauth.net/2/">OAuth</a> authentication flow to log you in and wondered how that works? For example, <a href="https://cloud.google.com/sdk">Google Cloud SDK</a> does this, as does <a href="https://devcenter.heroku.com/articles/heroku-cli">Heroku CLI</a>.</p>
<p>I’ve always found that a pretty neat way to handle authorization on the command line because it feels so <em>effortless</em> to the user. You run a command, your browser opens, you log in like you would on a website, and <em>Bam!</em>, you’re logged in on the command line.</p>
<p>So how does that work?</p>
<p>To find out, we’ll build a simple Thor app that supports OAuth login with Google. If you don’t care about words and just want to see the code you can find it on <a href="https://github.com/hschne/googleme">GitHub</a>.</p>
<p class="notice--warning">This post assumes you are somewhat familiar with OAuth. Also, the demo app we are building here should be considered a proof-of-concept. There are lots of holes and rough edges that still need ironing out before it can be used productively.</p>
<h2 id="basics">Basics</h2>
<p>OAuth can be a bit complicated. I’m not going to get into any details - there are tons of articles explaining it much better than I could. If you need a refresher I’m sure you can find some detailed information on the web :wink:</p>
<p>For now, just consider that two things make OAuth for command-line applications interesting:</p>
<ol>
<li>You do not own a trusted domain. The component that is starting the OAuth flow is a command-line application. There simply is no webpage to redirect the authentication provider to.</li>
<li>The client itself is untrusted. You do not own the platform where the code initiating the OAuth flow is running. Similar to a mobile app, you must assume that you cannot keep secrets secret, and as such, your OAuth flow cannot use a client secret.</li>
</ol>
<p>The first issue we can solve by starting a local server that we can redirect to. So, <code class="language-plaintext highlighter-rouge">localhost</code> becomes our callback domain. When authorizing with Google, this is already accounted for when we create OAuth credentials for <em>Desktop</em> applications.</p>
<p>To solve the second issue we’ll use the <a href="https://tools.ietf.org/html/rfc7636">PKCE extension</a> for OAuth. This aspect of OAuth, and the security implications of not being able to keep the client secret a secret, is a bit complicated. <a href="https://developer.okta.com/docs/concepts/oauth-openid/#authorization-code-with-pkce-flow">This Okta post</a> does a good job of explaining why PKCE works as a solution.</p>
<h2 id="creating-the-oauth-client">Creating the OAuth Client</h2>
<p>Let’s start by creating a simple command-line application. Our app will only provide two commands: A <code class="language-plaintext highlighter-rouge">login</code> command, which triggers the OAuth flow, and a <code class="language-plaintext highlighter-rouge">user</code> command, which performs an authorized request to retrieve some information from the Google API.</p>
<p>We’ll use <a href="https://github.com/erikhuda/thor">Thor</a> to create the app:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Error</span> <span class="o"><</span> <span class="no">Thor</span><span class="o">::</span><span class="no">Error</span><span class="p">;</span> <span class="k">end</span>
<span class="k">class</span> <span class="nc">Main</span> <span class="o"><</span> <span class="no">Thor</span>
<span class="n">desc</span> <span class="s1">'login'</span><span class="p">,</span> <span class="s1">'Login with Google'</span>
<span class="k">def</span> <span class="nf">login</span>
<span class="c1"># TODO: Login code</span>
<span class="k">end</span>
<span class="n">desc</span> <span class="s1">'user'</span><span class="p">,</span> <span class="s1">'Retrieve user data'</span>
<span class="k">def</span> <span class="nf">user</span>
<span class="c1"># TODO: API Request</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Before we can implement the OAuth flow we need to create OAuth Client IDs in the Google Cloud Console. If you are starting with a new project, you must create a new <a href="https://console.cloud.google.com/apis/credentials/consent">consent screen</a> first.</p>
<p>Fill in the required information - you do not need to provide authorized domains or app domains. When selecting scopes we only need the <code class="language-plaintext highlighter-rouge">userinfo.profile</code> scope, as that is the only information we want access to.</p>
<p>Head over to <a href="https://console.cloud.google.com/apis/credentials">credentials</a> and create new <code class="language-plaintext highlighter-rouge">OAuth client ID</code> credentials. As application type select <code class="language-plaintext highlighter-rouge">Desktop app</code>. Take note of both client ID and secret, you’ll need them later</p>
<p class="notice--info">‘Didn’t you just say we can’t use client secrets on untrusted platforms?’ I hear you say. Well, yes, but it seems that Google is a bit, like, doing their own thing here. Even though the desktop client goes through a PKCE flow, it must <em>still</em> provide a client secret and that secret is essentially treated as public information. <a href="https://stackoverflow.com/a/61970107/2553104">This SO comment</a> sheds some light on this weird situation.</p>
<h2 id="implementing-the-oauth-flow">Implementing the OAuth Flow</h2>
<p>As mentioned previously, to receive callbacks from the authorization server, we need to start a local server to receive those callbacks. Let’s create it.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'socket'</span>
<span class="nb">require</span> <span class="s1">'uri'</span>
<span class="nb">require</span> <span class="s1">'cgi'</span>
<span class="k">module</span> <span class="nn">Goggleme</span>
<span class="k">class</span> <span class="nc">Server</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="vi">@state</span> <span class="o">=</span> <span class="n">state</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="n">server</span> <span class="o">=</span> <span class="no">TCPServer</span><span class="p">.</span><span class="nf">new</span> <span class="mi">9876</span>
<span class="k">while</span> <span class="n">connection</span> <span class="o">=</span> <span class="n">server</span><span class="p">.</span><span class="nf">accept</span>
<span class="n">request</span> <span class="o">=</span> <span class="n">connection</span><span class="p">.</span><span class="nf">gets</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">handle</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">puts</span> <span class="s1">'OAuth request received. You can close this window now.'</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">close</span>
<span class="k">return</span> <span class="n">data</span> <span class="k">if</span> <span class="n">data</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">handle</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="n">_</span><span class="p">,</span> <span class="n">full_path</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span>
<span class="n">path</span> <span class="o">=</span> <span class="no">URI</span><span class="p">(</span><span class="n">full_path</span><span class="p">).</span><span class="nf">path</span>
<span class="n">handle_authorize</span><span class="p">(</span><span class="n">full_path</span><span class="p">)</span> <span class="k">if</span> <span class="n">path</span> <span class="o">==</span> <span class="s1">'/authorize'</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">handle_authorize</span><span class="p">(</span><span class="n">full_path</span><span class="p">)</span>
<span class="n">params</span> <span class="o">=</span> <span class="no">CGI</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="no">URI</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">full_path</span><span class="p">).</span><span class="nf">query</span><span class="p">)</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s1">'Invalid oauth request received'</span><span class="p">)</span> <span class="k">if</span> <span class="vi">@state</span> <span class="o">!=</span> <span class="n">params</span><span class="p">[</span><span class="s1">'state'</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">params</span><span class="p">[</span><span class="s1">'code'</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Executing this will start a server on port <code class="language-plaintext highlighter-rouge">9876</code> that listens for requests to the <code class="language-plaintext highlighter-rouge">/authorize</code> endpoint. Upon receiving such a request, we verify that it contains the correct parameters and return the authorization code.</p>
<p>After the local server is ready to receive requests we need to open the Browser to allow the user to login using the selected authentication provider - in our case Google. Because we use PKCE, there is a small twist. We need to create a <code class="language-plaintext highlighter-rouge">code_verifier</code> and a <code class="language-plaintext highlighter-rouge">code_challenge</code> additionally to the <code class="language-plaintext highlighter-rouge">state</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">state</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">base64</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span>
<span class="n">code_verifier</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">base64</span><span class="p">(</span><span class="mi">64</span><span class="p">).</span><span class="nf">tr</span><span class="p">(</span><span class="s1">'+/'</span><span class="p">,</span> <span class="s1">'-_'</span><span class="p">).</span><span class="nf">tr</span><span class="p">(</span><span class="s1">'='</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="n">code_challenge</span> <span class="o">=</span> <span class="no">Digest</span><span class="o">::</span><span class="no">SHA2</span><span class="p">.</span><span class="nf">base64digest</span><span class="p">(</span><span class="n">code_verifier</span><span class="p">).</span><span class="nf">tr</span><span class="p">(</span><span class="s1">'+/'</span><span class="p">,</span> <span class="s1">'-_'</span><span class="p">).</span><span class="nf">tr</span><span class="p">(</span><span class="s1">'='</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
</code></pre></div></div>
<p>We can then start the server in a background thread.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">server</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">report_on_exception</span> <span class="o">=</span> <span class="kp">false</span>
<span class="no">Server</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">state</span><span class="p">).</span><span class="nf">start</span>
<span class="k">end</span>
</code></pre></div></div>
<p>We can use <code class="language-plaintext highlighter-rouge">state</code> and <code class="language-plaintext highlighter-rouge">code_challenge</code> to initialize the OAuth flow. Note that we are using the <code class="language-plaintext highlighter-rouge">code</code> response type and the <code class="language-plaintext highlighter-rouge">S256</code> code challenge method.</p>
<p>We’ll use <a href="https://github.com/copiousfreetime/launchy">Launchy</a> to open the browser window, and after that is done, we wait for the local server to receive the callback.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">response_type: </span><span class="s1">'code'</span><span class="p">,</span>
<span class="ss">code_challenge_method: </span><span class="s1">'S256'</span><span class="p">,</span>
<span class="ss">code_challenge: </span><span class="n">code_challenge</span><span class="p">,</span>
<span class="ss">client_id: </span><span class="s1">'591376582274-ctrjhsj8fjjhn4pk1rknfvcfhrcc3af7.apps.googleusercontent.com'</span><span class="p">,</span>
<span class="ss">redirect_uri: </span><span class="s1">'http://localhost:9876/authorize'</span><span class="p">,</span>
<span class="ss">scope: </span><span class="s1">'https://www.googleapis.com/auth/userinfo.profile'</span><span class="p">,</span>
<span class="ss">state: </span><span class="n">state</span><span class="p">,</span>
<span class="ss">access_type: </span><span class="s1">'offline'</span>
<span class="p">}.</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="s2">"</span><span class="si">#{</span><span class="n">x</span><span class="si">}</span><span class="s2">=</span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="p">}.</span><span class="nf">reduce</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="s2">"</span><span class="si">#{</span><span class="n">x</span><span class="si">}</span><span class="s2">&</span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="no">Launchy</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"https://accounts.google.com/o/oauth2/v2/auth?</span><span class="si">#{</span><span class="n">params</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">exception</span><span class="o">|</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s2">"Attempted to open </span><span class="si">#{</span><span class="n">uri</span><span class="si">}</span><span class="s2"> and failed because </span><span class="si">#{</span><span class="n">exception</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">server</span><span class="p">.</span><span class="nf">join</span>
</code></pre></div></div>
<p>Once we have received the authorization code, we contact the authorization server to exchange it for an authorization token.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">code</span> <span class="o">=</span> <span class="n">server</span><span class="p">.</span><span class="nf">value</span>
<span class="n">uri</span> <span class="o">=</span> <span class="no">URI</span><span class="p">(</span><span class="s1">'https://oauth2.googleapis.com/token'</span><span class="p">)</span>
<span class="n">http</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">uri</span><span class="p">.</span><span class="nf">host</span><span class="p">,</span> <span class="n">uri</span><span class="p">.</span><span class="nf">port</span><span class="p">)</span>
<span class="n">http</span><span class="p">.</span><span class="nf">use_ssl</span> <span class="o">=</span> <span class="kp">true</span>
<span class="n">http</span><span class="p">.</span><span class="nf">verify_mode</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">SSL</span><span class="o">::</span><span class="no">VERIFY_NONE</span>
<span class="n">request</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Post</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">uri</span><span class="p">)</span>
<span class="n">request</span><span class="p">[</span><span class="s1">'content-type'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'application/x-www-form-urlencoded'</span>
<span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">grant_type: </span><span class="s1">'authorization_code'</span><span class="p">,</span>
<span class="ss">code_verifier: </span><span class="n">code_verifier</span><span class="p">,</span>
<span class="ss">code: </span><span class="n">code</span><span class="p">,</span>
<span class="ss">client_id: </span><span class="s1">'591376582274-ctrjhsj8fjjhn4pk1rknfvcfhrcc3af7.apps.googleusercontent.com'</span><span class="p">,</span>
<span class="ss">client_secret: </span><span class="s1">'cZAXyEkeV9kZNmDQyZsNLHaj'</span><span class="p">,</span>
<span class="ss">redirect_uri: </span><span class="s1">'http://localhost:9876/authorize'</span>
<span class="p">}.</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="s2">"</span><span class="si">#{</span><span class="n">x</span><span class="si">}</span><span class="s2">=</span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="p">}.</span><span class="nf">reduce</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="p">,</span> <span class="n">v</span><span class="o">|</span> <span class="s2">"</span><span class="si">#{</span><span class="n">x</span><span class="si">}</span><span class="s2">&</span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="n">request</span><span class="p">.</span><span class="nf">body</span> <span class="o">=</span> <span class="n">params</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">http</span><span class="p">.</span><span class="nf">request</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s2">"Invalid token response, got </span><span class="si">#{</span><span class="n">response</span><span class="p">.</span><span class="nf">code</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="k">unless</span> <span class="n">response</span><span class="p">.</span><span class="nf">code</span> <span class="o">==</span> <span class="s1">'200'</span>
</code></pre></div></div>
<p>If all goes well we should receive an access token along with additional data - which we’ll ignore for now to keep things simple :grin:</p>
<p class="notice--info">As you probably know, authorization tokens issued via OAuth expire after some time. The lifetime of the authorization token is part of that ‘additional data’, and would normally be used to have the user reauthorize your application.</p>
<h2 id="performing-authorized-requests">Performing Authorized Requests</h2>
<p>Now we’ll use the token we just received in our <code class="language-plaintext highlighter-rouge">user</code> command. We simply dump it in a file at the end of the <code class="language-plaintext highlighter-rouge">login</code> command and retrieve it when we need it. This is not the <a href="https://medium.com/@calavera/stop-saving-credential-tokens-in-text-files-65e840a237bb">right way to store credentials</a> but it will do for now.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">data</span> <span class="o">=</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="nf">body</span><span class="p">)</span>
<span class="n">path</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="no">Dir</span><span class="p">.</span><span class="nf">home</span><span class="p">,</span> <span class="s1">'.googleme'</span><span class="p">)</span>
<span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">f</span><span class="o">|</span> <span class="n">f</span><span class="p">.</span><span class="nf">write</span> <span class="n">data</span><span class="p">.</span><span class="nf">to_json</span> <span class="p">}</span>
</code></pre></div></div>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">path</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="no">Dir</span><span class="p">.</span><span class="nf">home</span><span class="p">,</span> <span class="s1">'.googleme'</span><span class="p">)</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s1">'No access token found, please login first'</span><span class="p">)</span> <span class="k">unless</span> <span class="no">File</span><span class="p">.</span><span class="nf">file?</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">path</span><span class="p">))</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s1">'No access token found, please login first'</span><span class="p">)</span> <span class="k">unless</span> <span class="n">data</span>
</code></pre></div></div>
<p>Now the only thing that remains is to retrieve user information:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">access_token</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="s1">'access_token'</span><span class="p">]</span>
<span class="n">uri</span> <span class="o">=</span> <span class="no">URI</span><span class="p">(</span><span class="s1">'https://www.googleapis.com/oauth2/v1/userinfo?alt=json'</span><span class="p">)</span>
<span class="n">http</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">uri</span><span class="p">.</span><span class="nf">host</span><span class="p">,</span> <span class="n">uri</span><span class="p">.</span><span class="nf">port</span><span class="p">)</span>
<span class="n">http</span><span class="p">.</span><span class="nf">use_ssl</span> <span class="o">=</span> <span class="kp">true</span>
<span class="n">http</span><span class="p">.</span><span class="nf">verify_mode</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">SSL</span><span class="o">::</span><span class="no">VERIFY_NONE</span>
<span class="n">request</span> <span class="o">=</span> <span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="o">::</span><span class="no">Get</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">uri</span><span class="p">)</span>
<span class="n">request</span><span class="p">[</span><span class="s1">'Authorization'</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"Bearer </span><span class="si">#{</span><span class="n">access_token</span><span class="si">}</span><span class="s2">"</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">http</span><span class="p">.</span><span class="nf">request</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="k">raise</span><span class="p">(</span><span class="no">Error</span><span class="p">,</span> <span class="s2">"Invalid token response, got </span><span class="si">#{</span><span class="n">response</span><span class="p">.</span><span class="nf">code</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="k">unless</span> <span class="n">response</span><span class="p">.</span><span class="nf">code</span> <span class="o">==</span> <span class="s1">'200'</span>
<span class="nb">puts</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="nf">body</span><span class="p">)</span>
</code></pre></div></div>
<p>And that’s it! Running this little demo should now give you the user data of the authorized user.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Login first</span>
<span class="nv">$ </span>googleme login
<span class="c"># Show me the profile info!</span>
<span class="nv">$ </span>googleme user
<span class="o">{</span>
<span class="s2">"id"</span> <span class="o">=></span> <span class="s2">"123455"</span>,
<span class="s2">"name"</span> <span class="o">=></span> <span class="s2">"Hans Schnedlitz"</span>,
<span class="s2">"given_name"</span> <span class="o">=></span> <span class="s2">"Hans"</span>,
<span class="s2">"family_name"</span> <span class="o">=></span> <span class="s2">"Schnedlitz"</span>,
<span class="s2">"picture"</span> <span class="o">=></span> <span class="s2">"https://lh3.googleusercontent.com/a-/xyz"</span>,
<span class="s2">"locale"</span> <span class="o">=></span> <span class="s2">"en"</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>This was a fun little exercise that needed way more research than I expected. I learned a thing or two about OAuth that I didn’t know before, and I hope you did too while reading this. As mentioned before, the implementation is a very rough prototype and there are a bunch of things that can be improved.</p>
<p>Taking care of token expiry and re-authentication for one. You also should not store credentials the way I did, but rather take advantage of secure vaults that your operating system provides. And last but not least, this prototype implementation’s error handling is practically non-existent, so that should probably be changed:sweat_smile:</p>
<p>That being said, I’m still pretty happy with the result and am looking forward to using this in the future.</p>Hans SchnedlitzHave you ever used a command-line application that triggered an OAuth authentication flow to log you in and wondered how that works? For example, Google Cloud SDK does this, as does Heroku CLI.Use GitHub Actions to find Outdated Dependencies2021-02-10T21:00:00+00:002021-02-10T21:00:00+00:00https://hansschnedlitz.com/2021/02/10/using-github-actions-to-detect-outdated-dependencies<p>Keeping up with dependencies can be a pain. That is especially true if you build a tool that heavily relies on some library. If that library changes in a major way, you’ll have to be quick with updating or risk issues piling up.</p>
<p>But how can you efficiently keep track of dependency updates?</p>
<h2 id="scheduled-github-actions">Scheduled GitHub Actions</h2>
<p>I’ve found GitHub Actions to be a simple, yet effective, solution for that particular problem.</p>
<p>A small workflow that runs on a schedule and checks if there are outdated dependencies does the job. I use something like this for <a href="https://github.com/hschne/reveal.js-starter">reveal.js-starter</a> to get notified when a new <a href="https://revealjs.com/">reveal.js</a> version is released:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Check Outdated</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">schedule</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">cron</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0</span><span class="nv"> </span><span class="s">12</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">1"</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">check_updates</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout repository</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install npm</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm install</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check outdated</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm outdated reveal.js</span>
</code></pre></div></div>
<p>If you are thinking “Well, that doesn’t look hard” - you are right. And isn’t that nice? :wink:</p>
<p>There isn’t a lot for me to cover here, but let’s go over the interesting parts real quick.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">on</span><span class="pi">:</span>
<span class="na">schedule</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">cron</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0</span><span class="nv"> </span><span class="s">12</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">1"</span>
</code></pre></div></div>
<p>Here we specify when our action should run. Consult the <a href="https://docs.github.com/en/actions">official documentation</a> for details, but I’ve found checking dependencies each Monday at noon to work quite nicely.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check outdated</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">npm outdated reveal.js</span>
</code></pre></div></div>
<p>Luckily, <code class="language-plaintext highlighter-rouge">npm</code> offers a simple command to check if any dependency is outdated (based on your current lock file). The nice thing here is that this command will return with exit code <code class="language-plaintext highlighter-rouge">1</code> if newer versions were detected. As a result, the GitHub Action will fail without us needing to do anything else. Easy!</p>
<p><img src="https://i.kym-cdn.com/photos/images/original/000/875/422/11f.gif" alt="Chefs Kiss" class="align-center" /></p>
<p>The same approach works quite well for Ruby-based applications since <a href="https://bundler.io/">bundler</a> offers an <code class="language-plaintext highlighter-rouge">outdated</code> command as well. I use this workflow for my custom Rails generator <a href="https://github.com/hschne/schienenzeppelin">Schienenzeppelin</a>:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Check Outdated</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">schedule</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">cron</span><span class="pi">:</span> <span class="s2">"</span><span class="s">0</span><span class="nv"> </span><span class="s">12</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">*</span><span class="nv"> </span><span class="s">1"</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">check_updates</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout repository</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up ruby</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@latest</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">ruby-version</span><span class="pi">:</span> <span class="s">3.0.0</span>
<span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check Outdated</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">bundle config unset deployment</span>
<span class="s">bundle outdated rails</span>
</code></pre></div></div>
<p>Now, I’ve kept these actions very simple. Of course, there are tons of things you can improve upon! Customizing how notifications are sent or only failing when a new major version is released are things that come to mind.</p>
<p>What do you think? Let me know if you’ve encountered other interesting uses for GitHub actions :hugs:</p>Hans SchnedlitzKeeping up with dependencies can be a pain. That is especially true if you build a tool that heavily relies on some library. If that library changes in a major way, you’ll have to be quick with updating or risk issues piling up.Pull Request Previews with Rails, Cloud Run, and GitHub Actions2021-01-18T19:38:00+00:002021-01-18T19:38:00+00:00https://hansschnedlitz.com/2021/01/18/pull-request-previews-with-rails-cloud-run-and-github-actions<p>If you work in a larger organization, chances are you have access to some sort of staging environment where you can preview work that is currently in progress. Staging environments can be immensely useful to get early feedback, be it from QA or other departments, which can speed up the development of new features significantly.</p>
<p>Several large platforms such as <a href="https://docs.netlify.com/site-deploys/overview/">Netlify</a>, <a href="https://vercel.com/">Vercel</a>, or <a href="https://www.serverless.com/">Serverless</a> offer preview deploys of one sort or another. There are also paid services such as <a href="https://pullpreview.com/">Pullpreview</a>, which runs on AWS and allows one-click deploys for GitHub pull requests.</p>
<p>We are going to be building something similar, but for a simple Ruby on Rails application, and using <a href="https://cloud.google.com/run/">Google Cloud Run</a>. We are going to use <a href="https://github.com/features/actions">GitHub Actions</a> to do the actual deployment and continuous integration part. In the end, we want a solution that can accomplish the following:</p>
<ul>
<li>An isolated preview environment is created for each pull request. That environment is easily accessible using a direct link.</li>
<li>On pushing changes to the pull request, the preview should automatically update.</li>
<li>The preview environment should be fast to set up and tear down. No manual intervention should be necessary at any point.</li>
</ul>
<p>To showcase how we can build such a solution we’ll do the following:</p>
<ol>
<li>Create a simple Rails application for demo purposes.</li>
<li>Build an application Docker image.</li>
<li>Create a Cloud Run Service that runs our application container.</li>
<li>Set up a GitHub action that automatically builds our application container and deploys it to Google Cloud Run</li>
</ol>
<p class="notice--info">This is a bit of a longer post, so if you are more of a TL;DR type head over to <a href="https://github.com/hschne/rails-gcloud-bookstore">this repository</a> to check out the complete solution.</p>
<p class="notice--warning">This guide is intended for people who are already somewhat experienced with Rails, Docker and the Google Cloud SDK. I reference documentation where possible, but you will get the most out of this guide if you have a basic understanding of the involved technologies already. This is not a tutorial on how to work with Ruby on Rails or Docker or Google Cloud.</p>
<h2 id="application-setup">Application Setup</h2>
<p>While this guide focuses on the whole deploying-a-pull-request part, we are going to need a simple application that we <em>can</em> actually deploy. We can keep it simple. How about an application that allows you to store some books? Perfect :ok_hand:</p>
<p>Make sure you have a recent version of <a href="https://www.ruby-lang.org/en/">Ruby</a> and <a href="https://yarnpkg.com/">Yarn</a> installed, then create the bookstore application. We’ll keep it minimal here, all we want is to store some books.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails new bookstore <span class="se">\</span>
<span class="nt">--database</span><span class="o">=</span>postgresql <span class="se">\</span>
<span class="nt">--webpack</span><span class="se">\</span>
<span class="nt">--skip-active-storage</span> <span class="se">\</span>
<span class="nt">--skip-action-cable</span> <span class="se">\</span>
<span class="nt">--skip-system-test</span> <span class="se">\</span>
<span class="nt">--skip-action-text</span> <span class="se">\</span>
<span class="nt">--skip-action-mailer</span>
</code></pre></div></div>
<p>To make it easier for us to configure our application down the line, let’s also add the <a href="https://github.com/bkeepers/dotenv">dotenv</a> gem to our Gemfile and run <code class="language-plaintext highlighter-rouge">bundle install</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'dotenv-rails'</span>
</code></pre></div></div>
<p>Because we will deploy our application on Cloud Run, we’ll have to dockerize it sooner or later. To make it easier for us to test our process locally - trust me, this is quite handy - we’ll create a docker-compose file for our application and it’s dependencies. Once you have <a href="https://www.docker.com/">Docker</a> and <a href="https://docs.docker.com/compose/">Docker Compose</a> set up, let’s create our <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> and add Postgres as a dependency.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3"</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">bookstore-db</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">postgres</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">db-data:/var/lib/postgresql/data</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">5432:5432</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">postgres_password</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">postgres_user</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="na">db-data</span><span class="pi">:</span> <span class="s">~</span>
</code></pre></div></div>
<p>Then start the database container.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose up <span class="nt">-d</span>
</code></pre></div></div>
<p>So far so good. Our bookstore doesn’t do anything yet. We’ll want to create, edit and delete books. But first, we have to make sure the application can connect to our Database. Open <code class="language-plaintext highlighter-rouge">config/database.yml</code> and update it with the following contents:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">default</span><span class="pi">:</span> <span class="nl">&default</span>
<span class="na">adapter</span><span class="pi">:</span> <span class="s">postgresql</span>
<span class="na">encoding</span><span class="pi">:</span> <span class="s">unicode</span>
<span class="na">host</span><span class="pi">:</span> <span class="s"><%= ENV['DATABASE_HOST'] %></span>
<span class="na">username</span><span class="pi">:</span> <span class="s"><%= ENV['DATABASE_USERNAME'] %></span>
<span class="na">password</span><span class="pi">:</span> <span class="s"><%= ENV['DATABASE_PASSWORD'] %></span>
<span class="na">pool</span><span class="pi">:</span> <span class="s"><%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %></span>
<span class="na">development</span><span class="pi">:</span>
<span class="s"><<</span><span class="pi">:</span> <span class="nv">*default</span>
<span class="na">database</span><span class="pi">:</span> <span class="s">bookstore_development</span>
<span class="na">test</span><span class="pi">:</span>
<span class="s"><<</span><span class="pi">:</span> <span class="nv">*default</span>
<span class="na">database</span><span class="pi">:</span> <span class="s">bookstore_test</span>
<span class="na">production</span><span class="pi">:</span>
<span class="s"><<</span><span class="pi">:</span> <span class="nv">*default</span>
<span class="na">database</span><span class="pi">:</span> <span class="s2">"</span><span class="s">bookstore_production"</span>
</code></pre></div></div>
<p>Next, create a <code class="language-plaintext highlighter-rouge">.env</code> file for <code class="language-plaintext highlighter-rouge">dotenv</code> to load and fill it with credentials matching what is in your <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>. For now, let’s have the database be located at localhost, but we’ll change that soon.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">DATABASE_HOST</span><span class="o">=</span>localhost
<span class="nv">DATABASE_USERNAME</span><span class="o">=</span>bookstore
<span class="nv">DATABASE_PASSWORD</span><span class="o">=</span>bookstore
</code></pre></div></div>
<p>If we did well so far we should be able to set up our database and start the server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails db:setup
rails server
</code></pre></div></div>
<p>You should be able to see the application running on <a href="http://localhost:3000">http://localhost:3000</a>. Next, let’s create some books, migrate our data and restart the rails server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails g scaffold books title:string author:string publication_year:integer
rails db:migrate
rails server
</code></pre></div></div>
<p>We can now add some books on <a href="http://localhost:3000/books">http://localhost:3000/books</a>. As a final tweak, let’s update the <code class="language-plaintext highlighter-rouge">config/routes.rb</code> file so we can save ourselves from having to append <code class="language-plaintext highlighter-rouge">/books</code> to our URLs in the future.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">routes</span><span class="p">.</span><span class="nf">draw</span> <span class="k">do</span>
<span class="n">root</span> <span class="s1">'books#index'</span>
<span class="n">resources</span> <span class="ss">:books</span>
<span class="k">end</span>
</code></pre></div></div>
<p>That’s good enough for a basic app I’d say. Let’s get started on dockerizing it.</p>
<h2 id="dockerizing-the-bookstore">Dockerizing the Bookstore</h2>
<p>We’ll be running our application in production mode within its container, so let’s prepare for that by setting up a secret key base for production.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails secret
rails credentials:edit <span class="nt">--environment</span> production
</code></pre></div></div>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">secret_key_base</span><span class="pi">:</span> <span class="s"><your-secret-goes-here></span>
</code></pre></div></div>
<p class="notice--warning">When editing your credentials for the first time take note of the generated master key. You can always check <code class="language-plaintext highlighter-rouge">config/credentials/production.key</code> later though. Do <em>not</em> commit this file to your version control.</p>
<p>To create a Docker image for an application, you need a <code class="language-plaintext highlighter-rouge">Dockerfile</code>. So let’s create that. Here’s one to get you started:</p>
<div class="language-docker highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> ruby:3.0-slim as cache</span>
<span class="k">RUN </span>apt-get update <span class="nt">-qq</span> <span class="o">&&</span> apt-get <span class="nb">install</span> <span class="nt">-y</span> <span class="se">\
</span> curl <span class="se">\
</span> build-essential <span class="se">\
</span> libpq-dev <span class="se">\
</span> postgresql-client
<span class="k">RUN </span>curl <span class="nt">-sL</span> https://deb.nodesource.com/setup_14.x | bash -
<span class="k">RUN </span>curl <span class="nt">-sL</span> https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - <span class="o">&&</span> <span class="se">\
</span> <span class="nb">echo</span> <span class="s2">"deb https://dl.yarnpkg.com/debian/ stable main"</span> | <span class="nb">tee</span> /etc/apt/sources.list.d/yarn.list
<span class="k">RUN </span>apt-get update <span class="nt">-qq</span> <span class="o">&&</span> apt-get <span class="nb">install</span> <span class="nt">-y</span> yarn
<span class="k">WORKDIR</span><span class="s"> /bookstore</span>
<span class="k">COPY</span><span class="s"> Gemfile /bookstore/Gemfile</span>
<span class="k">COPY</span><span class="s"> Gemfile.lock /bookstore/Gemfile.lock</span>
<span class="k">COPY</span><span class="s"> package.json /bookstore/package.json</span>
<span class="k">COPY</span><span class="s"> yarn.lock /bookstore/yarn.lock</span>
<span class="k">RUN </span>bundle <span class="nb">install</span> <span class="nt">--without</span> development
<span class="k">RUN </span>yarn <span class="nb">install</span>
<span class="k">FROM</span><span class="s"> cache</span>
<span class="k">COPY</span><span class="s"> . /bookstore</span>
<span class="k">ARG</span><span class="s"> DATABASE_HOST=db</span>
<span class="k">ARG</span><span class="s"> RAILS_ENV=production</span>
<span class="k">ENV</span><span class="s"> DATABASE_HOST=$DATABASE_HOST</span>
<span class="k">ENV</span><span class="s"> RAILS_ENV=$RAILS_ENV</span>
<span class="k">ENV</span><span class="s"> RAILS_LOG_TO_STDOUT=true</span>
<span class="k">ENV</span><span class="s"> RAILS_SERVE_STATIC_FILES=true</span>
<span class="k">RUN </span>bundle <span class="nb">exec </span>rails assets:precompile
<span class="k">COPY</span><span class="s"> entrypoint.sh /usr/bin/</span>
<span class="k">RUN </span><span class="nb">chmod</span> +x /usr/bin/entrypoint.sh
<span class="k">ENTRYPOINT</span><span class="s"> ["entrypoint.sh"]</span>
<span class="k">EXPOSE</span><span class="s"> 3000</span>
<span class="c"># Start the main process.</span>
<span class="k">CMD</span><span class="s"> ["rails", "server", "-b", "0.0.0.0"]</span>
</code></pre></div></div>
<p>Adjust this to suit your needs. We set some particular environment variables.</p>
<div class="language-docker highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RAILS_LOG_TO_STDOUT=true
RAILS_SERVE_STATIC_FILES=true
</code></pre></div></div>
<p>Most likely you would not set these in a ‘real’ application, but for demo purposes, letting Rails serve static assets simplifies the deployment process a lot, and logging to STDOUT makes debugging a lot easier when something goes awry.</p>
<p>You may have noticed that the Dockerfile references an <code class="language-plaintext highlighter-rouge">entrypoint.sh</code> file, so let’s create that as well.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nb">set</span> <span class="nt">-e</span>
<span class="c"># Create the Rails production DB on the first run</span>
bundle <span class="nb">exec </span>rails db:create
<span class="c"># Make sure we are using the most up to date</span>
<span class="c"># database schema</span>
bundle <span class="nb">exec </span>rails db:migrate
<span class="c"># Remove a potentially pre-existing server.pid for Rails.</span>
<span class="nb">rm</span> <span class="nt">-f</span> /bookstore/tmp/pids/server.pid
<span class="c"># Then exec the container's main process (what's set as CMD in the Dockerfile).</span>
<span class="nb">exec</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</code></pre></div></div>
<p>It is also a good idea to create a <code class="language-plaintext highlighter-rouge">.dockerignore</code> file - <a href="https://gist.github.com/neckhair/ace5d1679dd896b71403fda4bc217b9e">here</a> is a sample - although this is not strictly necessary. With the Dockerfile complete, we should be able to build our bookstore image.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build <span class="nt">-t</span> bookstore <span class="nb">.</span>
</code></pre></div></div>
<p>If you try to run your container and access the bookstore application now, you’ll notice some errors due to the database container no longer being accessible.</p>
<p>To fix that while keeping things simple, let’s just move our bookstore image to the <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> as well.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3"</span>
<span class="na">services</span><span class="pi">:</span>
<span class="s">...</span>
<span class="s">bookstore</span><span class="pi">:</span>
<span class="na">build</span><span class="pi">:</span> <span class="s">.</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">gcr.io/rails-gcloud-bookstore/bookstore</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">3000:3000</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">DATABASE_HOST</span><span class="pi">:</span> <span class="s">bookstore-db</span>
<span class="na">DATABASE_USERNAME</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">DATABASE_PASSWORD</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">depends_on</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">bookstore-db</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="na">db-data</span><span class="pi">:</span> <span class="s">~</span>
</code></pre></div></div>
<p class="notice--warning">Note the image name, as you’ll have to adjust it later. It has to match the name of your Google Cloud Project, which we’ll create when we push our image to the <a href="https://cloud.google.com/container-registry">Google Cloud Registry</a>.</p>
<p>Running <code class="language-plaintext highlighter-rouge">docker-compose up</code> now should build the bookstore application image and start the container. A production version of your application should now be accessible under <a href="localhost:3000">http://localhost:3000</a>.</p>
<p class="notice--info">If your container won’t start it can be helpful to connect to it using <code class="language-plaintext highlighter-rouge">docker-compose run --entrypoint /bin/bash bookstore</code> to verify the contents of environment variables or the presences of files within the container.</p>
<h2 id="set-up-google-cloud">Set Up Google Cloud</h2>
<p>Let’s recap. We built a simple Rails application - our trusty bookstore - and made sure we can build a production-ready Docker image for it. Now let’s set up the necessary Google Cloud infrastructure.</p>
<p>First things first. Make sure you have installed the <a href="https://cloud.google.com/sdk/install">Google Cloud SDK</a>. Once you have it installed log in.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud auth login
</code></pre></div></div>
<p class="notice--info">This guide uses Google Cloud SDK for convenience reasons. Copy-pasting a bunch of commands is just a lot easier than reproducing what’s shown on some screenshot. That being said, you can accomplish everything described thereafter using the <a href="https://console.cloud.google.com/">Google Cloud Console</a>. Pick your poison.</p>
<p>Let’s set up a new <a href="https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project">Google Cloud Project</a>. I’ll name mine <code class="language-plaintext highlighter-rouge">rails-gcloud-bookstore</code>. You’ll likely have to pick a different one because project names must be unique. Then set the new project as default project.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud projects create rails-gcloud-bookstore
gcloud config <span class="nb">set </span>project rails-gcloud-bookstore
</code></pre></div></div>
<p>Next, we’ll enable the Container Registry because that is where we are going to push our Docker image.</p>
<p class="notice--warning">You’ll have to enable billing for your project, because storing containers is not free. Head over to <a href="https://console.cloud.google.com/billing">Billing</a> and set your billing account. You can find more information on billing in <a href="https://cloud.google.com/billing/docs/how-to/modify-project#enable_billing_for_an_existing_project">the official documentation</a>. Do not worry, following this guide will cost you a couple of cents at the most 😉</p>
<p>The Cloud SDK provides a helpful Docker utility that makes it easy to push local images to the Google Container Registry, so let’s enable that.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud services <span class="nb">enable </span>containerregistry.googleapis.com
gcloud components <span class="nb">install </span>docker-credential-gcr
gcloud auth configure-docker
</code></pre></div></div>
<p>Now you should be able to push your local image:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker push gcr.io/rails-gcloud-bookstore/bookstore
</code></pre></div></div>
<p>This might take a while. Afterward, you can check that your image was indeed pushed by either running <code class="language-plaintext highlighter-rouge">gcloud container images list</code> or verifying the contents of the container registry in the <a href="https://console.cloud.google.com">Cloud Console</a>.</p>
<p>Before we can run our application on Cloud Run, we’ll have to take care of setting up Postgres in the Cloud. We need a place to store our books after all.</p>
<p>Let’s set up a <a href="https://cloud.google.com/sql/">Cloud SQL</a> instance and configure it. You can pick any of the <a href="https://cloud.google.com/compute/docs/regions-zones">available regions</a>. For the remainder of the guide, I’m going with <code class="language-plaintext highlighter-rouge">europe-north1</code> wherever a region needs to be specified.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud sql instances create bookstore-db <span class="se">\</span>
<span class="nt">--root-password</span><span class="o">=</span><span class="s2">"bookstore"</span> <span class="se">\</span>
<span class="nt">--database-version</span><span class="o">=</span>POSTGRES_13 <span class="se">\</span>
<span class="nt">--region</span><span class="o">=</span>europe-north1 <span class="se">\</span>
<span class="nt">--tier</span><span class="o">=</span>db-f1-micro <span class="se">\</span>
<span class="nt">--no-backup</span>
gcloud sql <span class="nb">users </span>create bookstore <span class="nt">--instance</span><span class="o">=</span>bookstore-db <span class="nt">--password</span><span class="o">=</span><span class="s1">'bookstore'</span>
</code></pre></div></div>
<p class="notice--info">Cloud SQL instances are not publicly accessible without additional configuration. As such, there is no need to worry too much about picking a password for demo purposes. For production uses you should still pick a strong password.</p>
<p>Almost everything we need is in place. Let’s enable Cloud Run.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud services <span class="nb">enable </span>run.googleapis.com
</code></pre></div></div>
<p>To make sure that your application can connect to the Cloud SQL instance we will need to set the necessary permissions. List all available service accounts and give the <code class="language-plaintext highlighter-rouge">compute</code> service account the Cloud SQL Client role. Google Cloud automatically uses this service account when running containers in Cloud Run.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud iam service-accounts list
gcloud projects add-iam-policy-binding rails-gcloud-bookstore <span class="se">\</span>
<span class="nt">--member</span> serviceAccount:<projectid>-compute@developer.gserviceaccount.com <span class="se">\</span>
<span class="nt">--role</span> roles/cloudsql.client
</code></pre></div></div>
<p>Now, let’s deploy the bookstore app.</p>
<p>Make sure that the environment variables match what you have configured previously. In particular, the database host variable should contain the SQL connection name, and the production key should match the key you got when setting up production credentials for the bookstore app previously. To find the SQL connection name you can run <code class="language-plaintext highlighter-rouge">gcloud sql instances describe bookstore-db</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud run deploy bookstore-preview <span class="se">\</span>
<span class="nt">--image</span><span class="o">=</span>gcr.io/rails-gcloud-bookstore/bookstore <span class="se">\</span>
<span class="nt">--platform</span><span class="o">=</span>managed <span class="se">\</span>
<span class="nt">--port</span><span class="o">=</span>3000 <span class="se">\</span>
<span class="nt">--add-cloudsql-instances</span><span class="o">=</span>rails-gcloud-bookstore:europe-north1:bookstore-db <span class="se">\</span>
<span class="nt">--set-env-vars</span> <span class="nv">DATABASE_HOST</span><span class="o">=</span>/cloudsql/rails-gcloud-bookstore:europe-north1:bookstore-db <span class="se">\</span>
<span class="nt">--set-env-vars</span> <span class="nv">DATABASE_USERNAME</span><span class="o">=</span><span class="s1">'bookstore'</span> <span class="se">\</span>
<span class="nt">--set-env-vars</span> <span class="nv">DATABASE_PASSWORD</span><span class="o">=</span><span class="s1">'bookstore'</span> <span class="se">\</span>
<span class="nt">--set-env-vars</span> <span class="nv">RAILS_PRODUCTION_KEY</span><span class="o">=</span><span class="s2">"your-production-master-key"</span> <span class="se">\</span>
<span class="nt">--region</span><span class="o">=</span>europe-north1 <span class="se">\</span>
<span class="nt">--allow-unauthenticated</span>
</code></pre></div></div>
<p>If everything was successful running the deploy command should give you a link that you can visit to view your application running on Cloud Run :ok_hand:</p>
<h2 id="creating-the-github-action">Creating the GitHub Action</h2>
<p>We are almost there. Before we create our GitHub action that automates the deployment process we’ll need to create a Google Cloud service account that has the necessary roles to:</p>
<ul>
<li>Push our image to the Container Registry</li>
<li>Deploy the image to Cloud Run</li>
</ul>
<p>Run the following commands to create a service account and give it the permissions required.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud iam service-accounts create <span class="se">\</span>
bookstore-build <span class="se">\</span>
<span class="nt">--display-name</span><span class="o">=</span><span class="s2">"Bookstore Builder"</span>
gcloud projects add-iam-policy-binding rails-gcloud-bookstore <span class="se">\</span>
<span class="nt">--member</span> serviceAccount:bookstore-build@rails-gcloud-bookstore.iam.gserviceaccount.com <span class="se">\</span>
<span class="nt">--role</span> roles/run.admin
gcloud projects add-iam-policy-binding rails-gcloud-bookstore <span class="se">\</span>
<span class="nt">--member</span> serviceAccount:bookstore-build@rails-gcloud-bookstore.iam.gserviceaccount.com <span class="se">\</span>
<span class="nt">--role</span> roles/storage.admin
gcloud projects add-iam-policy-binding rails-gcloud-bookstore <span class="se">\</span>
<span class="nt">--member</span> serviceAccount:bookstore-build@rails-gcloud-bookstore.iam.gserviceaccount.com <span class="se">\</span>
<span class="nt">--role</span> roles/iam.serviceAccountUser
</code></pre></div></div>
<p>Verify that all roles are set correctly either by running the following command or by checking the Cloud Console IAM section.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> gcloud projects get-iam-policy rails-gcloud-bookstore <span class="se">\</span>
<span class="nt">--flatten</span><span class="o">=</span><span class="s2">"bindings[].members"</span> <span class="se">\</span>
<span class="nt">--format</span><span class="o">=</span><span class="s1">'table(bindings.role)'</span> <span class="se">\</span>
<span class="nt">--filter</span><span class="o">=</span><span class="s2">"bindings.members:bookstore-build@rails-gcloud-bookstore.iam.gserviceaccount.com"</span>
</code></pre></div></div>
<p>To use this service account in the GitHub action download the service account key.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud iam service-accounts keys create bookstore-build.json <span class="se">\</span>
<span class="nt">--iam-account</span> bookstore-build@rails-gcloud-bookstore.iam.gserviceaccount.com
</code></pre></div></div>
<p class="notice--warning">Make sure to not commit this file, as to not give other persons access to you Google Cloud Resources. If you did so by accident you can always remove the compromised key and recreate a new one on the <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">service accounts page</a>.</p>
<p>If you haven’t already <a href="https://github.com/new">create a new GitHub repository</a>.</p>
<p>Copy the contents of the service account key file (yes, the entire content) and add them to your <a href="https://docs.github.com/en/actions/reference/encrypted-secrets">GitHub secrets </a> as <code class="language-plaintext highlighter-rouge">GCP_SA_KEY</code>. While you are at it, also add your <code class="language-plaintext highlighter-rouge">RAILS_PRODUCTION_KEY</code>, because we’ll need that as well. Your action secrets page should look something like this after you are done.</p>
<p><img src="https://hansschnedlitz.com/assets/images/posts/2021-01-18/secrets.png" alt="secrets" /></p>
<p>Now let’s create a new workflow file. If you haven’t worked with GitHub actions before, you can learn more about them <a href="https://docs.github.com/en/actions/learn-github-actions">here</a>.</p>
<p>You can create a GitHub action using the GitHub web interface, but we’ll use the command line as usual. In your bookstore repository create a new workflows folder.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">-p</span> .github/workflows
</code></pre></div></div>
<p>Create a workflow file in that new folder (I named mine <code class="language-plaintext highlighter-rouge">preview.yml</code>, but you can call it whatever you want), and let’s add the following content.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Preview</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">pull_request</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">main"</span>
<span class="na">types</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">synchronize</span><span class="pi">,</span> <span class="nv">opened</span><span class="pi">,</span> <span class="nv">reopened</span><span class="pi">]</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">tests</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">RAILS_ENV</span><span class="pi">:</span> <span class="s">test</span>
<span class="na">DATABASE_HOST</span><span class="pi">:</span> <span class="s">localhost</span>
<span class="na">DATABASE_USERNAME</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">DATABASE_PASSWORD</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">postgres</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">postgres:13</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">POSTGRES_PASSWORD</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">POSTGRES_USER</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">5432:5432</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Ruby</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">ruby-version</span><span class="pi">:</span> <span class="m">3.0</span>
<span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Node</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/setup-node@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">node-version</span><span class="pi">:</span> <span class="m">14.9</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install dependencies</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">sudo apt-get -yqq install libpq-dev build-essential libcurl4-openssl-dev</span>
<span class="s">gem install bundler</span>
<span class="s">bundle install --jobs 4</span>
<span class="s">yarn install</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup databases</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">bin/rails db:setup</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Run tests</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">bin/rails test</span>
<span class="na">container</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">needs</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">tests</span><span class="pi">]</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup Google Cloud SDK</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">google-github-actions/setup-gcloud@master</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">project_id</span><span class="pi">:</span> <span class="s">rails-gcloud-bookstore</span>
<span class="na">service_account_key</span><span class="pi">:</span> <span class="s">${{ secrets.GCP_SA_KEY }}</span>
<span class="na">export_default_credentials</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install Google Cloud Docker</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">gcloud components install docker-credential-gcr</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup Docker for GCR</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">gcloud auth configure-docker</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build Docker Image</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">echo ${{ secrets.RAILS_PRODUCTION_KEY }} > config/credentials/production.key </span>
<span class="s">docker build -t gcr.io/rails-gcloud-bookstore/bookstore:${{github.event.number}} .</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Push Docker Image</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">docker push gcr.io/rails-gcloud-bookstore/bookstore:${{github.event.number}}</span>
<span class="na">preview</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">needs</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">container</span><span class="pi">]</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup Google Cloud SDK</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">google-github-actions/setup-gcloud@master</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">project_id</span><span class="pi">:</span> <span class="s">rails-gcloud-bookstore</span>
<span class="na">service_account_key</span><span class="pi">:</span> <span class="s">${{ secrets.GCP_SA_KEY }}</span>
<span class="na">export_default_credentials</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Get HEAD Commit Hash</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">commit</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">echo "::set-output name=hash::$(git rev-parse --short HEAD)"</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy Revision On Cloud Run</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">gcloud run deploy bookstore-preview \</span>
<span class="s">--image=gcr.io/rails-gcloud-bookstore/bookstore:${{github.event.number}} \</span>
<span class="s">--platform=managed \</span>
<span class="s">--port=3000 \</span>
<span class="s">--revision-suffix=${{github.event.number}}-${{steps.commit.outputs.hash}} \</span>
<span class="s">--add-cloudsql-instances=rails-gcloud-bookstore:europe-north1:bookstore-db \</span>
<span class="s">--set-env-vars DATABASE_HOST=/cloudsql/rails-gcloud-bookstore:europe-north1:bookstore-db \</span>
<span class="s">--set-env-vars DATABASE_USERNAME='bookstore' \</span>
<span class="s">--set-env-vars DATABASE_PASSWORD='bookstore' \</span>
<span class="s">--set-env-vars DATABASE_NAME=bookstore_production_${{github.event.number}} \</span>
<span class="s">--set-env-vars RAILS_PRODUCTION_KEY=${{ secrets.RAILS_PRODUCTION_KEY }} \</span>
<span class="s">--allow-unauthenticated \</span>
<span class="s">--region=europe-north1</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Update Traffic</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">gcloud components install beta</span>
<span class="s">gcloud beta run services update-traffic bookstore-preview \</span>
<span class="s">--update-tags pr-${{github.event.number}}=bookstore-preview-${{github.event.number}}-${{steps.commit.outputs.hash}} \</span>
<span class="s">--platform=managed \</span>
<span class="s">--region=europe-north1</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Get Preview URL</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">preview-url</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">url=$(gcloud run services describe bookstore-preview --format 'value(status.url)' --platform=managed --region=europe-north1 | sed 's|https://|https://pr-${{github.event.number}}---|g')</span>
<span class="s">echo "::set-output name=url::$url"</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Post PR comment with preview deployment URL</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">mshick/add-pr-comment@v1</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">GITHUB_TOKEN</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_TOKEN }}</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">message</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">Successfully deployed preview at ${{steps.preview-url.outputs.url}}</span>
<span class="na">allow-repeats</span><span class="pi">:</span> <span class="no">false</span>
</code></pre></div></div>
<p>There’s a fair bit to unpack here, so let’s go over this step by step.</p>
<p>First, we need to define <em>when</em> our workflow should run. We want to deploy to our preview environment whenenver a pull request against our main branch is opened.</p>
<p>Furthermore, we want to do the same when a Pull Request is reopened or when new code is pushed to our pull request branch (which fires the <em>synchronize</em> event).</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">on</span><span class="pi">:</span>
<span class="na">pull_request</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">main"</span>
<span class="na">types</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">synchronize</span><span class="pi">,</span> <span class="nv">opened</span><span class="pi">,</span> <span class="nv">reopened</span><span class="pi">]</span>
</code></pre></div></div>
<p>To specify what our workflow should actually do we define several jobs. Testing the application before doing a deployment is not strictly necessary, but it doesn’t hurt either. You don’t want to deploy broken code, even if it is just to a preview environment :wink:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">tests</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">RAILS_ENV</span><span class="pi">:</span> <span class="s">test</span>
<span class="na">DATABASE_HOST</span><span class="pi">:</span> <span class="s">localhost</span>
<span class="na">DATABASE_USERNAME</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">DATABASE_PASSWORD</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">postgres</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">postgres:13</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">POSTGRES_PASSWORD</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">POSTGRES_USER</span><span class="pi">:</span> <span class="s">bookstore</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">5432:5432</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Ruby</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">ruby/setup-ruby@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">ruby-version</span><span class="pi">:</span> <span class="m">3.0</span>
<span class="na">bundler-cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Set up Node</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/setup-node@v1</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">node-version</span><span class="pi">:</span> <span class="m">14.9</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install dependencies</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">sudo apt-get -yqq install libpq-dev build-essential libcurl4-openssl-dev</span>
<span class="s">gem install bundler</span>
<span class="s">bundle install --jobs 4</span>
<span class="s">yarn install</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup databases</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">bin/rails db:setup</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Run tests</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">bin/rails test</span>
</code></pre></div></div>
<p>We can then repeat the steps which we already performed locally - but with some twists. Our container build job depends on our test job, as we don’t want to run this step if the tests failed. We set up Google Cloud and authenticate with the service account we previously created, then build our image and push it to the Container Registry.</p>
<p>Note that we inject the production key into our image. We also tag the created image with the pull request number - this is important so we can deploy multiple pull requests side by side.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">container</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">needs</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">tests</span><span class="pi">]</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup Google Cloud SDK</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">google-github-actions/setup-gcloud@master</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">project_id</span><span class="pi">:</span> <span class="s">rails-gcloud-bookstore</span>
<span class="na">service_account_key</span><span class="pi">:</span> <span class="s">${{ secrets.GCP_SA_KEY }}</span>
<span class="na">export_default_credentials</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install Google Cloud Docker</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">gcloud components install docker-credential-gcr</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup Docker for GCR</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">gcloud auth configure-docker</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build Docker Image</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">echo ${{ secrets.RAILS_PRODUCTION_KEY }} > config/credentials/production.key </span>
<span class="s">docker build -t gcr.io/rails-gcloud-bookstore/bookstore:${{github.event.number}} .</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Push Docker Image</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">docker push gcr.io/rails-gcloud-bookstore/bookstore:${{github.event.number}}</span>
</code></pre></div></div>
<p>Now we need to deploy our newly-created docker image. We won’t create a new service for each of our pull requests - instead, we’ll use <em>revisions</em>. Old revisions of Cloud Run services are automatically removed, which simplifies the clean up we’ll have to do when the pull request is closed.</p>
<p>However, we need some way to distinguish revisions and route traffic for the pull request to the most recent one - we’ll use the commit hash to identify those.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Get HEAD Commit Hash</span>
<span class="s">id</span><span class="pi">:</span> <span class="s">commit</span>
<span class="s">run</span><span class="pi">:</span> <span class="s">echo "::set-output name=hash::$(git rev-parse --short HEAD)"</span>
</code></pre></div></div>
<p>Then we deploy the new revision. This command is similar to the one we used locally, with some minor changes.</p>
<ul>
<li>We use the image tagged with the pull request number</li>
<li>We define a revision using <code class="language-plaintext highlighter-rouge">--revision-suffix</code> that contains the commit hash</li>
<li>We set the database name to include the PR number, so that pull requests use independent databases.</li>
<li>We set the production master key using the previously created secret</li>
<li>We configure the revision to not receive any traffic.</li>
</ul>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy Revision On Cloud Run</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">gcloud run deploy bookstore-preview \</span>
<span class="s">--image=gcr.io/rails-gcloud-bookstore/bookstore:${{github.event.number}} \</span>
<span class="s">--platform=managed \</span>
<span class="s">--port=3000 \</span>
<span class="s">--revision-suffix=pr-${{github.event.number}}-${{steps.commit.outputs.hash}} \</span>
<span class="s">--add-cloudsql-instances=rails-gcloud-bookstore:europe-north1:bookstore-db \</span>
<span class="s">--set-env-vars DATABASE_HOST=/cloudsql/rails-gcloud-bookstore:europe-north1:bookstore-db \</span>
<span class="s">--set-env-vars DATABASE_USERNAME='bookstore' \</span>
<span class="s">--set-env-vars DATABASE_PASSWORD='bookstore' \</span>
<span class="s">--set-env-vars DATABASE_NAME=bookstore_production_${{github.event.number}} \</span>
<span class="s">--set-env-vars RAILS_PRODUCTION_KEY=${{ secrets.RAILS_PRODUCTION_KEY }} \</span>
<span class="s">--allow-unauthenticated \</span>
<span class="s">--region=europe-north1</span>
</code></pre></div></div>
<p>We have created a new revision of our bookstore service serving the code from our pull request. Now we only need to make sure the revision is easily reachable. We can accomplish this using the <code class="language-plaintext highlighter-rouge">update-traffic</code> command.</p>
<p>Setting a tag allows us to reach to revision we created using some prefix. Assuming a base URL for accessing the bookstore application that looks like <code class="language-plaintext highlighter-rouge">https://bookstore-abcd12345-ab.a.run.app</code> and a tag <code class="language-plaintext highlighter-rouge">pr-1234</code> the revision can be reached under <code class="language-plaintext highlighter-rouge">https://pr-1234---bookstore-abcd12345-ab.a.run.app</code>.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Update Traffic</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">gcloud components install beta</span>
<span class="s">gcloud beta run services update-traffic bookstore-preview \</span>
<span class="s">--update-tags pr-${{github.event.number}}=bookstore-preview-${{github.event.number}}-${{steps.commit.outputs.hash}} \</span>
<span class="s">--platform=managed \</span>
<span class="s">--region=europe-north1</span>
</code></pre></div></div>
<p>Last but not least, let’s post a comment to our PR which contains the link to the preview environment. We can retrieve the base URL of our bookstore service and change it to include the tag we previously set. Once we have that, we use the <a href="https://github.com/marketplace/actions/add-pr-comment">add-pr-comment</a> action to post the comment.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Get Preview URL</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">preview-url</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">url=$(gcloud run services describe bookstore-preview --format 'value(status.url)' --platform=managed --region=europe-north1 | sed 's|https://|https://pr-${{github.event.number}}---|g')</span>
<span class="s">echo "::set-output name=url::$url"</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Post PR comment with preview deployment URL</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">mshick/add-pr-comment@v1</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">GITHUB_TOKEN</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_TOKEN }}</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">message</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">Successfully deployed preview at ${{steps.preview-url.outputs.url}}</span>
<span class="na">allow-repeats</span><span class="pi">:</span> <span class="no">false</span>
</code></pre></div></div>
<p>Add your GitHub repository as remote for your application and push your local changes.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git remote add origin git@github.com:<your-repository-url>
git add <span class="nb">.</span> <span class="o">&&</span> git commit <span class="nt">-m</span> <span class="s1">'Add pull request reviews'</span>
git push <span class="nt">--set-upstream</span> origin master
</code></pre></div></div>
<p>Opening a pull request should execute the GitHub workflow we defined. Congratulations for making it all the way! :tada:</p>
<h2 id="closing">Closing</h2>
<p>I hope this guide has given you a rough idea of how Cloud Run and GitHub Actions can be leveraged to create preview deployments for your pull requests.</p>
<p>I took a lot of inspiration from other posts - big shout out to <a href="https://twitter.com/MaximeHeckel">@MaximeHeckel</a> whose <a href="https://blog.maximeheckel.com/posts/build-serverless-preview-deployment">in depth post</a> helped me out a lot.</p>
<p>Building on existing solutions to make them work even for a simple Rails application was a nice challenge, and I had a lot of fun working on this. That being said, I’m aware that there is a lot of potential for improving the current solution.</p>
<p>We haven’t taken care of cleaning up after us at all. Right now each pull request creates a new database in our Cloud SQL instance. Both the GitHub Action as well as the container build can be made <em>a lot</em> more performant.</p>
<p>And of course, you can increase the complexity of the base application. How does this solution behave when using ActiveStorage? How can we deal with dependencies to other services, like an internal authentication service? The list goes on and on.</p>
<p>But this guide is long enough as it is.</p>
<p>I leave the rest for you to explore :slightly_smiling_face:</p>Hans SchnedlitzIf you work in a larger organization, chances are you have access to some sort of staging environment where you can preview work that is currently in progress. Staging environments can be immensely useful to get early feedback, be it from QA or other departments, which can speed up the development of new features significantly.