Cowie.meMadeline Cowie, Software Developer in ChicagoZola2022-06-23T00:00:00+00:00https://cowie.me/atom.xmlUsing Railway-Oriented Programming to Model Business Transactions2022-06-23T00:00:00+00:002022-06-23T00:00:00+00:00https://cowie.me/posts/using-railway-oriented-programming-to-model-business-transactions/<p>The more complex a process is, the more ways it can fail. The more forks in the road you have, the more dead ends you’ll come across. In a complex web application on you might validate input, locate records, communicate with external services, and more, and that’s on top of performing any number of operations the business logic calls for. This can lead to a messy web of interdependent code and lots of additional overhead to continually ensure we are in a good state.</p>
<span id="continue-reading"></span>
<p><a rel="noopener" target="_blank" href="https://fsharpforfunandprofit.com/rop/">Railway oriented programming</a> offers a solution. It’s a technique for dividing a complex set of logic into a series of discrete steps. Each step can either pass or fail. If it passes, the transaction continues. If it fails, the transaction splits off from the railway at that given stage and ends. This naturally fits the shape most processes have. Most of the time things are successful and stays on that main path, but there’s many things that <em>could</em> go wrong. It’s easy to fixate on that happy path, that’s where the stakeholders are going to be focused, and deal with the rest later. But, as developers we need to cover every possible state, and that’s what the Railway way helps us do. It both greatly reduces what we’re dealing with at at one time, as well as providing a natural place for any error handling required. Organizing our logic this way gives us a few additional intertwined advantages:</p>
<ul>
<li>We can more easily communicate what the system is doing to non-technical stakeholders. </li>
<li>We can explicitly define any requirements we have for any step of the process</li>
<li>When things fail, we can know where, when, and the exact conditions that caused them.</li>
<li>We get the additional benefit of having this kind of domain knowledge outside of application concerns like Models or Controllers. This lets us test our implementation directly and independently.</li>
</ul>
<h2 id="implementation">Implementation</h2>
<p>A few years ago, I came across <a rel="noopener" target="_blank" href="https://dry-rb.org/">dry-rb</a>, an evolving collection of useful abstractions for Ruby. It includes the simple yet powerful <a rel="noopener" target="_blank" href="https://dry-rb.org/gems/dry-monads/"><code>Dry::Monads</code></a> library. Monads are famously a bit difficult to grok, but they have lots of uses, some of which are laid out in their <a rel="noopener" target="_blank" href="https://dry-rb.org/gems/dry-monads/">documentation</a>. Encouraged additional reading!</p>
<p><code>Dry::Monads</code> provides what we need to build a railway-oriented process: conventions to report success or failure (<code>:result</code>), and an eject button when a failure is hit (<code>:do</code>). For this example, let’s walk through a someone registering to attend a paid online workshop.</p>
<p>To complete a registration, we need the workshop being registered for, basic information for the registrant (name, email), and optionally a discount code, and payment source to be charged if the workshop is not free (before or after discount!). This is a simplified example but already there’s a few ways it can go sideways. The workshop may not be open to registration, registrant’s email may be missing, the discount code may be expired, payment may or may not be due, and if so that charge has to process.</p>
<p>We’ll start with a class, including the modules we want from <code>Dry::Monads</code>, and initialized with the input we’ll need to process the registration.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#ff79c6;">require </span><span style="color:#f1fa8c;">'dry/monads'
</span><span>
</span><span style="color:#ff79c6;">class </span><span style="text-decoration:underline;color:#8be9fd;">WorkshopRegistrar
</span><span> </span><span style="color:#ff79c6;">include </span><span style="font-style:italic;color:#66d9ef;">Dry</span><span style="color:#ff79c6;">::</span><span style="font-style:italic;color:#66d9ef;">Monads</span><span>[</span><span style="color:#bd93f9;">:result</span><span>, </span><span style="color:#bd93f9;">:do</span><span>]
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">initialize</span><span>(</span><span style="font-style:italic;color:#ffb86c;">workshop</span><span>:, </span><span style="font-style:italic;color:#ffb86c;">discount_code</span><span>: </span><span style="color:#bd93f9;">nil</span><span>, </span><span style="font-style:italic;color:#ffb86c;">payment_source</span><span>: </span><span style="color:#bd93f9;">nil</span><span>, </span><span style="font-style:italic;color:#ffb86c;">registrant_info</span><span>: {})
</span><span> </span><span style="color:#ffb86c;">@workshop </span><span style="color:#ff79c6;">=</span><span> workshop
</span><span> </span><span style="color:#ffb86c;">@registrant_info </span><span style="color:#ff79c6;">=</span><span> registrant_info
</span><span> </span><span style="color:#ffb86c;">@discount_code </span><span style="color:#ff79c6;">=</span><span> discount_code
</span><span> </span><span style="color:#ffb86c;">@payment_source </span><span style="color:#ff79c6;">=</span><span> payment_source
</span><span> </span><span style="color:#ff79c6;">end
</span><span style="color:#ff79c6;">end
</span></code></pre>
<p>We’re ready for our first step. Each one must return <code>Success</code> or <code>Failure</code>. These can wrap any return value. I’ve not come across a case where there was really a need to return some resulting value. So, instead I like to pass a descriptive error code inside any Failures. This can provide immediate context when debugging and scanning logs. It could also be used later to provide specific feedback to the user.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">call
</span><span> </span><span style="color:#ff79c6;">yield</span><span> check_availability(</span><span style="color:#ffb86c;">@workshop</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">check_availability</span><span>(</span><span style="font-style:italic;color:#ffb86c;">workshop</span><span>)
</span><span> </span><span style="color:#ff79c6;">if</span><span> workshop</span><span style="color:#ff79c6;">.</span><span>blank?
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:workshop_not_found</span><span>)
</span><span> </span><span style="color:#ff79c6;">elsif</span><span> workshop</span><span style="color:#ff79c6;">.</span><span>online_registerable?
</span><span> </span><span style="color:#ffffff;">Success</span><span>(</span><span style="color:#bd93f9;">true</span><span>)
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:workshops_not_online_registerable</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span></code></pre>
<p>Next, we’re ready to generate a registration record. We will need this registration down the line, so we’ll return with our Success.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">call
</span><span> </span><span style="color:#ff79c6;">yield</span><span> check_availability(</span><span style="color:#ffb86c;">@workshop</span><span>)
</span><span> registration </span><span style="color:#ff79c6;">= yield</span><span> create_registration(</span><span style="color:#bd93f9;">workshop: </span><span style="color:#ffb86c;">@workshop</span><span>, </span><span style="color:#bd93f9;">registrant_info: </span><span style="color:#ffb86c;">@registrant_info</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">check_availability</span><span>(</span><span style="font-style:italic;color:#ffb86c;">workshop</span><span>)
</span><span> </span><span style="color:#6272a4;"># ...
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">create_registration</span><span>(</span><span style="font-style:italic;color:#ffb86c;">workshop</span><span>:, </span><span style="font-style:italic;color:#ffb86c;">registrant_info</span><span>: {})
</span><span> reg </span><span style="color:#ff79c6;">= </span><span style="font-style:italic;color:#66d9ef;">Registration</span><span style="color:#ff79c6;">.new</span><span>(registrant_info)
</span><span> reg</span><span style="color:#ff79c6;">.</span><span>workshop </span><span style="color:#ff79c6;">=</span><span> workshop
</span><span> reg</span><span style="color:#ff79c6;">.</span><span>confirmation_code </span><span style="color:#ff79c6;">=</span><span> reg</span><span style="color:#ff79c6;">.</span><span>generate_unique_confirmation_code
</span><span> </span><span style="color:#ff79c6;">if</span><span> reg</span><span style="color:#ff79c6;">.</span><span>save
</span><span> </span><span style="color:#ffffff;">Success</span><span>(reg)
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:registration_could_not_create</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span></code></pre>
<p>With the do syntax from <code>Dry::Monads</code>, to run through our steps, we <code>yield</code> to them in our desired order. Remember, if a Failure is returned from a <code>yield</code> the method exits early with that result. From here forward, we can know we have a valid, available workshop, and a valid (pending) registration.</p>
<p>Following this pattern, we can add each step of what needs to happen one by one. Here’s a more complete view at what that looks like in this simplified example.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#ff79c6;">require </span><span style="color:#f1fa8c;">'dry/monads'
</span><span>
</span><span style="color:#ff79c6;">class </span><span style="text-decoration:underline;color:#8be9fd;">WorkshopRegistrar
</span><span> </span><span style="color:#ff79c6;">include </span><span style="font-style:italic;color:#66d9ef;">Dry</span><span style="color:#ff79c6;">::</span><span style="font-style:italic;color:#66d9ef;">Monads</span><span>[</span><span style="color:#bd93f9;">:result</span><span>, </span><span style="color:#bd93f9;">:do</span><span>]
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">initialize</span><span>(</span><span style="font-style:italic;color:#ffb86c;">workshop</span><span>:, </span><span style="font-style:italic;color:#ffb86c;">discount_code</span><span>: </span><span style="color:#f1fa8c;">""</span><span>, </span><span style="font-style:italic;color:#ffb86c;">payment_source</span><span>: </span><span style="color:#f1fa8c;">""</span><span>, </span><span style="font-style:italic;color:#ffb86c;">registrant_info</span><span>: {}, </span><span style="font-style:italic;color:#ffb86c;">expected_price_cents</span><span>:)
</span><span> </span><span style="color:#ffb86c;">@workshop </span><span style="color:#ff79c6;">=</span><span> workshop
</span><span> </span><span style="color:#ffb86c;">@registrant_info </span><span style="color:#ff79c6;">=</span><span> registrant_info
</span><span> </span><span style="color:#ffb86c;">@discount_code </span><span style="color:#ff79c6;">=</span><span> discount_code
</span><span> </span><span style="color:#ffb86c;">@payment_source </span><span style="color:#ff79c6;">=</span><span> payment_source
</span><span> </span><span style="color:#ffb86c;">@expected_price </span><span style="color:#ff79c6;">= </span><span style="font-style:italic;color:#66d9ef;">Money</span><span style="color:#ff79c6;">.new</span><span>(expected_price_cents)
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">call
</span><span> </span><span style="font-style:italic;color:#66d9ef;">ApplicationRecord</span><span style="color:#ff79c6;">.</span><span>transaction </span><span style="color:#ff79c6;">do
</span><span> </span><span style="color:#ff79c6;">yield</span><span> check_availability(</span><span style="color:#ffb86c;">@workshop</span><span>)
</span><span> registration </span><span style="color:#ff79c6;">= yield</span><span> create_registration(</span><span style="color:#bd93f9;">workshop: </span><span style="color:#ffb86c;">@workshop</span><span>, </span><span style="color:#bd93f9;">registrant_info: </span><span style="color:#ffb86c;">@registrant_info</span><span>)
</span><span> </span><span style="color:#ff79c6;">yield</span><span> check_discount_code(</span><span style="color:#ffb86c;">@discount_code</span><span>)
</span><span> payment </span><span style="color:#ff79c6;">= yield</span><span> build_payment(</span><span style="color:#bd93f9;">registration:</span><span> registration, </span><span style="color:#bd93f9;">discount_code: </span><span style="color:#ffb86c;">@discount_code</span><span>, </span><span style="color:#bd93f9;">workshop: </span><span style="color:#ffb86c;">@workshop</span><span>)
</span><span> </span><span style="color:#ff79c6;">yield</span><span> verify_price(payment, </span><span style="color:#ffb86c;">@expected_price</span><span>)
</span><span> </span><span style="color:#ff79c6;">yield</span><span> charge_payment(payment, </span><span style="color:#bd93f9;">source: </span><span style="color:#ffb86c;">@payment_source</span><span>)
</span><span> </span><span style="color:#ff79c6;">yield</span><span> mark_completed(registration)
</span><span> </span><span style="color:#ff79c6;">yield</span><span> mail_registrant(registration)
</span><span>
</span><span> </span><span style="color:#ffffff;">Success</span><span>(registration)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">check_availability</span><span>(</span><span style="font-style:italic;color:#ffb86c;">workshop</span><span>)
</span><span> </span><span style="color:#ff79c6;">if</span><span> workshop</span><span style="color:#ff79c6;">.</span><span>blank?
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:workshop_not_found</span><span>)
</span><span> </span><span style="color:#ff79c6;">elsif</span><span> workshop</span><span style="color:#ff79c6;">.</span><span>online_registerable?
</span><span> </span><span style="color:#ffffff;">Success</span><span>(</span><span style="color:#bd93f9;">true</span><span>)
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:workshop_not_online_registerable</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">create_registration</span><span>(</span><span style="font-style:italic;color:#ffb86c;">workshop</span><span>:, </span><span style="font-style:italic;color:#ffb86c;">registrant_info</span><span>: {})
</span><span> reg </span><span style="color:#ff79c6;">= </span><span style="font-style:italic;color:#66d9ef;">Registration</span><span style="color:#ff79c6;">.new</span><span>(registrant_info)
</span><span> reg</span><span style="color:#ff79c6;">.</span><span>workshop </span><span style="color:#ff79c6;">=</span><span> workshop
</span><span> reg</span><span style="color:#ff79c6;">.</span><span>confirmation_code </span><span style="color:#ff79c6;">=</span><span> reg</span><span style="color:#ff79c6;">.</span><span>generate_unique_confirmation_code
</span><span> </span><span style="color:#ff79c6;">if</span><span> reg</span><span style="color:#ff79c6;">.</span><span>save
</span><span> </span><span style="color:#ffffff;">Success</span><span>(reg)
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:registration_could_not_create</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">check_discount_code</span><span>(</span><span style="font-style:italic;color:#ffb86c;">code</span><span>)
</span><span> </span><span style="color:#ff79c6;">if</span><span> code</span><span style="color:#ff79c6;">.</span><span>blank?
</span><span> </span><span style="color:#ffffff;">Success</span><span>()
</span><span> </span><span style="color:#ff79c6;">elsif </span><span style="font-style:italic;color:#66d9ef;">DiscountCode</span><span style="color:#ff79c6;">.</span><span>active</span><span style="color:#ff79c6;">.</span><span>lookup(code)
</span><span> </span><span style="color:#ffffff;">Success</span><span>(code)
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:discount_code_invalid</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">build_payment</span><span>(</span><span style="font-style:italic;color:#ffb86c;">registration</span><span>:, </span><span style="font-style:italic;color:#ffb86c;">discount_code</span><span>:, </span><span style="font-style:italic;color:#ffb86c;">workshop</span><span>:)
</span><span> </span><span style="color:#ff79c6;">if</span><span> registration</span><span style="color:#ff79c6;">&.</span><span>email</span><span style="color:#ff79c6;">.</span><span>blank? </span><span style="color:#ff79c6;">||</span><span> registration</span><span style="color:#ff79c6;">&.</span><span>confirmation_code</span><span style="color:#ff79c6;">.</span><span>blank?
</span><span> </span><span style="color:#ff79c6;">return </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:payment_missing_info</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> original_amount </span><span style="color:#ff79c6;">= if</span><span> discount_code</span><span style="color:#ff79c6;">.</span><span>present?
</span><span> workshop</span><span style="color:#ff79c6;">.</span><span>price
</span><span> </span><span style="color:#ff79c6;">else
</span><span> workshop</span><span style="color:#ff79c6;">.</span><span>current_price
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ffffff;">Success</span><span>(</span><span style="font-style:italic;color:#66d9ef;">Payment</span><span style="color:#ff79c6;">.new</span><span>(
</span><span> </span><span style="color:#bd93f9;">item:</span><span> registration,
</span><span> </span><span style="color:#bd93f9;">code:</span><span> discount_code,
</span><span> </span><span style="color:#bd93f9;">original_amount:</span><span> original_amount,
</span><span> </span><span style="color:#bd93f9;">email:</span><span> registration</span><span style="color:#ff79c6;">.</span><span>email,
</span><span> </span><span style="color:#bd93f9;">statement_descriptor: </span><span style="color:#f1fa8c;">"Smart Workshops Inc"</span><span>,
</span><span> </span><span style="color:#bd93f9;">description:</span><span> workshop</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">name</span><span>,
</span><span> </span><span style="color:#bd93f9;">additional_metadata: </span><span>{ </span><span style="color:#bd93f9;">confirmation_code:</span><span> registration</span><span style="color:#ff79c6;">.</span><span>confirmation_code }
</span><span> ))
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">verify_price</span><span>(</span><span style="font-style:italic;color:#ffb86c;">payment</span><span>, </span><span style="font-style:italic;color:#ffb86c;">expected_price</span><span>)
</span><span> </span><span style="color:#ff79c6;">if</span><span> payment</span><span style="color:#ff79c6;">.</span><span>discounted_amount </span><span style="color:#ff79c6;">==</span><span> expected_price
</span><span> </span><span style="color:#ffffff;">Success</span><span>(</span><span style="color:#bd93f9;">true</span><span>)
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:expected_price_differs</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">charge_payment</span><span>(</span><span style="font-style:italic;color:#ffb86c;">payment</span><span>, </span><span style="font-style:italic;color:#ffb86c;">source</span><span>:)
</span><span> </span><span style="color:#ff79c6;">if</span><span> payment</span><span style="color:#ff79c6;">.</span><span>charge!(</span><span style="color:#bd93f9;">source:</span><span> source)
</span><span> </span><span style="color:#ffffff;">Success</span><span>(payment)
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:payment_charge_failed</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">mark_completed</span><span>(</span><span style="font-style:italic;color:#ffb86c;">registration</span><span>)
</span><span> </span><span style="color:#ff79c6;">if</span><span> registration</span><span style="color:#ff79c6;">.</span><span>mark_completed!
</span><span> </span><span style="color:#ffffff;">Success</span><span>(</span><span style="color:#bd93f9;">true</span><span>)
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#ffffff;">Failure</span><span>(</span><span style="color:#bd93f9;">:regisration_could_not_mark</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">mail_registrant</span><span>(</span><span style="font-style:italic;color:#ffb86c;">registration</span><span>)
</span><span> </span><span style="font-style:italic;color:#66d9ef;">RegistrationMailer</span><span style="color:#ff79c6;">.</span><span>with(</span><span style="color:#bd93f9;">registration:</span><span> registration)</span><span style="color:#ff79c6;">.</span><span>registered</span><span style="color:#ff79c6;">.</span><span>deliver_later
</span><span> </span><span style="color:#ffffff;">Success</span><span>(registration)
</span><span> </span><span style="color:#ff79c6;">rescue </span><span>=> error
</span><span> </span><span style="color:#6272a4;"># This step is non-critical, a human can send this information if it fails.
</span><span> </span><span style="color:#6272a4;"># But do track the error.
</span><span> </span><span style="font-style:italic;color:#66d9ef;">ExceptionTracker</span><span style="color:#ff79c6;">.</span><span>track(error, </span><span style="color:#bd93f9;">registration:</span><span> registration, </span><span style="color:#bd93f9;">action: </span><span style="color:#f1fa8c;">"mail_registrant"</span><span>)
</span><span> </span><span style="color:#ffffff;">Success</span><span>(registration)
</span><span> </span><span style="color:#ff79c6;">end
</span><span style="color:#ff79c6;">end
</span></code></pre>
<p>You can see we don’t really do much heavy lifting directly here. The transaction should be a level above what it’s working with, focused on orchestrating the pieces rather than doing the work. And because transactions themselves return <code>Success</code> or <code>Failure</code>, they can easily be split up or strung together as needed.</p>
<p>This also makes testing easier. Each step can be called independently, and any tests of the transaction itself only need to be concerned with the control flow (those methods already have unit tests, right? 😉). Like any method, keeping steps small and single purpose helps testing, maintenance, and refactoring.</p>
<p>Leveraging a bit of syntactical sugar from <a rel="noopener" target="_blank" href="https://relishapp.com/rspec/">RSpec</a> and <a rel="noopener" target="_blank" href="https://github.com/thoughtbot/factory_bot">FactoryBot</a>, we can quickly iterate through cases like this:</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="font-style:italic;color:#66d9ef;">RSpec</span><span style="color:#ff79c6;">.</span><span>describe </span><span style="color:#ffffff;">WorkshopRegistrar </span><span style="color:#ff79c6;">do
</span><span> let(</span><span style="color:#bd93f9;">:registrar</span><span>) { </span><span style="font-style:italic;color:#66d9ef;">WorkshopRegistrar</span><span style="color:#ff79c6;">.new</span><span>(</span><span style="color:#bd93f9;">workshop:</span><span> workshop, </span><span style="color:#bd93f9;">expected_price_cents:</span><span> workshop</span><span style="color:#ff79c6;">.</span><span>current_price</span><span style="color:#ff79c6;">.</span><span>fractional) }
</span><span>
</span><span> describe </span><span style="color:#f1fa8c;">"#check_availability" </span><span style="color:#ff79c6;">do
</span><span> let(</span><span style="color:#bd93f9;">:subject</span><span>) { registrar</span><span style="color:#ff79c6;">.</span><span>check_availability(workshop) }
</span><span>
</span><span> context </span><span style="color:#f1fa8c;">"workshop not provided" </span><span style="color:#ff79c6;">do
</span><span> let(</span><span style="color:#bd93f9;">:workshop</span><span>) { </span><span style="color:#bd93f9;">nil </span><span>}
</span><span> it { is_expected</span><span style="color:#ff79c6;">.</span><span>to be_a_failure }
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> context </span><span style="color:#f1fa8c;">"workshop requires external registration" </span><span style="color:#ff79c6;">do
</span><span> let(</span><span style="color:#bd93f9;">:workshop</span><span>) { build(</span><span style="color:#bd93f9;">:workshop</span><span>, </span><span style="color:#bd93f9;">:external_registration</span><span>) }
</span><span> it { is_expected</span><span style="color:#ff79c6;">.</span><span>to be_a_failure }
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> context </span><span style="color:#f1fa8c;">"workshop is online registerable" </span><span style="color:#ff79c6;">do
</span><span> let(</span><span style="color:#bd93f9;">:workshop</span><span>) { build(</span><span style="color:#bd93f9;">:workshop</span><span>, </span><span style="color:#bd93f9;">:registerable</span><span>) }
</span><span> it { is_expected</span><span style="color:#ff79c6;">.</span><span>to be_a_success }
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> </span><span style="color:#6272a4;"># ...
</span><span> </span><span style="color:#ff79c6;">end
</span></code></pre>
<p>Now, it’s ready to be used. Our controller only has to do controller-y things, like assembling the parameters and rendering the result. </p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#ff79c6;">class </span><span style="text-decoration:underline;color:#8be9fd;">RegistrationsController </span><span>< </span><span style="text-decoration:underline;font-style:italic;color:#8be9fd;">ApplicationController
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">create
</span><span> workshop </span><span style="color:#ff79c6;">= </span><span style="font-style:italic;color:#66d9ef;">Workshop</span><span style="color:#ff79c6;">.</span><span>published</span><span style="color:#ff79c6;">.</span><span>find_by(</span><span style="color:#bd93f9;">id:</span><span> registration_params[</span><span style="color:#bd93f9;">:workshop_id</span><span>])
</span><span>
</span><span> registrar </span><span style="color:#ff79c6;">= </span><span style="font-style:italic;color:#66d9ef;">WorkshopRegistrar</span><span style="color:#ff79c6;">.new</span><span>(
</span><span> </span><span style="color:#bd93f9;">workshop:</span><span> workshop,
</span><span> </span><span style="color:#bd93f9;">discount_code:</span><span> registration_params[</span><span style="color:#bd93f9;">:discount_code</span><span>],
</span><span> </span><span style="color:#bd93f9;">payment_source:</span><span> registration_params[</span><span style="color:#bd93f9;">:payment_source</span><span>],
</span><span> </span><span style="color:#bd93f9;">registrant_info:</span><span> registration_params[</span><span style="color:#bd93f9;">:registrant_info</span><span>],
</span><span> </span><span style="color:#bd93f9;">expected_price_cents:</span><span> registration_params[</span><span style="color:#bd93f9;">:expected_price_cents</span><span>]
</span><span> )
</span><span> result </span><span style="color:#ff79c6;">=</span><span> registrar</span><span style="color:#ff79c6;">.</span><span>call
</span><span>
</span><span> </span><span style="color:#ff79c6;">if</span><span> result</span><span style="color:#ff79c6;">.</span><span>success?
</span><span> </span><span style="color:#ffb86c;">@registration </span><span style="color:#ff79c6;">=</span><span> result</span><span style="color:#ff79c6;">.</span><span>value!
</span><span> render </span><span style="color:#bd93f9;">json: </span><span style="color:#ffb86c;">@registration</span><span style="color:#ff79c6;">.</span><span>as_api_json
</span><span> </span><span style="color:#ff79c6;">else
</span><span> failure_code </span><span style="color:#ff79c6;">=</span><span> result</span><span style="color:#ff79c6;">.</span><span>failure
</span><span> </span><span style="font-style:italic;color:#66d9ef;">Rails</span><span style="color:#ff79c6;">.</span><span>logger</span><span style="color:#ff79c6;">.</span><span>info </span><span style="color:#f1fa8c;">"Registration unsuccessful. Code: </span><span>#{failure_code}</span><span style="color:#f1fa8c;">"
</span><span> render_error </span><span style="color:#bd93f9;">code:</span><span> failure_code, </span><span style="color:#bd93f9;">message: </span><span style="font-style:italic;color:#66d9ef;">I18n</span><span style="color:#ff79c6;">.</span><span>t(</span><span style="color:#f1fa8c;">"registrations.errors.</span><span>#{failure_code}</span><span style="color:#f1fa8c;">"</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span style="color:#ff79c6;">end
</span></code></pre>
<h2 id="conclusion">Conclusion</h2>
<p>The fastest way of fighting complexity is to reduce the possibilities that have to be dealt with. Transactions should represent a specific set of operations, but not necessarily the operations themselves. <strong>By setting the bounds and order of those operations, we can clearly express our expectations at each step along the way.</strong> Taking a process with many inputs and many possible outcomes and compressing them down to a series of discrete pass/fail steps, we only have to deal with one thing at a time.</p>
How to use Rails and Webpack to develop and serve a Javascript client library2020-07-24T00:00:00+00:002020-07-24T00:00:00+00:00https://cowie.me/posts/using-rails-and-webpacker-to-serve-a-javascript-library/<blockquote>
This post was originally written for <a href="https://www.neotericdesign.com">Neoteric Design</a>.
</blockquote>
<p>At Neoteric, we’ve been developing more sites and apps using the JAMstack framework, especially when it’s a better fit than a full CMS web application. But what about projects that still need <em>sprinkles</em> of a web app?</p>
<p>For these cases, we build a quick, minimal Rails API to serve the app needs, and get the best of both worlds. On the one hand, we get the speed, security, and ease of growth that comes with a static site generator for most of the site. On the other hand, we get rapid and reliable Rails development tools to implement business logic that drives features like registrations, e-commerce, accounts, and more.</p>
<p>Developing an API with Rails is well-worn territory, but when it comes to building the code that consumes it, there are many different ways to approach it. How to choose? When it comes time to solve new problems, I find it best to try the simplest possible way until I feel the pain.</p>
<p>I started just writing client Javascript code within the static site project, with no build system, and only a couple of plain script includes. This Javascript quickly (and not <em>too</em> surprisingly) exceeded the scope I was comfortable with to ship it live.</p>
<p>We want to keep static site deploys lean and uncomplicated. So another option, introducing a Node dependency, was tossed out: it’s the opposite of lean and uncomplicated.</p>
<p>How about developing the client as its own project, as an NPM/ad-hoc package? That option seemed a bit overkill for something used for only one site, that was not very large, and would likely be low-churn after launch.</p>
<p>But in developing the Rails API, we already have a perfectly good Webpack pipeline, testing infrastructure, and hosting. That’s when I realized: maybe we can use Webpack to develop and serve the client.</p>
<h2 id="expose-your-functions">Expose your functions</h2>
<p>Webpack doesn’t pollute the global namespace by default, which is sensible. But, what if you really do need other code to use it? There’s a loader for that! <a rel="noopener" target="_blank" href="https://github.com/webpack-contrib/expose-loader">expose-loader</a> can take exports from a file and make them available under a namespace. Here’s a basic expose-loader configuration. The <code>options</code> it takes is the namespace the exports will be accessible under. In this case, we export a class called <code>Client</code>, and we can call it from the outside as <code>Courses.Client</code></p>
<pre data-lang="javascript" style="background-color:#282a36;color:#f8f8f2;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="color:#6272a4;">// config/webpack/loaders/expose.js
</span><span>
</span><span style="font-style:italic;color:#66d9ef;">module</span><span style="color:#ff79c6;">.</span><span style="font-style:italic;color:#66d9ef;">exports </span><span style="color:#ff79c6;">= </span><span>{
</span><span> rules: [{
</span><span> test: </span><span style="color:#f1fa8c;">/courses_client/</span><span>,
</span><span> use: [{
</span><span> loader: </span><span style="color:#f1fa8c;">'expose-loader'</span><span>,
</span><span> options: </span><span style="color:#f1fa8c;">'Courses'
</span><span> }]
</span><span> }]
</span><span>}
</span></code></pre>
<p>Then, add the loader to webpack:</p>
<pre data-lang="javascript" style="background-color:#282a36;color:#f8f8f2;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span style="color:#6272a4;">// config/webpack/environment.js
</span><span>
</span><span style="font-style:italic;color:#8be9fd;">const </span><span>{ </span><span style="color:#ffffff;">environment </span><span>} </span><span style="color:#ff79c6;">= </span><span style="color:#8be9fd;">require</span><span>(</span><span style="color:#f1fa8c;">'@rails/webpacker'</span><span>)
</span><span style="font-style:italic;color:#8be9fd;">const </span><span style="color:#ffffff;">expose </span><span style="color:#ff79c6;">= </span><span style="color:#8be9fd;">require</span><span>(</span><span style="color:#f1fa8c;">'./loaders/expose'</span><span>)
</span><span>
</span><span style="color:#ffffff;">environment</span><span style="color:#ff79c6;">.</span><span style="color:#ffffff;">loaders</span><span style="color:#ff79c6;">.</span><span style="color:#8be9fd;">prepend</span><span>(</span><span style="color:#f1fa8c;">'expose'</span><span>, </span><span style="color:#ffffff;">expose</span><span>)
</span><span>
</span><span style="font-style:italic;color:#66d9ef;">module</span><span style="color:#ff79c6;">.</span><span style="font-style:italic;color:#66d9ef;">exports </span><span style="color:#ff79c6;">= </span><span style="color:#ffffff;">environment
</span></code></pre>
<h2 id="link-the-pack">Link the pack</h2>
<p>Webpacker packs are still fingerprinted like Asset Pipeline assets. So, we have to get the current path from Rails. The most straightforward approach is to set up a controller action with a redirect. There are likely more lightweight ways to do this, but this has worked well enough for us:</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#ff79c6;">class </span><span style="text-decoration:underline;color:#8be9fd;">ClientAssetsController </span><span>< </span><span style="text-decoration:underline;font-style:italic;color:#8be9fd;">ApplicationController
</span><span> </span><span style="color:#ff79c6;">include </span><span style="font-style:italic;color:#66d9ef;">ActionView</span><span style="color:#ff79c6;">::</span><span style="font-style:italic;color:#66d9ef;">Helpers</span><span style="color:#ff79c6;">::</span><span>AssetUrlHelper
</span><span> </span><span style="color:#ff79c6;">include </span><span style="font-style:italic;color:#66d9ef;">Webpacker</span><span style="color:#ff79c6;">::</span><span>Helper
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">courses_client
</span><span> redirect_to asset_pack_path(</span><span style="color:#f1fa8c;">"courses_client.js"</span><span>)
</span><span> </span><span style="color:#ff79c6;">end
</span><span style="color:#ff79c6;">end
</span></code></pre>
<h2 id="set-up-some-examples">Set up some examples</h2>
<p>Setting up a reference implementation inside the Rails app was hugely helpful, even in our case where there’s only one consumer of the library. Having both keeps us honest about the contract the API promises and gives us an easy way to do full-stack system tests.</p>
<p>To isolate the reference implementation from the rest of the Rails app, and to simulate being an outside site, I create a layout and JS pack for it:</p>
<pre data-lang="html" style="background-color:#282a36;color:#f8f8f2;" class="language-html "><code class="language-html" data-lang="html"><span style="color:#6272a4;"><!-- app/layouts/examples.html.erb -->
</span><span>
</span><span><</span><span style="color:#ff79c6;">script </span><span style="color:#50fa7b;">src</span><span>=</span><span style="color:#f1fa8c;">"<%= courses_client_url %>"
</span><span> </span><span style="color:#50fa7b;">type</span><span>=</span><span style="color:#f1fa8c;">"text/javascript"</span><span>></</span><span style="color:#ff79c6;">script</span><span>>
</span><span>
</span><span><%= javascript_pack_tag "examples" %>
</span></code></pre>
<pre data-lang="javascript" style="background-color:#282a36;color:#f8f8f2;" class="language-javascript "><code class="language-javascript" data-lang="javascript"><span>
</span><span style="color:#6272a4;">// app/javascript/packs/examples.js
</span><span>
</span><span>document</span><span style="color:#ff79c6;">.</span><span style="color:#8be9fd;">addEventListener</span><span>(</span><span style="color:#f1fa8c;">"DOMContentLoaded"</span><span>, () </span><span style="font-style:italic;color:#8be9fd;">=> </span><span>{
</span><span> </span><span style="color:#6272a4;">// ...
</span><span>
</span><span> </span><span style="font-style:italic;color:#8be9fd;">let </span><span style="color:#ffffff;">client </span><span style="color:#ff79c6;">= new </span><span>Courses</span><span style="color:#ff79c6;">.</span><span>Client()
</span><span> </span><span style="color:#ffffff;">client</span><span style="color:#ff79c6;">.</span><span style="color:#50fa7b;">doThings</span><span>()
</span><span>}
</span></code></pre>
<p>I then add the controller and routes, taking care not to have them accessible in production:</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#6272a4;"># config/routes.rb
</span><span>
</span><span style="font-style:italic;color:#66d9ef;">Rails</span><span style="color:#ff79c6;">.</span><span>application</span><span style="color:#ff79c6;">.</span><span>routes</span><span style="color:#ff79c6;">.</span><span>draw </span><span style="color:#ff79c6;">do
</span><span> </span><span style="color:#6272a4;"># ...
</span><span> scope </span><span style="color:#f1fa8c;">"/examples"</span><span>, </span><span style="color:#bd93f9;">constraints: </span><span style="color:#ff79c6;">-> </span><span>{ </span><span style="color:#ff79c6;">!</span><span style="font-style:italic;color:#66d9ef;">Rails</span><span style="color:#ff79c6;">.</span><span>env</span><span style="color:#ff79c6;">.</span><span>production? } </span><span style="color:#ff79c6;">do
</span><span> get </span><span style="color:#f1fa8c;">"register" </span><span>=> </span><span style="color:#f1fa8c;">"examples#register"
</span><span> </span><span style="color:#ff79c6;">end
</span><span style="color:#ff79c6;">end
</span><span>
</span><span style="color:#6272a4;"># app/controllers/examples_controller.rb
</span><span>
</span><span style="color:#ff79c6;">class </span><span style="text-decoration:underline;color:#8be9fd;">ExamplesController </span><span>< </span><span style="text-decoration:underline;font-style:italic;color:#8be9fd;">ApplicationController
</span><span> layout </span><span style="color:#f1fa8c;">"examples"
</span><span>
</span><span> </span><span style="color:#ff79c6;">def </span><span style="color:#50fa7b;">register
</span><span> </span><span style="color:#ff79c6;">end
</span><span style="color:#ff79c6;">end
</span></code></pre>
<p>From there, we’re ready to start test cases and system tests to execute against them. It’s still useful to have unit tests for the API and JS separately. But testing the entire stack using friendly and familiar tools dramatically increases our confidence in the reliability of our shipped product. It’s also tremendously useful to reference in the documentation. Unlike usual documentation, it won’t fall out of date with the code, because it’s a part of the test suite/CI.</p>
<p>With this in place, we have all the pieces needed to sprinkle app-like features onto a static site, and all covered by the easy-to-use Rails tooling.</p>
Run Rails Test Suite with Dockerized Selenium on Gitlab CI2018-04-10T00:00:00+00:002018-04-10T00:00:00+00:00https://cowie.me/posts/running-your-rails-test-suite-with-selenium-on-gitlab-ci/<blockquote>
This post was originally written for <a href="https://www.neotericdesign.com">Neoteric Design</a>.
</blockquote>
<p>With just 3 drop-in tweaks, it’s possible to run Rails’ System Tests on Gitlab CI, or other Docker-based continuous integration services.</p>
<p>We’re a small team that, between active development and legacy support, has dozens of projects to pull from that might need to be worked on. While the backbone of most projects is the familiar components of our CMS, each project we work on is tailored to the needs of the client and we can’t rely on our internal knowledge alone. So, each project ships with a robust test suite, featuring a healthy mix of unit and integration tests.</p>
<p>We use RSpec because we find the DSL allows us to write code that is close enough to the English business logic from the client. Starting with Rails 5.1 we migrated fully to Rails’ System Tests for integration testing. It gives us a lot of goodies, like database transaction cleanup, and automatic screenshots of failures.</p>
<p>While the Docker executor makes a lot of sense for the CI environment, we typically don’t need it for dev or production. We still need to be able to run our tests locally, so we strive to make things work in both places, with minimal configuration changes. We use <code>selenium-webdriver</code> with <code>chromedriver</code> to execute tests that involve javascript locally, but installing chrome and <code>chromedriver</code> on our runner each time would be slow and painful. Thankfully, Selenium provides Docker images that are ready to go, and Gitlab Runner’s Docker support lets just just add it as another networked service alongside the database.</p>
<p>Let’s walk through an annotated copy of our <code>.gitlab-ci.yml</code> configuration to see all it’s doing for us.</p>
<pre data-lang="yaml" style="background-color:#282a36;color:#f8f8f2;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#6272a4;"># .gitlab-ci.yml
</span><span>
</span><span style="color:#ff79c6;">image</span><span>: </span><span style="color:#f1fa8c;">"ruby:2.5.1"
</span><span>
</span><span style="color:#6272a4;"># Add Postgres and Selenium Docker services. A Firefox container is also available.
</span><span style="color:#ff79c6;">services</span><span>:
</span><span> - </span><span style="color:#f1fa8c;">postgres:latest
</span><span> - </span><span style="color:#f1fa8c;">selenium/standalone-chrome:latest
</span><span>
</span><span style="color:#6272a4;"># Set our environment variables. Note it's supplying where Gitlab's
</span><span style="color:#6272a4;"># Docker service will make Selenium available at. The Postgres
</span><span style="color:#6272a4;"># config matches a database.yml.gitlab we'll add later.
</span><span style="color:#ff79c6;">variables</span><span>:
</span><span> </span><span style="color:#ff79c6;">POSTGRES_DB</span><span>: </span><span style="color:#f1fa8c;">"test_db"
</span><span> </span><span style="color:#ff79c6;">POSTGRES_USER</span><span>: </span><span style="color:#f1fa8c;">"runner"
</span><span> </span><span style="color:#ff79c6;">POSTGRES_PASSWORD</span><span>: </span><span style="color:#f1fa8c;">""
</span><span> </span><span style="color:#ff79c6;">RAILS_ENV</span><span>: </span><span style="color:#f1fa8c;">"test"
</span><span> </span><span style="color:#ff79c6;">SELENIUM_URL</span><span>: </span><span style="color:#f1fa8c;">"http://selenium__standalone-chrome:4444/wd/hub"
</span><span>
</span><span style="color:#6272a4;"># Cache gems in between builds. We use the project path slug
</span><span style="color:#6272a4;"># as the key because one cache per project works well enough
</span><span style="color:#6272a4;"># for us
</span><span style="color:#ff79c6;">cache</span><span>:
</span><span> </span><span style="color:#ff79c6;">key</span><span>: </span><span style="color:#f1fa8c;">${CI_PROJECT_PATH_SLUG}
</span><span> </span><span style="color:#ff79c6;">paths</span><span>:
</span><span> - </span><span style="color:#f1fa8c;">vendor/ruby
</span><span>
</span><span style="color:#6272a4;"># Setup shell commands. We need nodejs for asset compilation,
</span><span style="color:#6272a4;"># And libgmp for the bcrypt gem. Then we override the database
</span><span style="color:#6272a4;"># configuration with our Gitlab configuration. Now, we can
</span><span style="color:#6272a4;"># install our gem dependences, bundle to the vendor folder so we # can cache them, and finally prep the database.
</span><span style="color:#ff79c6;">before_script</span><span>:
</span><span> - </span><span style="color:#f1fa8c;">apt-get update -q && apt-get install nodejs libgmp-dev -yqq
</span><span> - </span><span style="color:#f1fa8c;">cp config/database.yml.gitlab config/database.yml
</span><span> - </span><span style="color:#f1fa8c;">gem install bundler rubocop --no-ri --no-rdoc
</span><span> - </span><span style="color:#f1fa8c;">bundle install -j $(nproc) --path vendor
</span><span> - </span><span style="color:#f1fa8c;">rails db:schema:load
</span><span>
</span><span style="color:#6272a4;"># We have two jobs, first, we lint the project with Rubocop to
</span><span style="color:#6272a4;"># keep us honest and clean
</span><span style="color:#ff79c6;">rubocop</span><span>:
</span><span> </span><span style="color:#ff79c6;">script</span><span>:
</span><span> - </span><span style="color:#f1fa8c;">rubocop
</span><span>
</span><span style="color:#6272a4;"># Then we run our RSpec suite. When we run tests against
</span><span style="color:#6272a4;"># Selenium, Rails will save screenshots of failures. We capture
</span><span style="color:#6272a4;"># Them as artifacts so we can grab them through Gitlab's UI
</span><span style="color:#6272a4;"># later.
</span><span style="color:#ff79c6;">rspec</span><span>:
</span><span> </span><span style="color:#ff79c6;">script</span><span>:
</span><span> - </span><span style="color:#f1fa8c;">rspec spec
</span><span> </span><span style="color:#ff79c6;">artifacts</span><span>:
</span><span> </span><span style="color:#ff79c6;">when</span><span>: </span><span style="color:#f1fa8c;">on_failure
</span><span> </span><span style="color:#ff79c6;">expire_in</span><span>: </span><span style="color:#f1fa8c;">1 week
</span><span> </span><span style="color:#ff79c6;">paths</span><span>:
</span><span> - </span><span style="color:#f1fa8c;">tmp/screenshots/
</span><span> - </span><span style="color:#f1fa8c;">log/
</span></code></pre>
<p>And here’s the database config for running on CI. The CI is always a one-off, isolated environment, so we don’t need the typical project specific naming. So, we use generic names here so we can recycle this file as-is in any project. CI having it’s own clearly defined database configuration is one concession that’s proven worth it, especially when we can make it as easy as dropping in this one file.</p>
<pre data-lang="yaml" style="background-color:#282a36;color:#f8f8f2;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#6272a4;"># config/database.yml.gitlab
</span><span style="color:#ff79c6;">test</span><span>:
</span><span> </span><span style="color:#ff79c6;">adapter</span><span>: </span><span style="color:#f1fa8c;">postgresql
</span><span> </span><span style="color:#ff79c6;">encoding</span><span>: </span><span style="color:#f1fa8c;">unicode
</span><span> </span><span style="color:#ff79c6;">pool</span><span>: </span><span style="color:#bd93f9;">5
</span><span> </span><span style="color:#ff79c6;">timeout</span><span>: </span><span style="color:#bd93f9;">5000
</span><span> </span><span style="color:#ff79c6;">host</span><span>: </span><span style="color:#f1fa8c;">postgres
</span><span> </span><span style="color:#ff79c6;">username</span><span>: </span><span style="color:#f1fa8c;">runner
</span><span> </span><span style="color:#ff79c6;">password</span><span>: </span><span style="color:#f1fa8c;">""
</span><span> </span><span style="color:#ff79c6;">database</span><span>: </span><span style="color:#f1fa8c;">test_db
</span></code></pre>
<p>In our RSpec configuration, the only change we have to make is to check if we’re and use that instead of loading a local <code>chromedriver</code>. When using a remote Selenium we do need to tell Capybara where our app is for Selenium to connect to. We set that through the <code>host!</code> method provided by Rails’ SystemTestRunner. This was the key bit not covered by Gitlab’s examples or other articles we read on the topic.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#6272a4;"># spec/support/capybara.rb
</span><span>
</span><span style="color:#6272a4;"># Hide Puma start up notification
</span><span style="font-style:italic;color:#66d9ef;">Capybara</span><span style="color:#ff79c6;">.</span><span>server </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">:puma</span><span>, { </span><span style="color:#bd93f9;">Silent: true </span><span>}
</span><span>
</span><span style="font-style:italic;color:#66d9ef;">RSpec</span><span style="color:#ff79c6;">.</span><span>configure </span><span style="color:#ff79c6;">do </span><span>|</span><span style="font-style:italic;color:#ffb86c;">config</span><span>|
</span><span> config</span><span style="color:#ff79c6;">.</span><span>before(</span><span style="color:#bd93f9;">:each</span><span>, </span><span style="color:#bd93f9;">type: :system</span><span>) </span><span style="color:#ff79c6;">do
</span><span> driven_by </span><span style="color:#bd93f9;">:rack_test
</span><span> </span><span style="color:#ff79c6;">end
</span><span>
</span><span> config</span><span style="color:#ff79c6;">.</span><span>before(</span><span style="color:#bd93f9;">:each</span><span>, </span><span style="color:#bd93f9;">type: :system</span><span>, </span><span style="color:#bd93f9;">js: true</span><span>) </span><span style="color:#ff79c6;">do
</span><span> </span><span style="color:#ff79c6;">if </span><span style="color:#ffffff;">ENV</span><span>[</span><span style="color:#f1fa8c;">"SELENIUM_URL"</span><span>]</span><span style="color:#ff79c6;">.</span><span>present?
</span><span>
</span><span> </span><span style="color:#6272a4;"># Make the test app listen to outside requests, for the remote Selenium instance.
</span><span> </span><span style="font-style:italic;color:#66d9ef;">Capybara</span><span style="color:#ff79c6;">.</span><span>server_host </span><span style="color:#ff79c6;">= </span><span style="color:#f1fa8c;">'0.0.0.0'
</span><span>
</span><span> </span><span style="color:#6272a4;"># Specify the driver
</span><span> driven_by </span><span style="color:#bd93f9;">:selenium</span><span>, </span><span style="color:#bd93f9;">using: :chrome</span><span>, </span><span style="color:#bd93f9;">screen_size: </span><span>[</span><span style="color:#bd93f9;">1400</span><span>, </span><span style="color:#bd93f9;">2000</span><span>], </span><span style="color:#bd93f9;">options: </span><span>{ </span><span style="color:#bd93f9;">url: </span><span style="color:#ffffff;">ENV</span><span>[</span><span style="color:#f1fa8c;">"SELENIUM_URL"</span><span>] }
</span><span>
</span><span> </span><span style="color:#6272a4;"># Get the application container's IP
</span><span> ip </span><span style="color:#ff79c6;">= </span><span style="font-style:italic;color:#66d9ef;">Socket</span><span style="color:#ff79c6;">.</span><span>ip_address_list</span><span style="color:#ff79c6;">.</span><span>detect { |</span><span style="font-style:italic;color:#ffb86c;">addr</span><span>| addr</span><span style="color:#ff79c6;">.</span><span>ipv4_private? }</span><span style="color:#ff79c6;">.</span><span>ip_address
</span><span>
</span><span> </span><span style="color:#6272a4;"># Use the IP instead of localhost so Capybara knows where to direct Selenium
</span><span> host! </span><span style="color:#f1fa8c;">"http://</span><span>#{ip}</span><span style="color:#f1fa8c;">:</span><span>#{</span><span style="font-style:italic;color:#66d9ef;">Capybara</span><span style="color:#ff79c6;">.</span><span>server_port}</span><span style="color:#f1fa8c;">"
</span><span> </span><span style="color:#ff79c6;">else
</span><span> </span><span style="color:#6272a4;"># Otherwise, use the local machine's chromedriver
</span><span> driven_by </span><span style="color:#bd93f9;">:selenium_chrome_headless
</span><span> </span><span style="color:#ff79c6;">end
</span><span> </span><span style="color:#ff79c6;">end
</span><span style="color:#ff79c6;">end
</span></code></pre>
<p>When the app needs to make network requests, say to an external API, we use Webmock to stub those requests in testing. In strict mode, Webmock will raise an error if a request isn’t whitelisted or in your provided stubs. Capybara needs to be able to talk to the remote selenium, thankfully it’s easy to add it to the whitelist.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#6272a4;"># spec/support/webmock.rb
</span><span>
</span><span style="font-style:italic;color:#66d9ef;">WebMock</span><span style="color:#ff79c6;">.</span><span>disable_net_connect!(</span><span style="color:#bd93f9;">allow: </span><span>[</span><span style="color:#f1fa8c;">'localhost'</span><span>, </span><span style="color:#f1fa8c;">'127.0.0.1'</span><span>, </span><span style="color:#ff5555;">/selenium/</span><span>])
</span></code></pre>
<p><em>As a side note, breaking up our RSpec configuration into these initializer-style, single-purpose, drop-in configuration files makes juggling all these test suites much easier!</em></p>
<p>With this setup, we’ve been able to simply drop in these 3 files (<code>.gitlab-ci.yml</code>, <code>database.yml.gitlab</code>, and the updated <code>capybara.rb) </code>to several Rails 5.1+ projects and have a working CI pipeline.</p>
Deliver Paperclip and Pipeline Assets with Amazon CloudFront2017-04-27T00:00:00+00:002017-04-27T00:00:00+00:00https://cowie.me/posts/delivering-paperclip-and-pipeline-assets-through-amazon-cloudfront/<blockquote>
This post was originally written for <a href="https://www.neotericdesign.com">Neoteric Design</a>.
</blockquote>
<p>Content Delivery Networks (CDNs) can greatly speed up asset load times. We’re always looking for ways to improve the user experience, and page load times are the number one technical benchmark. Using a CDN speeds up things in a few ways. First, just by being on purpose-built servers with fantastic network capacity. Secondly, the content can be mirrored to different physical locations around the world, so that the bits don’t have to bounce across the globe. Thirdly, it frees up the application server to do its job handling business logic and talking to the database rather than simply serve up files. And while S3 can be used to serve images, CloudFront is cheaper and much more suited to the task.</p>
<h2 id="how-cloudfront-works">How CloudFront Works</h2>
<p>First, you create a <em>distribution</em>, essentially where the files will be served from. A distribution has one or many <em>origins</em>, for our purposes that’s either an S3 bucket or our Rails server and its assets folder. And for each origin, there is at least one <em>behavior</em>, which sets how files are slurped into the distribution. For an S3 origin, the default behavior is to cache everything; and notably, it’s done up front. For generic HTTP sources like a Rails project, it’s done lazily. Don’t think that’s a bad thing! Simply, as visitors’ browsers request assets, CloudFront first checks its cache, and if it’s not there or expired, it goes ahead and grabs it from the server, and onto the visitor and subsequent visitors.</p>
<figure class='full'>
<img src="https://cowie.me/processed_images/316c5e654adea00b00.webp" alt="Diagram of communications between your app and cloudfront" width="700" height="108"/>
</figure>
<h2 id="set-up">Set up</h2>
<p><a rel="noopener" target="_blank" href="http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-creating.html">Create your distribution</a>. If you’re using S3 to store Paperclip uploads, add that bucket as an origin. Paperclip uses query strings for cache busting, so it’s recommended to forward those, otherwise, the default settings should work. To follow <a rel="noopener" target="_blank" href="https://12factor.net/">12 Factor</a> and for simplicity, I recommend storing the distribution’s hostname in an environment variable, e.g CLOUDFRONT_ENDPOINT. It’ll be referred to in a few different place in Rails’ configuration, so it’s good to set up a single source of truth.</p>
<h2 id="configuring-for-paperclip-uploads">Configuring for Paperclip uploads</h2>
<p>Assuming you’ve got S3 storage for Paperclip already configured, you just need two additional settings, <code>s3_host_alias</code> pointing to the distribution’s hostname, i.e xxxxxxxx.cloudfront.net, and <code>url</code> overridden to Paperclip’s special flag :s3_alias_url, that will make sure Paperclip::Attachment#url returns the proper CloudFront URL.</p>
<p>Here’s a simple snippet that can be safely appended after your existing configuration.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#6272a4;"># config/initializers/paperclip.rb
</span><span>
</span><span style="color:#6272a4;"># ...
</span><span>
</span><span style="color:#ff79c6;">if </span><span style="color:#ffffff;">ENV</span><span>[</span><span style="color:#f1fa8c;">'CLOUDFRONT_ENDPOINT'</span><span>]
</span><span> </span><span style="font-style:italic;color:#66d9ef;">Paperclip</span><span style="color:#ff79c6;">::</span><span style="font-style:italic;color:#66d9ef;">Attachment</span><span style="color:#ff79c6;">.</span><span>default_options</span><span style="color:#ff79c6;">.</span><span>merge!(
</span><span> </span><span style="color:#bd93f9;">s3_host_alias: </span><span style="color:#ffffff;">ENV</span><span>[</span><span style="color:#f1fa8c;">'CLOUDFRONT_ENDPOINT'</span><span>],
</span><span> </span><span style="color:#bd93f9;">url: </span><span style="color:#f1fa8c;">':s3_alias_url'
</span><span> )
</span><span style="color:#ff79c6;">end
</span></code></pre>
<p>That’s it really! Assuming the distribution has finished processing your bucket, you should be good to go. For reference here’s one of our common default Paperclip configurations, note the cache control headers.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="font-style:italic;color:#66d9ef;">Paperclip</span><span style="color:#ff79c6;">::</span><span style="font-style:italic;color:#66d9ef;">Attachment</span><span style="color:#ff79c6;">.</span><span>default_options</span><span style="color:#ff79c6;">.</span><span>merge!(
</span><span> </span><span style="color:#bd93f9;">:storage </span><span>=> </span><span style="color:#bd93f9;">:s3</span><span>,
</span><span> </span><span style="color:#bd93f9;">:bucket </span><span>=> </span><span style="font-style:italic;color:#66d9ef;">ENV</span><span style="color:#ff79c6;">.</span><span>fetch(</span><span style="color:#f1fa8c;">'S3_BUCKET'</span><span>, </span><span style="color:#f1fa8c;">'project-default-bucket'</span><span>),
</span><span> </span><span style="color:#bd93f9;">:path </span><span>=> </span><span style="color:#f1fa8c;">"/system/:class/:attachment/:id/:style/:filename"</span><span>,
</span><span> </span><span style="color:#bd93f9;">:s3_protocol </span><span>=> </span><span style="color:#f1fa8c;">'https'</span><span>,
</span><span> </span><span style="color:#bd93f9;">:s3_region </span><span>=> </span><span style="color:#f1fa8c;">'us-east-1'</span><span>,
</span><span> </span><span style="color:#bd93f9;">:s3_headers </span><span>=> {
</span><span> </span><span style="color:#f1fa8c;">'Cache-Control' </span><span>=> </span><span style="color:#f1fa8c;">'max-age=315576000'</span><span>,
</span><span> </span><span style="color:#f1fa8c;">'Expires' </span><span>=> </span><span style="color:#bd93f9;">10</span><span style="color:#ff79c6;">.</span><span>years</span><span style="color:#ff79c6;">.</span><span>from_now</span><span style="color:#ff79c6;">.</span><span>httpdate
</span><span> }
</span><span>)
</span></code></pre>
<h2 id="configuring-for-the-asset-pipeline">Configuring for the Asset Pipeline</h2>
<p>Setting up CloudFront for a generic server takes a little more effort since it has to know a little more about it. First, create an origin with the production domain. Then, create a behavior. To cache asset pipeline requests, set the path pattern to /assets/*. The other vital portion configuring what HTTP headers to forward. This will vary depending on your needs, but you’ll at least want to forward Access-Control-Allow-Origin, for the bundle of joy that is <a rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Cross-origin_resource_sharing">CORS</a>, otherwise, browsers may refuse to load your assets if the domain isn’t whitelisted.</p>
<h3 id="rails-configuration">Rails Configuration</h3>
<p>First, set the asset host to our CloudFront endpoint, this will prepend all of the asset_path, image_path, etc helpers with the CloudFront subdomain. Rails is still serving the assets, as you can see if you navigate to them manually, but visitors only see the CloudFront URLs, and CloudFront will hit the Rails server to grab an asset if needed.</p>
<p>Remember the CORS headers that were whitelisted? This is where they’ll come from. The cache-age can be cranked up because the Asset Pipeline fingerprints the filenames, so old versions of assets simply won’t be linked to, and will eventually be purged from CloudFront, no intervention necessary.</p>
<pre data-lang="ruby" style="background-color:#282a36;color:#f8f8f2;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#6272a4;"># config/environments/production.rb
</span><span>
</span><span>config</span><span style="color:#ff79c6;">.</span><span>action_controller</span><span style="color:#ff79c6;">.</span><span>asset_host </span><span style="color:#ff79c6;">= </span><span style="color:#ffffff;">ENV</span><span>[</span><span style="color:#f1fa8c;">'CLOUDFRONT_ENDPOINT'</span><span>]
</span><span>
</span><span>config</span><span style="color:#ff79c6;">.</span><span>public_file_server</span><span style="color:#ff79c6;">.</span><span>enabled </span><span style="color:#ff79c6;">= </span><span style="color:#bd93f9;">true
</span><span>config</span><span style="color:#ff79c6;">.</span><span>public_file_server</span><span style="color:#ff79c6;">.</span><span>headers </span><span style="color:#ff79c6;">= </span><span>{
</span><span> </span><span style="color:#f1fa8c;">'Cache-Control' </span><span>=> </span><span style="color:#f1fa8c;">'public, max-age=2592000'</span><span>,
</span><span> </span><span style="color:#f1fa8c;">'Access-Control-Allow-Origin' </span><span>=> </span><span style="color:#f1fa8c;">'*'</span><span>,
</span><span> </span><span style="color:#f1fa8c;">'Access-Control-Allow-Methods' </span><span>=> </span><span style="color:#f1fa8c;">'GET, HEAD'</span><span>,
</span><span> </span><span style="color:#f1fa8c;">'Access-Control-Allow-Headers' </span><span>=> </span><span style="color:#f1fa8c;">'*'</span><span>,
</span><span> </span><span style="color:#f1fa8c;">'Access-Control-Max-Age' </span><span>=> </span><span style="color:#f1fa8c;">'1728000'
</span><span>}
</span></code></pre>
<p>This works for using Puma on Heroku, but if you’re serving static files some other way, through Apache, NGINX, or Passenger Standalone, you can leave the Rails public file server disabled and configure things at that level.</p>
<h2 id="deploy">Deploy</h2>
<p>If CloudFront is finished processing your bucket, you ought to be ready to deploy the Rails’ configuration. Paperclip attachments should now be giving the CloudFront URL, and stylesheets, javascript, fonts, and images from the Asset Pipeline should have the CloudFront host as well. Some things to check for when troubleshooting:</p>
<ul>
<li>In the views/stylesheets that the you’re using the <a rel="noopener" target="_blank" href="http://guides.rubyonrails.org/asset_pipeline.html#coding-links-to-assets">proper helpers</a>.</li>
<li>The distribution is ready and not InProgress</li>
<li>The CORS headers, specifically the origin</li>
<li>The file truly exists on your production domain.</li>
</ul>
<p>While it does add complexity to a server set up, CloudFront largely Just Works(<sup>tm) </sup>after it’s configured. And it becomes less daunting to set up once you understand its components.</p>