Building Hybrid Apps with Rails - a Case Study

I gave my first tech talk at my local ruby meetup, it was all about my experience with building hybrid mobile apps with Rails (using turbolinks 5).

Take a look.

Getting into Rails Services

So the latest Has It Shipped Yet has been running in the wild for just a little over two weeks now and has tracked close to 4500 orders. Which is pretty awesome, but at the same time a tad bit scary!

This past week, I’ve dedicated to take a step and focus on stability rather than pushing ahead and adding new features.

Part of my stability plan was to introduce a proper test suite. I jumped into this app, without paying too much attention to writing test cases and instead focused on manual testing and in some cases just using the logger.debug method in Rails.

It worked at first, but it didn’t lend me the peace of mind I really wanted.

Luckily for me, Chris Kottom published an awesome book on working with Minitest (aka Minitest Cookbook). (Why Minitest - I mostly wanted to pickup the default testing tool for Rails, and I wanted to focus on improving my Ruby chops - but whatever makes you a better tester will work too).

After jumping around a bit, I’ve been able to add around 27 different tests. Most of them account for my Shop model (which is a tad bit harry at the moment), but I’ve now started to dive into testing my Jobs.

One of the things I noticed was that my Jobs did way to much.

For example, when a new Shop signs up and clicks “Finish Setup”, I fire a new job called: FinishShopSetupJob.

The main purpose of the job is to “Finish Setting Up a Shop”, which includes: installing the remaining webhooks, creating a new page on their shop, pulling some extra shop details, and some more. All these tasks fall under the service of setting up a shop.

Initially it kinda made sense, I need this Job to finish setting up this Shop but once I started looking into testing, suddenly it felt really bloated.

Technically, this Job is responsible of creating a new Job adding it to a queue, talking to the setup queue, and finishing up the actual task at hand.

So really, it’s trying to do two things - Job related stuff and everything else required to finish setting up a shop.

After a little bit of research, I was able to come across the design pattern consisting of service objects. This article helped quite a bit.

My goal is to port over my existing job and turn it into a service object so I don’t have so much complicated logic stuck in my job.

I’ll report back how my refactoring went.

Utilizing one assert per test

One of the problems I’ve been having is trying to keep just to a single test.

For example, I have a Shop model that I wanted to make sure some defaults are set properly.

Originally, I thought it was best to just test if the defaults were all set. This to me sorta made sense.

test "newly created shop should have initial defaults set properly" do
  s = Shop.create(shopify_domain: 'testshop1.myshopify.com', shopify_token: 'somethingsupersecure')
  assert s.valid?
  assert s.install_required # my defaults I'm trying to test for
  assert s.setup_required
end

But I came across this post from 2007 by Jay Fields. That outlined an argument for just having a single assert per test.

The biggest argument against multiple asserts is the fact that you find out all your failing tests right away. Instead you might need to fix the first bug in order to uncover the next.

So here’s what my new tests look like:

test "newly created shop should be valid" do
  s = Shop.create(shopify_domain: 'testshop1.myshopify.com', shopify_token: 'somethingsupersecure')
  assert s.valid?
end

test "newly created shop should have install_required true" do
  s = Shop.create(shopify_domain: 'testshop1.myshopify.com', shopify_token: 'somethingsupersecure')
  assert s.install_required
end

test "newly created shop should have setup_required true" do
  s = Shop.create(shopify_domain: 'testshop1.myshopify.com', shopify_token: 'somethingsupersecure')
  assert s.setup_required
end

Did not expect key while parsing yaml

I’ve been in the process of building out a proper test suite to Has It Shipped Yet, and one of the problems I ran into was fixing a small yaml parsing error.

In particular this error:

did not expect key while parsing yaml

Unfortunately when running rake test I was met this this cryptic error:

Psych::SyntaxError: (<unknown>): did not find expected key while parsing a block mapping at line 10 column 2

Luckily I was able to stumble upon this GitHub issue.

It turns out that Psych doesn’t have the greatest error messages, so I looked out for a simple yaml validator and case across this Code Beautify’s YAML Validator.

I was able to narrow done the issue down to the line number.

Hope that helps.

Rails Migration. Up and down vs change.

So I’m starting to plan out my migrations. I’ve already but a database schema for Has It Shipped Yet, so I want to port it over to rails.

Since I’m working with Shopify, I thought it was best to include their ShopifyApp gem that allows you to focus on building your app and not worry about things like getting OAuth setup.

I noticed though, in their migration they used self.up and self.down when making the shop migration.

It’s a bit weird (for me at least) because I’ve only ever seen the change method.

With a bit of searching, I was able to find this Stack Overflow post.

This gist was that up/down methods were used in Rails 3. They were used originally because Rails didn’t know how to rollback the migration. For example if you accidentally added the wrong field, you wanted to rollback but it wasn’t sure how to proceed.

With Rails 4 (I’m using the current stable release), you can usually just consider using the change method.

The Rails Guide talks about it nicely here.

I think for the fields I’ll be adding. Change should work just fine.

The Stack

Side: This is a post in my on going series about Rewriting Has It Shipped Yet in Go.

I have a bit of an obsession with software stacks. Usually it’s just a matter of hitting the BuiltWith icon on Chrome and checking out which framework tokens are being used, but sometimes I go straight to the source - the career’s page.

This weekend I’ve been recovering from a bit of burn out from working on Has It Shipped Yet. It’s not too much but I’m starting to feel the pain of the stack I chose.

Originally, I thought Go was the perfect decision. It just made sense. Think about it, I want to display this widget on someone else’s store without worrying about the performance hit, and I just need to expose a single JSON endpoint.

But, things started to break down. To support that single endpoint I needed to create several more endpoints to listen for webhook requests, I need to interface with a MySQL DB (luckily for sqlx), I needed good background processing support since I have several long processing jobs to run i.e. fetching product images from Shopify’s product API.

This was all pretty manageable until I ran into working with background jobs. So far I’ve used Job Runner, but it’s pretty limited (there isn’t a good way to try a retry a failed job).

I thought that it might be best to look at introducing beanstalkd to the mix, but then I started thinking about moving away from Go altogether (at least for the main portion of my app - maybe I’ll use it for a tiny microservice).

But where to?

I’ve been attending a local Ruby meetup each month and it’s been really amazing. The ruby community is extremely welcoming (last meetup we were joined by the local Python meetup).

I’ve also had a soft spot for rails. Despite this past year of me wanting to avoid it at all costs - because I wanted to write my own SQL instead of using an ORM, I’m finding that this was a huge mistake.

Has It Shipped Yet, is pretty much a CRUD app plus I’m never going to run into Twitter style scale problems, so it shouldn’t be an issue.

Plus by moving to rails I can see a few noticeable benefits:

• It’s old & mature: Now that I have a couple of paying customers, I want to deliver a better more stable product.
• It’s community is huge
• Ruby is designed for programmer happiness: This is mostly a side project to test the waters. I really want something that is a joy to program with.
• Rails is constantly improving: With Rails 5 is almost out of beta and I’m pretty interested in learning more about ActionCable
• Convention over Configuration: I’m pretty excited about this. Rails does have a bit of a steeper learning curve, but once I get going things should be released much sooner.
• Better Resale Value: I’m not heading into this for money, but if the time does come. I rather not have technology be a limiting factor to the sale.
• Testing Frameworks: One of my struggles was figuring out testing my app from top of bottom (integration tests). I’m looking forward to see what the community has in place.

So I’m going to rewrite Has It Shipped Yet again - but this time (and hopefully the last time) it will be done using rails.

I’ll be writing about it along the way - so maybe if you run into some beginner rails problems we can hash them out together.

Till next time.

Has It Shipped Yet - 2016 Update

Side: This is a post in my on going series about Rewriting Has It Shipped Yet in Go.

Well it’s been a while since I updated this blog. I’ve mostly been bogged down with exams and getting Has It Shipped Yet 2 out the door.

Some good news - it’s out in the wild and my installs are slowly going up. But what I’m really excited about is that there hasn’t been a single uninstall yet (not counting the Shopify App team)!

I’m hoping it has to do with the new on boarding email that I send out with the install - that and the installation welcome video (both kinda rough, but at least it’s something).

So where to next?

Well I’m planning on moving from the jobrunner package to beanstalkd. The main motivation for this move is pretty simple - I don’t have a good way of knowing if a background job is running or not. I also don’t have a way to try broken jobs.

I’m also planning on beefing up my Shopify SDK. It’s really quite rough and has some serious problems - I wouldn’t even call it a SDK. My goal is to try my best to clean it up before the official release. That way I can have improved error logging.

Finally, I’m planning on building my own API service that will will handle package tracking from USPS, UPS, Canada Post, and Royal Mail (I’m find that more people from the UK are installing my app). I tried to stick with just using EasyPost, but it ultimately turned out to be too unreliable. The alternative (being AfterShip) is simply too costly for my usage (it’s hard to offer an unlimited plan when you’re paying 5 cent’s per package).

Where’s this series going?

Well, I’m hoping to get back into blogging building Has It Shipped Yet - but I’m also planning on blogging about Trolley (my Shopify app incubator).

My main goal is to work out problems in the public. I’m finding that it forces me to produce better more thoughtful work.

Till next time.

Planning Out my Routes

Side: This is a post in my on going series about Rewriting Has It Shipped Yet in Go.

Planning Out my Routes

So for the next version of Has It Shipped Yet (HISY), I’m incorporating a new feature - theming.

Originally, I hosted my JS widget on S3 and it was just static. It’s main purpose was use jQuery & talk to my API.

This worked pretty well, but everything had to be hardcoded.

With this version, I’m thinking of building a new endpoint that would host a dynamically built JS widget.

This would allow me to hold their customizations and support multiple third-party libraries on an as needed basis.

The Routes

Dynamically embedded widget:

/embed/{store_name}.js

Public API Routes:

/api/{store_name}/orders/{order_id}/lookup
  - Params: Email ([email protected])

I’m still not 100% sure about including version support.

Mostly because no one would be building directly against my API. Plus, one issue I found was that most of my users had problems with installing/modifying a page’s source code. So if I did want to create a new version, I would either need to build a new app, or try to build some sort of upgrader.

Widget Script

On installation, I would add the widget like this:

<script src="https://hasitshippedyet.com/embed/not-your-mothers-buttons.js">

<div id="hasitshippedyet-embed"></div>

From there, the widget would communicate to my API and and then depending on the theme chosen would render out a response.

Rewriting my Web App in Go

Side: I’m still relatively new to Go (Golang), and a bit new to building web apps, but I was able to successfully ship the first version of Has It Shipped Yet (HISY) over the summer. Now with more customer feedback and better go skills, I’m rewriting the next version from the ground up. If you spot something that’s a bit wonky or just have a better way of doing things, please let me know in the comment section below.

Rewrite

Although Joel Spolsky (from Joel on Software) has a great article about how you should never do a rewrite, I was inspired after watching DHH’s talk on Rewriting Basecamp 2 & now 3.

The current version is pretty buggy and at times, I’m a bit shocked that’s it’s still chugging away.

Instead of trying to get in there and change up somethings, I’m going in for a complete rewrite.

Since the first version, I was able to work on my next app Persistent Discounts, originally I had plans on launching it (it did make it to the beta phase), but it lacks too many features to be meaningful. Instead I hope to port over some of it’s technology in the next “larger” app I build.

But, building that app, gave me some really good insights into building a embedded Shopify app, and I was able to build my own middleware for logging errors (largely thanks to elithrar to his excellent post on handling errors.)

Reinventing the MVC Wheel

The one thing I’ve notice building things web apps in Go, is trying to reinvent the MVC wheel. Before digging into Go, I was in a large part an avid Rails noob (I still attend my local ruby meetup, because there an awesome group of experienced devs), so when I dig into Go I attempt to badly copy over rails MVC style.

Just to give you an idea, here is part of my “routes” file:

// Campaign - Index
mx.Handle("/campaigns", m.Authenticate(Handler{env, campaigns.HandleIndex})).Methods("GET")

// Campaign - Create
mx.Handle("/campaigns", m.Authenticate(Handler{env, campaigns.HandleCreate})).Methods("POST")

It somewhat follows Rail’s Single Routing.

If you notice I wrap things in a m.Authenticate. This was because I needed a way to support CSRF protection and user authentication. Originally, I planned not to include this, but ran into some weird bugs on my admin panel.

Anyway, going forward, I’ve figured out a nice little balance between full MVC rails land and something light weight that involves a tad bit more code, but much easier to manage. (I’ll be sharing it when it comes time to get the project up and running.)

Getting Started

Here’s the mission statement of Has It Shipped Yet? “Answer just one question, has my package shipped yet.”. In a more complex way - Help Shopify stores build trust by helping their customers know that their package has been fulfilled and is on their way.

If you’ve ever bought anything online, you might have run into the issue of trying to track down your package. Having just a tracking number works okay when it’s just one item, but when things are fulfilled separately it gets a bit overwhelming. Instead, it would be nice to visit just one site and see the status in one place. Etsy & American Eagle do this pretty well.

Lessons learned from V1

Because I never shipped a Shopify App before, I wasn’t sure how user friendly I would need to make things. I learned pretty fast when the support requests started to flood in:

• Theming is a huge pain! Don’t expect your users to understand how to code things with HTML. It’s not going to happen! Stick to WYSIWYG.

• Your users aren’t just based in North America. It’s time to support multiple languages, currencies, and shipping carriers.

• Spend time getting onboarding right. You’ll get too many uninstalls otherwise.

• Build an email onboarding course too. Let your users get to know who you are. (Build Trust)

• Always be building trust. Always! This is a huge common theme in the Shopify community. Broken Apps, and broken support. It’s easy to forget that your user actually depends on your App to run their business. Don’t forget that!

• Take advantage of webhooks. Right now V1 doesn’t support the uninstall webhook, which caused a bunch of pain when it came to installation issues.

• Invest time making your App Embeddable. It’s easy to have an external link to some other site, but it does kill your user experience (especially if your App is more of an add-on to Shopify).

Committed Features

• Introduce support for theming

• Add support for a third-party shipping API tracking service (in my case EasyPost is affordable and covers a large range of carriers)

• Build a better onboarding flow

• Build support for email course (make it clear that I’m here to help)

• Introduce a sustainable billing model. Right now I’m free, most Apps tend to offer little but charge quite a bit. My hope with theming, I can offer a range of pricing plans (free included) that work for most stores.

The hardest one will probably be theming. If you ever need a great book on where to start with building widgets checkout out Third-Party JS. It’s definitely helped improve my JS skills.

The Timeline

My original idea was shipping V2 by mid-October. But because I fell behind with my other App, and ran into a heavier course load with school, V2 will probably not be out until late-November or December. My goal is to invest more in the user experience instead of just shipping right away. V1 is still running despite it’s bugs, so I have more time to get it right this time.

With V2, I really want to write about the process. This way I can help others building their Apps in Go, and also since I’m working alone I find writing helps with hashing out ideas and figuring out problems.

Plus it would be nice to improve my writing skills too.

How Should I Structure My Golang Web App?

So I just launched and deployed Has It Shipped Yet. A simple Shopify order tracking app.

I’ve always found that the best way to actually pick up a language is to build something - particularly something real, that doesn’t get thrown away.

I decided to structure the app like this:

main.go <-- holds the server setup code
/app    <-- all the app code
    /router      <-- just a single route file
    /controllers <-- handles HTTP calls
    /models      <-- interacts with my DB
    /views       <-- holds all my templates (grouped by controllers)
    /helpers     <-- holds my external API code
    /public      <-- anything static
    database.go  <-- central db connect
    logger.go    <-- sets up my logger

I wanted to use an MVC style setup, which was largely borrowed from developing a handful of Rails apps.

When it came to deploying, I found my templates to be kinda messy. Basically, I needed to recreate this structure just for my templates to be parsed.

For my next project (Trolley). I think I’m going to try something like this reddit commenter said:

.../project/main.go = executable, only Go file here
.../project/db/... = Go code for database access
.../project/web/... = Go code for web (dispatch, handlers)
.../project/model/... = Go code for models
.../project/(other)/... = Go code for other things (several of these)
.../project/templates/... = HTML templates (compile-time)
.../project/deb/... = .deb file structure
.../project/vendor/... = vendor directories (jquery, etc.)

The beauty with Golang is that you’re not tied down to an opinionated framework, so you can always make things a bit leaner.

I’m always fascinated by how others structure their Go projects. How do You structure yours?