open source Archives - {5} Setfive - Talking to the World

When Amazon Web Services rolled out their version 4 signature we started seeing sporadic errors on a few projects when we created pre-authenticated link to S3 resources with a relative timestamp. Trying to track down the errors wasn’t easy. It seemed that it would occur rarely while executing the same exact code. Our code was simply to get a pre-authenticated URL that would expire in 7 days, the max duration V4 signatures are allowed to be valid. The error we’d get was “The expiration date of a signature version 4 presigned URL must be less than one week”. Weird, we kept passing in “7 days” as the expiration time. After the error occurred a couple of times over a few weeks I decided to look into it.

The code throwing the error was located right in the SignatureV4 class. The error is thrown when the end timestamp minus the start timestamp for the signature was greater than a week. Looking through the way the timestamps were generated it went something like this:

Generate the start timestamp as current time for the signature assuming one is not passed.
Do a few other quick things not related to this problem.
Do a check to insure that the end minus start timestamp is less than a week in seconds.

So a rough example with straight PHP could of the above steps for a ‘7 days’ expiration would be as follows:

Straight forward enough, right? the problem lies when a second “rolls” between generating the `$start` and the end timestamp check. For example, if you generate the `$start` at `2017-08-20 12:01:01.999999`. Let’s say this gets assigned the timestamp of `2017-08-20 12:01:01`. Then the check for the 7 weeks occurs at `2017-08-27 12:01:02.0000` it’ll throw an exception as duration between the start and end it’s actually now for 86,401 seconds total. It turns outs triggering this error is easier than you’d think. Run this script locally:

That will throw an exception within a few seconds of running most likely.

After I figured out the error, the next step was to submit a issue to make sure I’m not misunderstanding how the library should be used. The simplest fix for me was to generate the end expiration timestamp before generating the start timestamp. After I made the PR, Kevin S. from AWS pointed out that while this fixed the problem, the duration still wasn’t guaranteed to always be the same for the same relative time period. For example, if you created 1000 presigned URLs all with ‘+7 days’ as the valid period, some may be 86400 in duration others may be 86399. This isn’t a huge problem, but Kevin made a great point that we could solve the problem by locking the relative timestamp for the end based on the start timestamp. After adding that to the PR it was accepted. As of release 3.32.4 the fix is now included in the SDK.

I’m pretty bullish on Bitcoin so a few months ago I set out to build a “pure” Bitcoin related application. Specifically, I was looking to build an application that leveraged the Bitcoin network directly, without using any third party APIs or services. The goal behind avoiding third party services was to explore how difficult using the Bitcoin network directly is and also to embrace Bitcoin’s decentralized nature and not rely on another company to move coins.

Conceptually the way the Bitcoin network works is relatively straightforward. You move coins by creating transactions which are just messages written and cryptographically signed in a specific format and then you listen for transactions which include your addresses to keep your balance up to date. Of course, the devil is in the details and there’s a dauntingly large number of them. For example, Ken Shiriff explains how to craft a transaction by hand in Bitcoins the hard way: Using the raw Bitcoin protocol and it’s no easy read. Given that just crafting transactions involved so much code, I started researching existing open source libraries that facilitate working with Bitcoin.

After doing some research, it looked like the most popular approach to interfacing with the network directly was to run the bitcoind deamon and then make RPC calls to the exposed functions. Objectively, using RPC calls to bitcoind qualifies as a “pure” solution but I still didn’t love it. After a bit more searching, I came across bitcoinj which is a pure Java library for working with Bitcoin.

Unlike bitcoind, bitcoinj is a library so its designed to be embedded in other codebases and it supports simplified payment verification (SPV) which allows it to operate without downloading the entire blockchain, ~25GB as of today. On top of this, its written in Java so it’s easy to use from Scala, something I’d been looking to experiment with.

So what are some key takeaways from using bitcoinj?

Test on the TestNet: Although debugging new code with live money would be thrilling. the TestNet allows you to develop against the Bitcoin network without using “live” coins. With bitcoinj, if you instantiate TestNet3Params you’ll connect into the Testnet.
TestNet Fountains: Testing is pretty tough without some coins but luckily you can use fountains to grab some. http://faucet.xeno-genesis.com/ worked well for me to grab some coins. Just a heads up, TestNet transactions take awhile to get broadcast and included into the Blockchain so be patient.
Extend WalletAppKit: WalletAppKit is a utility class which helps bootstrap everything you need for a SPV wallet and will save you some headaches. I started off trying to hand roll everything and quickly switched over to extending WalletAppKit.
Chain API: Since the Chain API can be used on the TestNet it’s useful for testing your own applications. Specifically, Bitcoinj treats sending and receiving coins to “yourself” weirdly so it’s more effective to use Chain to test transactions.
Coin.parseCoin: Sorting out how many Satoshis you want to send is surprisingly difficult since sendCoins accepts a Coin object. Luckily, Coin.parseCoin will accept a BTC value and return representative Coin object.
Blockchain downloads and wallet start up slow down development: Even in SPV mode, it still takes a non-trivial amount of time to “catch up” if your client hasn’t been running for awhile. On top of that, starting up the wallet takes a few minutes since it has to discover and connect to beers. What this translates to is resuming development after being away for a while is slow and the “edit, compile, test” loop isn’t as tight as usual.
It’s real money: This should go without saying but once you’re running on the live Bitcoin network you’re playing with real money so be careful. If your keys get out anyone will be able to move your coins out of your addresses. So be careful what you check in on git and if you’re building something public definitely take a second to lock down your server.

Anyway, this was my first time building something Bitcoin related and it was a positive experience. The project is still private but I’ll definitely share it once it’s released. As always, questions or comments are welcome!

Last week we were looking to leverage a set of JSON API endpoints in a Symfony2 project to power a single page Javascript app. The way the API had been setup, the routes were all secured with an HTTP Basic Auth firewall matching on “/api”. This worked great for the mobile apps but for a Javascript app it would be awkward to have the user re-enter their credentials to authorize the basic auth firewall. What we really wanted to do was to leave everything “as is” but have Symfony use the normal cookie based firewall when we passed in a special “isOnlineApp” parameters on the URL.

Unfortunately, setting something like this up with the default “pattern” setting in your security.yml file isn’t possible. The “pattern” setting only matches on the route URL, not the parameters so there’s no way to have it selectively trigger when a parameter is present on a URL. So how do you do it? Well as it turns out, there’s a firewall configuration called “reuqest_matcher” which lets you “match” a firewall using a service. Just create a service that extends the RequestMatcherInterface, implment a “matches” function, and then add the class as a service.

Our code for the service ended up looking like:

And then the actual firewall configuration ends up being:

You don’t need a “pattern” setting anymore since the “matches” function supersedes it. Anyway, let me know if you have any questions!

At the beginning of the summer we decided to redo our website. The design on the old site was looking a bit dated and more importantly the content didn’t really reflect the types of projects we’re looking to work on. From a technology perspective, our old site was built on WordPress with the explicit goal of being able to share the same WordPress theme as our blog. The two sites did in fact share the same theme but looking back, we never updated the main site to really make it “worth it”. With that experience in mind, we started looking around for what we could use to build setfive.com.

Stepping back and looking at our requirements, we really don’t need a CMS. I’d argue this holds true for most website projects when there’s less than 20 pages, everyone who might edit it is technical, and the content isn’t updated frequently. Specifically looking at some major WordPress features, we don’t need the WYSIWYG editor, plugin ecosystem, media handling, or theming capabilities. So what capabilities do we need?

Routing / Pretty URLs:
Serving “raw” URLs like “about.php” hasn’t been OK since IE6 so this is obviously a “must have”. Basically, we need some way to map human readable URLs to specific pieces of content. Sure, you could do all of this with mod_rewrite or nginx’s location directive but that just sounds awkward. Whatever we pick should be able to handle these translations internally.
A modern templating engine:
PHP “grew up” as a web templating system but today it’s missing some key features in comparison to purpose built templating engines. Being handicapped by only having “require” or “include” and being forced to set global variables in 2014 is terrible so “real” templating should be a requirement.
Access to a CGI/scripting language:
I’m sure there’s “razor” of some sort describing a phenomenon where eventually a “static” website will need access to dynamic capabilities. For us, from the outset we knew we’d need to do things like pull in RSS feeds and send contact emails so we knew we needed access to some sort of CGI.

There’s certainly more capabilities static websites could need but I think this is a decent list for the “general” case and it captures our requirements. After doing some research, it looks like there’s currently a few options that would satisfy these requirements:

Static site generators: Generally, products like Jekyll or Phrozn will “compile” a set of your templates into static HTML pages which would then just be served by Apache or nginx. These solutions are fine except that they’d make things like integrating a RSS feed awkward.
“Lightweight” CMS: There’s a slew of “lighter” WordPress alternatives like Apostrophe or Ghost that check all the boxes but come with their own issues. With any “platform” you’d be back to learning the quirks of another platform and of course keeping the software up to date.
Use a micro-framework: Frameworks like Sinatra or Silex would offer the templating and routing features found in a “full framework” without the associated code footprint. In my opinion, this option offers the best combination of familiarity, extensibility, and a small enough footprint that updates shouldn’t be a huge issue.

I ultimately chose Silex because our team has deep PHP experience, especially with Symfony2. Because of that we’d be right at home with the Routing component and of course Twig for templating.

OK so how do you actually get this to work? I ran across Jonathan Petitcolas’s Building a static website with Silex post and used it as a guide. Here are the actual commands you’d need to get this all setup though:

Now, you just need to create a file named “index.php” which contains:

And finally, in the “views” directory add a file called “index.html.twig” which contains some content. If you have a web server setup, just point a vhost at the “web” directory, load it, and you should see the content of your index file.

If you don’t have a web server setup, a nifty trick via Gonzalo Ayuso, create a in the “web” directory named “router.php” containing:

And now, you can start the built in PHP 5.4+ server by running:

You can load your Silex app by loading http://localhost:8888 in your browser.

Anyway, as always questions and comments are welcome!

I was building out an API test console a few days ago and realized I’d never actually looked into how to grab all available routes in Symfony2. The “console” is basically a form with a select box and textarea that lets you “ping” the REST API routes in one of our applications. To make this work, I wanted to traverse all the registered routes, filter for the ones that contained “api_”, and then generate dummy URLs for those routes.

I searched around a bit for how to grab all the registered routes and the only link seems to be https://gist.github.com/hubgit/3380250 Unfortunately, if you try and use the code you’ll discover that “getPattern” no longer exists in the CompiledRoute class. It looks like it’s been replaced by getPathVariables

So, working code to generate a list of route names and “dummy” URLs for you API routes ends up looking like:

Tag: open source

An Edge Case of Time in AWS PHP SDK

Bitcoin: Tips for building a native Bitcoin app

So what are some key takeaways from using bitcoinj?

Symfony2: Using “request_matcher” for custom firewall rules

setfive.com: Why you should build a static site with Silex

Symfony2: Get all available routes in an app