Category: General

On a project we were working on recently it appeared that we had data coming into our Extract, Transform, Load (ETL) processes which should have been filtered out. In this particular case the files which we imported only would exist at max up to 7 days and on any given day we’d have tens of thousands of files that would be created and imported. This presented a difficult problem to trace down if something inside our ETL had gone awry or if we were being fed bad data. Furthermore as the files always would be deleted after importing we didn’t keep where a data point was created from.

Instead of updating our ETL process to track where a specific piece of data originated from we wanted to basically ‘grep’ the files in S3. After looking around it doesn’t look like anyone has built a “Grep for S3”, so we built one. The reason we didn’t simply download the files locally and then process them one at a time is it’d take forever to transfer, then grep each one individual sequentially. Instead we wanted to do the search in parallel and not hold the entire files on the local disk.

With this we came up with our simple S3Grep java app (a pre-built jar is located in the releases) which will search all files in a specific bucket for a specific string. It currently supports both regex or non-regex search strings. You can specify how many threads you want it to use to process the files or it by default will try to use the same number of CPU’s on your machine. It utilizes the S3 Java adapter to read the files as a stream rather than a single transfer, than read from disk. Using the tool is very simple:

A the file is a config file where you setup what you are searching for. An example:

For the most part this is self explanatory. The log level will default to INFO, however if you specify DEBUG it will output some more information such as what file’s it is currently checking. The logger_pattern parameter defaults to “%d{dd MMM yyyy HH:mm:ss} [%p] %m%n” and can be any pattern you want. For more information on the formatting visit the PatternLayout Documentation.

The default output format would look something like this:

If you want a little less verbose and more of just log lines you can update the logger_pattern to be just %m%n and end up with something similar to:

The format of the output is FILE:LINE_NUMBER:matching_string.

Anyways hope this helps you if you are trying to hunt down what file contains a text string in your S3 buckets. Let us know if you have any questions or if we can help!

Posted In: Amazon AWS, General, Java, Tips n' Tricks

Tags: , , , ,

In my last post I talked about setting up Symfony2 entities for translation and integrating it with Sonata Admin. One of the trickier parts of moving from a non-translatable entity to a translatable one is the migration of your data.

To understand some of the complexities with the migration you must understand the changes to the database that occur when taking an entity from being a regular entity to a translatable one. Any columns that are translatable will now live on a separate table and the old column is no longer used. Let’s use the following pre-translation entity DB schema as an example:

For this entity we’ll make visible_label translatable, following the instructions in my previous post. This will result in the following final schema:

The column “visible_label” has moved from the regular entity table to the entity’s translation table. If you had data in the visible_label previously it would be lost as that column no longer exists. Since we had tons of data in our case this wasn’t acceptable.

To make sure we didn’t lose data, we did the translatable migration in two stages. First, we kept the columns we were translating in the original entity and only removed the getters and setters. The reason we removed the getter and setters is we wanted to utilize the magic __call() method so it would return values from the translatable entity. All that was left was the original column declaration. At first it seemed like making the column variable public for the time being would be a quick and easy solution, then run a script that reads the public variable and migrates it to the translation. The problem with this approach is Twig will read out the public variable rather than calling through the __call() method to the translatable entity. Since we were testing at the same time as trying to build the migration, we needed the tests to access the translatable entity and not the old public variable. We ended up using Reflection Classes and keeping the column declared as a private. With reflection you can change properties to be accessible outside of the class even though they are declared private. For example:

By using the reflection we’re able to access the original “visible_label” column and migrate the data to the translation entity. We built similar routines for each of the entities that we had to migrate. After the migration and everyone confirmed that the live site was functioning properly, we removed the translated columns from the original entity and database.

By taking this two staged approach we were able to move to translatable entities while not losing any data in the migration. In our case we also marked (//START TRANS, //END TRANS) on each entity the start of translatable columns and end so that we could use sed to go through all of them and remove the old columns once the migration was finished.

Happy translating!

Posted In: General

Tags: , , ,

We’ve worked on a number of projects which require the UI to be translated using the standard Symfony2 translator and YAML files. Recently we came into a few projects which also required different entities to have certain fields translated. Most of our applications we build use Sonata Admin for the admin backend so making sure we could integrate with it nicely was important. Looking around on Google and Stackoverflow it was clear that there are several different ways to go about getting your entities translated from the Gedmo Translatable, KnpLabs Translatable, to A2LiX I18n. Many of the packages have different takes on the “proper” way translation should be setup for the project. There are other nuances between each package such as supporting a fallback locale.

In the end we settled on using the KnpLabs Translatable bundle as it ticked all the boxes we wanted in functionality including fallback locale and a nice integration with the form (more on that later in this post). Installing the bundle follows the standard add it to composer and enable it in the kernel. From there setting up an entity was pretty straight forward:

Taking a look at how it actually works. First in the main entity you use the Translatable trait. Then in the translation entity (which is your original Entity with the name Translation appended to it) you add what fields you want translated, as well as the Translation trait. From there you can do something like $entity->translate(‘en’)->getName(). In our case we had a fairly large application already built and having to go through everywhere to update it to $entity->translate(…)->getXXX() would of been a huge pain and time waste. Luckily there is a fairly easy way to get around this. Using PHP’s magic __call method you can intercept all the calls so that it will go through the translations automatically:

The reason that it checks if arguments were passed in is that the Symfony2 property accessor doesn’t support passing arguments. We wanted to use it though when no arguments were passed since twig would otherwise first try on a call of “$entity->name()” which would fail as no name exists. You could wrap a few checks to make sure the method exists, instead since majority of our gets from twig do not pass any parameters we opted to just use the property accessor if no arguments were passed. This fixed the problem of {{ }} in Twig causing an error that the callback doesn’t exist and causing a 500. We ended up making our own Translatable trait which included this special __call override.

The final piece of advice on getting the translations working is when you add new translations to make sure you call $entity->mergeNewTranslations(). If you don’t you’ll be confused on why for it seems that none of your translations are being saved. This is documented, I just had looked over it first.

Now our second goal was a nice integration with Sonata Admin and any other forms we needed to use the translatable fields on. Luckily the A2LiX Translation Form Bundle already existed and we went forward with using it. Using the bundle was very easy. It was a simple as installing it, configuring it(just indicating what locales you want to use), and then updating the different form fields/admin setups. One thing to note is in the documentation it uses $form->add(‘translations’,’a2lix_translations’) as a bare minimum use case. At first, like me, you may think that the “translations” field is one of your field names. In fact that is used to load all translatable fields from your entity. It drops it into a nice tabbed input box. If you want to customize the field types and other options you can pass an array of options to set each field up in terms of labels, field type, etc. All in all it was really a huge time saver to have this bundle and was very easy to use from both a developer and user standpoint.

For the most part this is how we went about enabling translations on different entities in our application. In my next post I’ll write up the steps we used to migrate all the data from our existing entities to the new translations.

Update: My post on how to migrate your data to translatable entities is now available.

Posted In: General

Tags: , , , ,

Unless you’ve been living under a rock for the last couple of years, it’s clear that smartphones are kind of a big deal. Today there are close to 2.6 billion subscriptions globally, and this number stands to grow rapidly as less developed markets turn into substantial electronic consumers.

With new applications and games hitting app stores daily, it’s no surprise that people are spending more and more time with their eyes glued to the glass screens of these compact, universal media agents. Phones have single handedly changed the way that people live, becoming pivotal for people to communicate, go online, and access and share information.

The ubiquitous nature of the smartphone has opened the floodgates of opportunity, creating new markets in the process and forcing pre-existing ones to modernize or be strangled at the hands of innovation. One market in particular that has been significantly impacted by the mobile revolution is the advertising and marketing industry.

The rise of the Internet era has led to a rapid decrease in the effectiveness of traditional forms of advertising media, and have forced the hand of the industry to take the plunge into to more digital forms. Marketing companies have to continuously find new and improved ways of reaching their targets in a world where TV and print simply will not do. One unique strategy that companies have used to connect with the masses and promote brand awareness is branded Apps. Smartphone users spend close to 90% of their time on devices using apps so they offer an incredible opportunity to connect with consumers.

Recently here at Setfive we have taken a look at building some brandable tools for mobile, and looked for the most original branded apps out there for some inspiration. Here are some of the most interesting apps that we found during our quest for inspiration.

Pop Secret Perfect Pop App

This app uses your phones speakers to listen to the pops coming from the bag of popcorn in your microwave, and tells you the precise moment when your bag of popcorn is a peak popped-ness.

Charmin SitOrSquat App

This app uses a map to display available public restrooms in your area and lets you know how clean they are (hence the name SitOrSquat). Additionally this app utilizes crowd sourcing, letting users to rate and write reviews about the public bathrooms that they use.

The Snow Report by North Face

So what has the North face been up to besides making incredibly epic TV commercials? That’s easy, they’ve been building a location based mobile app that helps users check the condition on powder before they head to the slopes. Check the status of runs around you or the 10 top slopes globally.

Tide Stain Brain

When stains happen, StainBrain gives you simple solutions on the spot. Blot or soak? Cold water or hot? Get the scoop on how to rid yourself of more than 85 different stains with on-the-go tips and easy, step-by-step washing instructions from the laundry pros at Tide.

Bosch Unit Converter: Professional converter for over 50 units

The Bosch Professional Unit Converter turns a smartphone into a universal unit converter. The ideal app for quick conversions on the building site or in the workshop. Completely free of charge and ads.

Posted In: General

Tags: ,

Our yearlong delve into the sea that is Angular Js has kept our engineering team busy for quite some time now. Throughout this fun time of exploration there have been ups and downs, muddled coding snares caused by niche problems that are almost impossible to conform to the constantly looming “Angular Way”, and complicated problems that could be solved so easily you wished you had a Staples button just so you could push it for the “That was easy”.

One invaluable resource that has continually helped guide us through our use of this robust MVC framework is the vast Angular Js community, that has helped answer most of our questions, from questions about basic tutorials, to more overarching questions about best practices when structuring applications.

On one of our more recent Angular projects, we ran into an issue implementing a fairly common need among web applications – generating an auto submit form. After poking around Google and StackOverflow for a bit, surprisingly, there was a lot less information on this subject than expected. So, we decided to write a blog post on the subject, giving back to the AngularJS community, which has given so much to us.

Interestingly, this problem can be solved quite easily once you have a more comprehensive understanding about the Angular directive’s Link function. Here’s a good StackOverflow answer if you’re interested in some light reading: AngularJS: What is the need of the directive’s link function when we already had directive’s controller with scope?. However, if you’re anything like us then you’re probably solely interested in learning just enough to solve the immediate problem at hand, so for you, the main “gotcha” that you need to know is that the template for any directive is wrapped as an angular element, and can be accessed inside of the Link function using the angular.element() function.

The basic idea is simple, create a custom directive for your auto submit form, make the template for this directive a form element with the information you want to send, and control its rendering using an ng-if directive. Using ng-if is important in this instance because this directive doesn’t just show/hide a portion of the DOM tree, but actually removes/recreates it, allowing you to take advantage of the directives Link function. Then, inside of the link function for the auto submit form directive, grab the form element using the angular.element() function and submit it using the javascript function submit().

If the above paragraph didn’t completely explain it to you, please don’t worry, here is a nice, concrete example to help you solve this problem quickly should you ever run into it again in the future.

Posted In: General