General Archives - Page 7 of 25 - {5} Setfive

Recently I was working on a project where part of it was doing data exports. Exports on the surface are quick and easy – query the database, put it into the export format, send it over to the user. However, as a data set grows, exports become more complicated. Now processing it in real time no longer works as it takes too long or too much memory to export. This is why I’ll almost always use a background process (notified via Gearman) to process the data and notify the user when the export is ready for download. On separate background threads you can have different memory limits and not worry about a request timeout. I suggest trying to not use Doctrine’s objects for the export, but get the query back in array format (via getArrayResult). Doctrine objects are great to work with, but expensive in terms of time to populate and memory usage; if you don’t need the object graph results in array format are much quicker and smaller memory wise.

On this specific export I was exporting an entity which had a foreign key to another table that needed to be in the export. I didn’t want to create a join over the entire data set as it was unnecessary. For example, a project which has a created by user as a relation. If I simply did the following:

I’d end up with an array which had all the project columns except any that are defined as a foreign key. This means in my export I couldn’t output the “Created by user id” as it wasn’t included in the array. It turns out that Doctrine already has this exact situation accounted for. To include the FK columns you need to set a hint on the query to include meta columns to true. The updated query code would look similar to:

Now you can include the foreign key columns without doing an joins on a query that returns an array result set.

As a team last week we decided to work through a “Coding Kata” as a learning experiment. For those unfamiliar with the concept, a Kata is a Japaneese word that refers to a choreographed series of movements. In the context of programming, a Kata refers to repeatedly solving a problem with incrementally improving technique. The challenge we decided to look at was a somewhat contrived Bank OCR problem described on codingdojo.org.

We only looked at “User Story 1” which describes a process for translating lines of ASCII text into the decimal numbers which they represent. The task isn’t particularly hard but what we were interested in was how different people approached the problem, what considerations they had, and where they got hung up. As a group, the consensus approach was to decompose the problem into pieces, decide on data structures, and then sketch out some pseudo code.

Breaking down the problem is fairly straightforward. We’re basically looking to read in the file, separate the lines into the “bank numbers”, recognize individual numbers, and then combine the numbers we’ve recognized. An additional benefit of approaching the Kata like this was that individual team members could craft solutions in different programming styles – imperative, functional, etc.
Once we had the “big” pieces of the solution, the next thing we flushed out was which data structures to use. After some discussion it was clear that reading the file and recognizing the numbers were related and the data structures used would need to be related. Interestingly, there was a decent amount of debate around what to use to represent the read lines and then how to match them. The team was split between using a 3×3 array versus using a 9 character string.
The final step we took was jotting down some pseudo code on a whiteboard to actually perform all the steps. I’ve always found it interesting to watch people code on a whiteboard and this time wasn’t any different. Our team seemed to struggle with visualizing what the data looked like and also how much detail to convey in the pseudo code.

After we wrapped up on the whiteboard we ended up implementing 3 solutions. A functional solution using a 3×3 array for the numbers, an imperative solution also using a 3×3 array, and an imperative solution using a 9 character string to represent the numbers. The solutions are similar but the style of the original author still definitely comes through.

Anyway, how would you have approached a problem like this? We’d love to hear in the comments!

Congrats! You’ve survived the first full work week of 2015. Hopefully your resolutions are still intact and if they’re not there’s always next year. Shockingly informational links ahead so hold on to your hats.

Are there any reasons to use PostgreSQL DB over MySQL DB?

PostgreSQL vs. MySQL is as close to a religious war as you’ll get in the database world. If you’re wondering about the pros and cons of PostgreSQL then check out this Quora thread. Read More >

Efficient Marketing Research for Startups Using Facebook Advertising

Market research is hard, especially when you’re on a budget. Luckily for you Facebook is a great underutilized tool to perform research and validate some hypotheses. Learn More >

Bitcoin: Looking forward to 2015

The price of Bitcoin might be falling off a cliff but there’s still big things ahead. adatta02 expands on a pmarca Tweetstorm to see what’s in store. Keep reading >

Welcome to 2015! Hopefully everyone had a great New Years Eve and is gearing up for 2015. There’s been a slew of PHP related news over the last week or so and we’ve got it wrapped up just for you.

How we made editing Wikipedia twice as fast

The Wikipedia foundation switched to running HHVM and the results are nothing short of impressive. They’ve provided a full breakdown of the process, considerations, and results. Read More >

PHP’s new hashtable implementation

PHP7 is getting a new implementation for it’s venerable hashtable data structure. @nikita_ppv who wrote the new implementation explains the details of the PHP5 version and how the new one is better across the board. Read More >

PHP Install Statistics

Is the version of PHP you’re running secure? Statistically probably not. @ircmaxell runs the numbers to analyze what portion of PHP installs are probably running insecure versions.
Read More >

Last week we officially did a very quiet launch of HotelSaver.io. The concept is fairly simple: You submit your existing hotel reservation, we constantly monitor for price drops, if we find one we notify you immediately and you save money. I had the idea for this site a few months ago when I had made reservations in New Orleans for a bachelor party, then noticed the next day the prices dropped and I managed to save over 40% of the reservation by getting it price matched and discounted. At this point I thought “wow, that was incredibly easy, took little time and saved a ton of money”. Shortly thereafter and shooting it around with everyone here, it was decide we’re going to launch a MVP product and see what the reception is.

With this concept it is really easy to quickly blow it up into a massive product with complex algorithms, payment types, etc. however trying to follow our own advice to our clients we launched with the minimal features to make it useful to the end user: simple reservation monitoring and payment processing to get us paid. We knew the first version of the product would be far from finished in terms of design and feature complete, but we wanted to see if others thought the idea had legs. Here are a few things we did to cut down on the time to launch even more:

The initial design is based on a free template we found which allowed us to spend near no time on design. It works on mobile devices and doesn’t look terrible. None of us are great designers, so we figured this was well worth it for the first release.
Not over thinking user management. Over time we plan to add accounts to the site so you can see your existing reservations from a dashboard, however for this first version we opted to go with a simple “email” to link together accounts. Users submit their existing hotel reservation with their email which we use later if you need to retrieve it. From there you can retrieve your “active” (reservations that have not yet past) reservations in an email we email to you.
Payment processing. This one was a no-brainer for us. We wanted to be PCI compliant and also have a good user experience. Stripe we had worked with in the past and knew it was incredibly easy to use. We went with the checkout feature so we never would have any of their credit card information and it never hit our servers, making us PCI compliant.

We also wanted to get feedback from a small group of users. We posted it on HackerNews and immediately started getting great feedback. We knew posting this on the day we were traveling for the Holidays wasn’t optimal as we couldn’t respond to feedback immediately, we wanted to get this launched. We managed to make it to the front page of HackerNews for a while and instantly had 2,500 unique visitors that day, up from zero the day before! The feedback was great the main points were:

Everyone loved the idea and thought if executed properly it’d be great!
People didn’t like that we wanted to charge $19.99 regardless of if we could find a lower cost reservation. It was too risky.
Some of the design could use some love.
Pricing would be more interesting/better if it was a percentage saved or a money back guarantee.

Today we revamped our pricing strategy after the feedback. We knew the upfront cost was most likely a turn away for many users but didn’t know what percentage would hate it. After reading the feedback on the post and numerous emails, we’ve switched to a 20% of the amount saved. This makes it 100% risk free to the user. We won’t make money unless you save money. If we save you $100 dollars, you get $80 of it. We’ll be next week working on promoting the revised pricing strategy to see what additional feedback we can get as well as addressing the other parts of the feedback.

We’ll be trying to keep everyone updated on our adventures of launching our own product in house. We’re excited to try some techniques we’ve seen over the years and testing them out ourselves as well as trying some new ideas. If you have any feedback let us know!

Category: General

Including Foreign Keys in Doctrine2 Array Results

Coding: How do you approach an engineering problem?

Friday Links: RDMS, Bitcoin, and Facebook