Category: General

Learning to learn to use Linux can be challenging, as everything from its reputation to general look can be intimidating. Having never used it before, instead using Windows both at home and for school I always saw Linux as something that was in the realm of true veteran software engineers and computer programmers.

During my first few weeks at Setfive I’ve had the chance to begin learning Linux and I have found it to be a useful and powerful tool. After learning some common and not so common commands I’ve really started to appreciate the flexibility and ease of use of the command line.

Here are 10 easy Linux “one liners” that allow us to accomplish some everyday tasks in a simple and efficient manner.

1. Sort a file

$ sort myfile.txt

This will sort the given file in numerical and alphabetical order:

Here a file, num, is being sorted alphanumerically, resulting in a sorted list.

2. Delete duplicate lines within a file

$ sort myfile.txt | uniq

This small addition to the sort command will not only sort the file but will remove any duplicates found within the file:

Here we can see the same num file being sorted, however this time the duplicates are being removed leaving us only with 1 of each entry.

3. Convert .mp3 and .wav files

$ ffmpeg -i input.mp3 output.wav

With this example we convert an .mp3 file to a .wav file, and it can be done the other way as well converting .wav to .mp3.

$ ffmpeg -i input.wav output.mp3

4. Recursively creating a directory structure

$ mkdir -p new/directory/structure/example

Using mkdir we are able to create subdirectories, however with the -p option we can tell it to create the subdirectories but also any parent directories that don’t already exist. Allowing an entire directory tree to be created with one line:

Here we can see an entire directory tree “/new/directory/example/setfive” being created.

5. Extract specific pages from a PDF

$ pdfjam pdf1.pdf 2-4 -o 2.pdf

Using the pdfjam command we are able to take pages 2, 3 and 4 from the pdf pdf1.pdf and save them in a separate file, here called 2.pdf.

6. Create a thumbnail image for a PDF

$ convert -thumbnail x80 file.pdf[0] thumb.png

Here we are using convert to create a thumbnail for a pdf, with the [0] indicating that the thumbnail will be made using the first page of the PDF file.

7. Make all file names lowercase

(Assuming you have a bash like shell)

$ for i in *; do mv "$i" "${i,,}"; done

This command will loop through the directory and use the built in case modification extension that comes with Bash to change all the file names to be lowercase.

8. Create an animated gif

$ convert -delay 20 -loop 0 *.jpg myimage.gif

This can be accomplished using the imagemagick package, which can be installed with “sudo apt-get install imagemagick”. This allows for full image manipulation including conversion and editing.

9. Create a file with some text

$ echo "a new string" >> file

It appends the string into the file, followed by a newline, if the newline is wanted then by adding the -n flag after echo will append the string without the following newline.

10. Split up a file

$ split -b 50MB verylargemovie.mp4

Split will break up a large file automatically to whatever sizes you need. Here we’re breaking up our big movie file into 50MB chunks.

Bonus: Rerun the last command, replacing part of the command

This allows for an easy method to rerun a command with the swapped string which can be useful with long commands where finding and altering a single string by hand would be tedious.

Using the convert example from above:

$ convert -delay 20 -loop 0 *.jpg myimage.gif
$ ^myimage^newimage
convert -delay 20 -loop 0 *.png newimage.gif

Will re-run the command and create an image named “newimage.gif”

Note: This post originally appeared on Codeburst.io

One of the biggest differences between Angular 1.5+ and 2 is the latter’s approach to dependency injection. Back on Angular 1.5 there were a couple of ways to configure dependency injection but they relied on naming things consistently. If you needed the “$http” service you had to specify that by specifying a configuration explicitly mentioning “$http”. And unfortunately, it was also possible to do this with implicit annotations which would cause minification to break your code.

Angular2 with TypeScript dramatically improves on this by introducing a type based DI system where you just correctly specify the type of an injectable you want and Angular will handle the rest. But since TypeScript compiles to JavaScript and in doing so wipes any type information how can this DI system work? It’s not magic so lets take a look!

TypeScript Decorators

The first piece of the puzzle is the @Injectable TypeScript decorator which marks a class as available for DI. TypeScript decorators are a language feature which enables developers to use annotations to modify the behavior of classes, methods, and properties at run time. Within Angular, the @Injectable annotation is “class annotation” which Angular uses to register information about the target class with the framework. The relevant framework code is in the packages/core/src/di namespace with the most interesting files being:

• injector.ts
• reflective_injector.ts
• metadata.ts

Reading through the code is a bit challenging but the overall idea is that the framework keeps track of classes that have been annotated with @Injectable and then has a “get” method to correctly construct instances of those classes. OK but what about that type information?

reflect-metadata

The second piece of the DI puzzle is the reflect-metadata package and TypeScript’s “emitDecoratorMetadata” option. When used together they will cause the TypeScript compiler to output metadata for classes that have been annotated with a class decorator. And most importantly this metadata is accessible at runtime by userland JavaScript.

Concretely, the Angular DI system uses this metadata to introspect the arguments that the constructor a class marked @Injectable requires. And then naturally using that information the DI framework can automatically construct a class on demand for you.

An example DI system

Finally what we’ve all been waiting for, some sample code! In order to compile, you’ll need to enable the experimentalDecorators and emitDecoratorMetadata compiler flags, install the reflect-metadata package, and also use a recent version of TypeScript.

import "reflect-metadata";

type Newable<T> = new (...args : any[]) => T;

class Injector {

    private static constructedInstances : {cx: Newable<any>, object: any}[] = [];

    public static inject<T>(originalConstructor : Newable<T>) : Newable<T> {
        const paramTypes = Reflect.getOwnMetadata("design:paramtypes", originalConstructor);

        if(paramTypes.length == 0){
            Injector.constructedInstances.push({cx: originalConstructor, object: null});
            return originalConstructor;
        }

        const newArgs = paramTypes.map((f : any) => Injector.get(f));
        const newConstructor = originalConstructor.bind(null, newArgs);

        Injector.constructedInstances.push({cx: newConstructor, object: null});

        return newConstructor;
    }

    public static get(cx : Newable<any>) : any {
        const ci = this.constructedInstances.find(f => f.cx == cx);

        if(!ci){
            throw "Invalid DI configuration.";
        }

        if(ci.object == null){
            ci.object = new ci.cx();
        }

        return ci.object;
    }

}

function Inject<T>(originalConstructor : Newable<T>) : Newable<T> {
    return Injector.inject(originalConstructor);
}

@Inject
class Engine {
    displacement : number;
    maker : string;

    constructor(){
        this.maker = "Tesla";
        this.displacement = 500;
    }
}

@Inject
class Car {
    constructor(private ex : Engine){}
}

console.log( Injector.get(Car) );

If you compile and run it you should get the following output:

Car { ex: [ Engine { maker: ‘Tesla’, displacement: 500 } ] }

So as expected we retrieved a Car without manually constructing an Engine and the Engine constructor was correctly invoked since the class properties were set.

Couple of things of note:

• I created the “Newable” type alias to represent a constructor
• The Inject decorator calls the Injector class in order to create some encapsulation
• On line 10, the Reflect.getOwnMetadata(“design:paramtypes”, originalConstructor); call retrieves constructor information for the class that the decorator has been applied to.
• Line 18 uses bind() to modify the class constructor to use the Injector to retrieve instances of the required classes

And that’s about it. After creating this example it’s clear that TypeScript’s decorators and the reflect-metadata are both powerful tools and I’m excited to explore what else they could enable.

Interested in adopting Angular or TypeScript at your organization? We’d love to chat.

(Note: This originally appeared on Codeburst)

As of 2017 TypeScript has emerged as one of the most popular languages which can be “compiled down” to JavaScript for both browser and nodejs development. Compared to alternatives like Facebook’s Flow or CoffeeScript one of TypeScript’s most unique features is its expressive type system. Although TypeScript’s type system is technically dynamic, if you fully embrace it you’ll be able to leverage several features of the TypeScript compiler.

So what are a couple of these features? We’ll be looking at code from Setfive’s CloudWatch Autowatch an AWS cloud monitoring tool written in TypeScript. Since TypeScript is evolving so quickly it’s worth noting that these examples were run on version 2.4.2. Anyway, enough talk lets code!

Non-nullable types

If you have any experience with Java you’ve probably encountered the dreaded NullPointerException when you tried to deference a variable holding a “null” value. It’s certainly a pain and has been derided as a “billion dollar mistake” by its creator. To tackle NullPointerException bugs TypeScript 2.0 introduced the concept of non-nullable types. It’s “opt-in” via the “strictNullChecks” compiler flag which you can set your tsconfig.json

Consider this simple sample:

function getPartyGuests() : string[] {
    if(Date.now() % 2 == 0){
        return [];
    }else {
        return null;
    }
}

const myGuests : string[] = getPartyGuests();
myGuests.push("Ashish");
console.log(myGuests);

By default, the TypeScript compiler will compile that code since “null” is assignable to string[] but you’ll get an error about half the time you run it. Now, if we set “strictNullChecks: true” and run the compiler we’ll get an error:

partyguests.ts(5,9): error TS2322: Type ‘null’ is not assignable to type ‘string[]’.

Since the compiler can infer that at least one code path in the function produces a null which is now incompatible with an array. An example of this in the Autowatch code are the checks to ensure that PutMetricAlarmInput instances aren’t created with null dimensions. At line 462 for example.

Exhaustive type matching

Most programming languages with a Hindley–Milner type system have some functionality to perform a “pattern match” over a type. In practice, that allows a developer to make decisions about what to do with a set of objects based on the concrete type vs. their abstract signatures. For example, in Scala:

// From https://docs.scala-lang.org/tour/pattern-matching.html

def showNotification(notification: Notification): String = {
  notification match {
    case Email(email, title, _) =>
      s"You got an email from $email with title: $title"
    case SMS(number, message) =>
      s"You got an SMS from $number! Message: $message"
    case VoiceRecording(name, link) =>
      s"you received a Voice Recording from $name! Click the link to hear it: $link"
  }
}

At face value it appears as if you can replicate this ES2015 JavaScript with something like:

function showNotification(notification){
    if(notification instanceof Email){
        return "You got an email from $email with title: $title";
    }else if(notification instanceof SMS){
        return "You got an SMS from $number! Message: $message";
    }else if(notification instanceof VoiceRecording){
        return "you received a Voice Recording from $name! Click the link to hear it: $link";        
    }
}

That misses an important piece from the Scala example, the compiler guarantees an exhaustive match. If you added a new subclass of “Notification” like BulkSMS the Scala compiler would error because showNotification is missing that case. The JavaScript example wouldn’t be able to provide this and you’d end up getting a runtime error.

However, with TypeScript’s “never” type it’s possible to have the compiler guarantee an exhaustive match for us. We could do something like the following:

type MyNotification = Email | SMS | VoiceRecording;
class Email {}
class SMS {}
class VoiceRecording {}
class BulkMessage {}

function assertNever(error : never) : string {
  throw Error("Invalid: " + error);
}

function showNotification(notification : MyNotification) : string {
  if(notification instanceof Email){
    return "You got an email from $email with title: $title";
  }else if(notification instanceof SMS){
    return "You got an SMS from $number! Message: $message";
  }else if(notification instanceof VoiceRecording){
    return "you received a Voice Recording from $name! Click the link to hear it: $link"; 
  }else{
    return assertNever(notification);
  }
}

The important part is the call to “assertNever” which the compiler will error on if it detects is a reachable code path. We can confirm this if we add “BulkMessage” to the “MyNotification” type but not the if:

type MyNotification = Email | SMS | VoiceRecording | BulkMessage;
class Email {}
class SMS {}
class VoiceRecording {}
class BulkMessage {}

function assertNever(error : never) : string {
  throw Error("Invalid: " + error);
}

function showNotification(notification : MyNotification) : string {
  if(notification instanceof Email){
    return "You got an email from $email with title: $title";
  }else if(notification instanceof SMS){
    return "You got an SMS from $number! Message: $message";
  }else if(notification instanceof VoiceRecording){
    return "you received a Voice Recording from $name! Click the link to hear it: $link"; 
  }else{
    return assertNever(notification);
  }
}

If you run the TypeScript compiler against that you’ll get an error highlighting that your if isn’t exhaustive since it’s hitting the “never”:

match2.ts(19,24): error TS2345: Argument of type ‘BulkMessage’ is not assignable to parameter of type ‘never’.

It’s certainly not as elegant as the Scala example but it does the job. You can see a real example in Autowatch starting at line 168 where we used it to guarantee exhaustive matching on the available AWS services.

Read-only class properties

Marking a class property “readonly” signals to the compiler that code shouldn’t be able to modify the value after initialization. Although “readonly” may sound similar to marking a property as “private” it actually enhances the type system in important ways. First, “readonly” properties make it possible to more faithfully represent immutable data. For example, a HTTP Request has a “url” which will never change after the request has started. And by marking properties as “readonly” as opposed to private you’re still able to return literal objects with matching properties.

Let’s look at an example:

class HttpRequest {
  readonly url : string;  
}

const req = new HttpRequest();
req.url = "http://www.setfive.com";

If you run that through the TypeScript compiler you’ll get an error advising that the property is readonly:

readonly.ts(6,5): error TS2540: Cannot assign to ‘url’ because it is a constant or a read-only property.

Now, if you try and mark the url property as private and create a literal HttpRequest you’ll notice you’ll get an error:

class HttpRequest {
  private url : string;  
}

function fromGlobals() : HttpRequest {
  return {url: "http://www.setfive.com"};
}

But if you switch it back to “readonly” it’ll work as expected.

You can see real world usage of this in Autowatch, where we marked properties in our Config class as readonly since they should never change once the object has been constructed.

That’s a wrap

Well that’s three pretty cool features of the TypeScript type system that should help you be more productive and write better code. If you found these interesting, there’s several other interesting type related features that landed in 2.3+ versions of TypeScript that are worth checking out.

(Note: This is a guest post from our friends at Panoply)

Cloud-based data services are all the rage these days for many good reasons, and AWS (Amazon Web Services) is the current king of cloud-based data service providers, as this analysis carried out by StackOverflow indicates.

Two popular AWS cloud computing services for data analytics and BI are Amazon Redshift and Amazon Athena, both of which are useful for delivering actionable insights that drive better decision making from your data. However, with a dizzying amount of information available on both services, it’s a challenge to recognize what to look out for when choosing a cloud-based data service to meet your needs.

In this post, you’ll get a broad overview of cloud-based data warehousing, and you’ll come to understand the main differences between Amazon Redshift and Amazon Athena (also see this post by Panoply on the subject).

When you’re finished reading, you’ll know which service you should choose between Athena and Redshift. The comparison can also teach you what to look for in more general terms when considering any cloud-based data solution currently available.

A Data Warehouse in the Cloud

Traditional on-premise data warehouses are used for analyzing an organization’s historical data in one unified repository, pulling data from many different source systems, such as operational databases. Physical data warehouses are complex and expensive to build and maintain, though.

Cloud-based data warehouse services offer a much cheaper and easier way to use a data warehouse without needing any physical resources on site. Cloud-based providers host the necessary physical resources “in the cloud” while you simply pay for using the service.

Some examples of data warehouses in the cloud are:

• Amazon Redshift—in Redshift, you provision resources and manage those resources similar to how you would in a traditional data warehouse, without the need to host physical computing resources on-site—AWS hosts them in the cloud instead as “clusters”. You simply connect your operational data sources to the cloud and move the data into Redshift for use with analytics and BI tools. Redshift uses a customized version of PostgreSQL.
• Amazon Athena—Athena provides an interactive query service that makes it possible to directly analyze data stored in Amazon S3, which is the cloud storage service provided by AWS. You query data in Athena with standard ANSI SQL. In contrast to Redshift, Athena takes a serverless approach to data warehousing because you don’t need to provision resources or manage the infrastructure.
• Google BigQuery—BigQuery is a serverless cloud-based data analytics platform that enables querying of very large read-only datasets, and it works in conjunction with Google Storage.
• Azure SQL Data Warehouse—this Microsoft offering is a scalable and fully managed cloud-based data warehouse service.

You could write a book comparing all four of these services, so we’re going to hone in on both Amazon Redshift and Amazon Athena below.

Amazon Athena vs. Amazon Redshift - Feature Comparison

• Initialization Time: Amazon Athena is the clear winner here because you can immediately begin querying data stored on Amazon S3. Redshift, on the other hand, requires you to prepare a cluster of computing resources and load data into the tables you create.
• User-defined Functions: Amazon Redshift supports user-defined functions (UDFs), which are procedures that accept parameters, perform an action, and return the result of that action as a value. Amazon Athena has no support for UDFs.
• Data Types: Amazon Athena supports more complex data types, such as arrays, maps, and structs, while Redshift has no support for such complex data types.
• Performance: For basic table scans and small aggregations, Amazon Athena outperforms Redshift. However, for complex joins and larger aggregations, Redshift is a better option.
• Cost: Athena’s cost is based on the amount of data scanned in each query, which means it’s important to compress and partition data. Since Amazon Athena queries data on S3, the total cost of S3 data storage combined with Athena query costs gives the full price. Redshift’s cost depends on the type of cloud instances used to build your cluster, and whether you want to pay as you use (on demand) or commit to a certain term of usage (reserved instances).

Athena’s cost is $5 per terabyte of data scanned, while Redshift’s hourly costs range from $0.250 to $4.800 per hour for a DC instance, and $0.850 to $6.800 per hour for a DS instance.

AWS Redshift Spectrum, Athena, S3

Redshift Spectrum is a powerful feature that enables data querying in Redshift directly from S3. With Spectrum you can create a read-only external table, with its data located in a specified S3 path, and immediately begin querying that data without inserting it into Redshift. You can also join the external tables with tables that already reside on Redshift.

Querying data in S3: sounds familiar, right? That’s because Amazon Athena performs a similar function—it’s an S3 querying service. It’s important to note, however, that Spectrum is not an integration between Redshift and Athena—Redshift queries the relevant data on its own from S3 without the help of Athena.

If you are already an Amazon Redshift user, it makes sense to opt for Spectrum over Athena because of the convenience. However, if you aren’t currently using Redshift, it’s best to choose Athena over Spectrum because your investment in computing resources might go underutilized in Redshift. For your current analytics needs, Athena is likely to do the job—you can always invest in a Redshift+Spectrum combination later on when it’s needed to handle lots of data.

Athena vs. Redshift - Which Should You Choose?

There is no widespread consensus on whether Amazon Athena is better than Redshift or vice versa—both services suit different uses.

• Amazon Athena is much quicker and easier to set up than Redshift, and this querying service outperforms Redshift on all basic table scans and small aggregations. The accessibility of Athena makes it better suited to running quick ad hoc queries.
• For complex joins, larger aggregations, and very large datasets, Redshift is better due to its computational capacity and highly scalable infrastructure.
• The conclusion, therefore, is that Redshift is likely the cloud-based data warehouse platform of choice if you have lots of data and many complex queries to run. Amazon Athena is the recommended cloud-based service for analyzing your data in all other cases, and it’s suitable for small- to medium-sized organizations.

Closing Thoughts

Cloud-based data warehouses are quickly replacing traditional on-premise data warehouses because of their convenience, lower cost, and scalability.

Amazon Athena and Amazon Redshift take differing approaches to cloud-based data analytics services—Redshift requires resource provisioning and infrastructural management while Athena abstracts operational management away from users and allows direct querying of data stored on Amazon S3.

Amazon Spectrum provides separation of storage and compute in Redshift by allowing you to directly query data in S3, similar to Athena. Spectrum is useful if you already use Redshift, but you shouldn’t base your decision on Athena versus Redshift on the Spectrum feature.

The relevant comparison between Amazon Athena and Redshift relates to how they perform, what they cost, which tools they support, their usability, their accessibility, supported data types, and user-defined functions. You should base your end decision between Redshift and Athena on these factors, prioritizing the most important aspects of each service for your particular business. Maybe you prefer Athena’s effortless accessibility? Or maybe you’d rather the control and scalability you get in Redshift?

When weighing up any potential cloud-based data warehouse, always consider the above factors, instead of just choosing the most affordable solution.

It has become a bi-weekly ritual. The professor spent too much time on the course material again and is left mumbling through a complex project description during the 11th hour of class. All the while, you’re off somewhere else. As you sling your backpack over your shoulder, you catch the only words you’ll need to hear: “You can download the syllabus along with the source code from the CS department’s website,” they say. Great! You hustle back to study location of choice, open your laptop, and extract the project files. After the obligatory knuckle crack, you look down at the method stubs spelled out for you. “All I have to do is fill-in these functions?” you think to yourself. And as you’re getting familiar with the project structure, a couple flicks of the scroll wheel reveal hundreds, sometimes thousands of lines of unexplained boilerplate code.

You eventually finish up the assignment and push it to the CS department’s server for grading. Without fail, someone raises their hand during the next class asking the instructor if they could explain what some of that boilerplate code was for, at which point the student is usually told to refer to the language documentation to figure it out for themselves. And for the most part, this makes perfect sense. After all, you’re there to learn about some of the more complex topics in computer science, not to write setter and getter methods all day. That’s what your data structures class was for.

But I would like to share with you the first few months of my experience as a Jr. Software Engineer and compare it to my time as an undergraduate student. You might be not-so-surprised to hear I have spent more time writing code similar to the boilerplate stuff mentioned above than I have perfecting the space and time complexity of my pioneering solution to The Traveling Salesman problem.

As an undergraduate student, I was an ace at avoiding merge conflicts in repositories where I was the only contributor. I could even run a build script with the best of ‘em. Nobody ever really told me how to use version control systems to manage a collaborative project with tens of thousands of lines of code strewn across a mess of files and directories. And if, for some reason, those same build scripts broke or a merge conflict popped up on a group project? Well, I was pretty much at the mercy of Stack Overflow. At Setfive, when I was tasked with setting up a relational database schema for my first real project, I wasn’t really sure where to begin. There was no syllabus to refer to and no professor to schedule office hours with. While I was aware of relational database software such as MySQL and NodeJS, I had never really written a query, so I certainly didn’t know the difference between an inner and outer join. And while coordinating all those AJAX calls and setting up the Symfony bundle configs was a little confusing at first, I think I’m starting to learn how to apply my undergraduate education to these real-world projects.

So far, I have found that industry-level programming helps hone a much more practical skill set than academic programming. Don’t get me wrong, I learned a ton in college, and I know the concepts taught are not only important to a fundamental understanding of the field of computer science, but also have profound and meaningful applications elsewhere, such as in operating systems, machine learning, and so on. But when I look back on the things I have learned in such a short period of time over these past few months, it gets me excited for the road ahead. I owe an enormous thanks to Setfive for bringing me on as an entry-level software developer and advising me with patience.