Archive for the ‘hacks’ Category

A LAMP guy’s n00b quick start to Amazon Web Services

Sunday, January 16th, 2011

If you haven’t dabbled with AWS before and are impatient but want to kick off a free instance to play around with it, here’s a perspective and lessons learned from a LAMP + Django guy who had no prior experience. Caution, this is just enough for you to wrap your head around the major concepts and get your first hello world instance off the ground. Beginning is always the hardest, I hope this will help lift you off the ground as I document the things I found out the time-consuming way.

EC2 instance is like a VM – but without a local hard disk. Instead, the VM uses a highly reliable external drive (EBS). Think of EBS as a really big USB drive, and just like a USB stick, it’s a raw device that needs to be formatted first. The obvious upside of this setup is you can use a low-powered VM (to save money), and when your site gets popular and you need to scale up – just quickly turn off that low-powered VM (the EC2 instance), fire up a beefier EC2 instance and attach to it the same EBS. And just like that, you’ve got a beefier machine running to handle the load.

Note: The EC2 instance just described above is what is known as an “EBS-backed” instance (the root device is an EBS volume, which is where the OS boots from), as opposed to the other option “instance store” (a.k.a. S3-backed). The latter option is for advanced use – not the focus of this primer ;) So when you’re firing up your hello-world instance, choose EBS-backed, not instance store. Similarly, ignore the AMI creation process, that’s advanced stuff you can visit later. Use one of the already existing AMI’s – like the ones provided by Ubuntu. My example uses Lucid Lynx.

Statistically, EBS is also actually more reliable than a local physical hard drive. However, you’d probably want to take snapshots of the volume (how confident are you that your data is safe on a USB stick?) Snapshots are stored in S3, which is even safer because it’s stored around the world (in case one continent gets wiped off the planet). Subsequent snapshots are also incremental deltas from previous snapshots, thus you save space. Although you’re fine if you just want to keep just daily 1 snapshot (unless you have a reason to want many daily snapshots).

Ok, at this point in the post, you should go ahead and follow AWS’s guide to firing up your 1st Ubuntu instance (remember, you can ignore the AMI creation stuff for now, and choose the 8GB EBS-backed instance – especially if you want to use the 1-year free micro instance). I’ll wait.

I’m still waiting.

Ok, at this point you should have it fired up and running, and be able to ssh into it. I ssh into mine with: ssh -i username.pem ubuntu@ec2-111-222-333-444.us-west-1.compute.amazonaws.com

Now on to some basic first time house-keeping items, best practice stuff.

The Delete On Termination flag on your EBS volume

On your new EBS-backed instance, that EBS volume has a flag called the “delete on termination” flag, which default to true. You’d probably want to set it to false. If it’s set to true, that means EBS root volume is deleted when you terminate the instance. Instances can be started, stopped, and terminated. “Stop” means you can start it back later. “Terminate” means delete, and you can’t un-delete. The following example uses Amazon’s EC2 API tools. Be sure to download your X509 certificate and private key (these are both files you download from your Amazon account). I’m using OS X 10.5, so change accordingly. Be sure to set your Java home too.

OSX-LEOPARD:bin jliew$ export EC2_HOME=/Users/jliew/Desktop/ec2-api-tools-1.3-62308/
OSX-LEOPARD:bin jliew$ export EC2_PRIVATE_KEY=/Users/jliew/pk-q3ef98zHSDg872hGTQpoX.pem
OSX-LEOPARD:bin jliew$ export EC2_CERT=/Users/jliew/cert-SDkhIzWU3HDqS83sXdsefh.pem
OSX-LEOPARD:bin jliew$ export JAVA_HOME=/Library/Java/Home

Once that’s set up, you’re ready to go. Don’t forget to pass it the region (mine is us-west-1), your instance’s id, and volume id (and volume mount point):

OSX-LEOPARD:bin jliew$ ./ec2-modify-instance-attribute -b /dev/sda1=vol-wuh5da87:false i-ppo983x1 --region=us-west-1
Unexpected error:
java.lang.ClassCastException: com.amazon.aes.webservices.client.InstanceBlockDeviceMappingDescription
	at com.amazon.aes.webservices.client.cmd.Outputter.outputInstanceAttribute(Outputter.java:664)
	at com.amazon.aes.webservices.client.cmd.ModifyInstanceAttribute.invokeOnline(ModifyInstanceAttribute.java:149)
	at com.amazon.aes.webservices.client.cmd.BaseCmd.invoke(BaseCmd.java:795)
	at com.amazon.aes.webservices.client.cmd.ModifyInstanceAttribute.main(ModifyInstanceAttribute.java:269)
OSX-LEOPARD:bin jliew$ ./ec2-modify-instance-attribute --region=us-west-1 -b /dev/sda1=vol-wuh5da87:false i-ppo983x1

Noticed how it actually crashed with an error? Turns out, other people faced this problem too, but the command actually succeeded. To verify it succeeded, run:

OSX-LEOPARD:bin jliew$ ./ec2-describe-instance-attribute --region=us-west-1 --block-device-mapping -v i-ppo983x1

and you should notice this that somewhere within the output is a line that looks like this: <deleteOnTermination>false</deleteOnTermination> (more…)

San Diego’s 1st SuperHappyDevHouse (SDSHDH1)

Sunday, July 4th, 2010

Update 7/11/2010: Coverage of the event from the Del Mar Times by Steve Perez!

San Diego’s 1st ever SuperHappyDevHouse was a blast and success! Special thanks to Erica and Richard for hosting the hackathon :) We had about 17-18 software + hardware folks (and one reporter! we have no idea how that happened). The venue was perfect, people brought snacks, drinks, lawn chairs, and we ordered pizza. Most of the attendees are not surprisingly, from the San Diego Hacker News meetup.

While the attendance was really great good for the SDSHDH1, I suspect that it would have been as much as 30% higher if the semester was in session as many of those who voiced interest are college students from the nearby UCSD. Below are some pictures and videos from the event. I’m already looking forward to the next one! :) Thanks to all who stopped by—”network effects” is key to having a fun SHDH ;)

SDSHDH1

SDSHDH1

SDSHDH1

Personal Robotics

Monday, January 25th, 2010

Wow. First, computers were these gigantic machines that took up all the space in a single room (or more!), made for the government and big corporations. Then prices dropped, technology improved (Moore’s law), and the computer was personalized, and today everybody practically carries one in their pocket.

When I had to first decide what to major in when in college, I did want to major in something to do with robotics. Robotics however, wasn’t exactly a popular or major discipline that was easily available anywhere. Fast forward to today .. we have personalized robots.

Enter the world of Arduino.

Here’s a very basic robot that exhibits basic facial expression, speaks from text, complete with LEDs. This is hilarious. And SOOOOO COOL.

Read more about the Ganzbot here.

Next, we have Arduway, a Segway-like self-balancing robot. It’s like creating a fragile and little life-form from your bare hands.

Read more about the Arduway here.

And alright .. enough of toys. How about a real world application? Here’s a robot that uses GPS and other sensors to steer a 30,000 lbs beast of a machine for a farmer – to harvest soybeans!

Can’t wait for the robotic revolution!

If you are interested in getting started with learning Arduino, here’s a quick link to the books recommended by the official Arduino site.

Getting Started with Arduino (Make: Projects)

Making Things Talk: Practical Methods for Connecting Physical Objects

Holding a program in your head

Sunday, August 26th, 2007

Here’s some good advice from Paul Graham writes about good practices that any good programmer can relate to.

  1. Avoid distractions. Distractions are bad for many types of work, but especially bad for programming, because programmers tend to operate at the limit of the detail they can handle.

    The danger of a distraction depends not on how long it is, but on how much it scrambles your brain. A programmer can leave the office and go and get a sandwich without losing the code in his head. But the wrong kind of interruption can wipe your brain in 30 seconds.

    Oddly enough, scheduled distractions may be worse than unscheduled ones. If you know you have a meeting in an hour, you don’t even start working on something hard.
  2. Work in long stretches. Since there’s a fixed cost each time you start working on a program, it’s more efficient to work in a few long sessions than many short ones. There will of course come a point where you get stupid because you’re tired. This varies from person to person. I’ve heard of people hacking for 36 hours straight, but the most I’ve ever been able to manage is about 18, and I work best in chunks of no more than 12.

    The optimum is not the limit you can physically endure. There’s an advantage as well as a cost of breaking up a project. Sometimes when you return to a problem after a rest, you find your unconscious mind has left an answer waiting for you.
  3. Use succinct languages. More powerful programming languages make programs shorter. And programmers seem to think of programs at least partially in the language they’re using to write them. The more succinct the language, the shorter the program, and the easier it is to load and keep in your head.

    You can magnify the effect of a powerful language by using a style called bottom-up programming, where you write programs in multiple layers, the lower ones acting as programming languages for those above. If you do this right, you only have to keep the topmost layer in your head.
  4. Keep rewriting your program. Rewriting a program often yields a cleaner design. But it would have advantages even if it didn’t: you have to understand a program completely to rewrite it, so there is no better way to get one loaded into your head.
  5. Write rereadable code. All programmers know it’s good to write readable code. But you yourself are the most important reader. Especially in the beginning; a prototype is a conversation with yourself. And when writing for yourself you have different priorities. If you’re writing for other people, you may not want to make code too dense. Some parts of a program may be easiest to to read if you spread things out, like an introductory textbook. Whereas if you’re writing code to make it easy to reload into your head, it may be best to go for brevity.
  6. Work in small groups. When you manipulate a program in your head, your vision tends to stop at the edge of the code you own. Other parts you don’t understand as well, and more importantly, can’t take liberties with. So the smaller the number of programmers, the more completely a project can mutate. If there’s just one programmer, as there often is at first, you can do all-encompassing redesigns.
  7. Don’t have multiple people editing the same piece of code. You never understand other people’s code as well as your own. No matter how thoroughly you’ve read it, you’ve only read it, not written it. So if a piece of code is written by multiple authors, none of them understand it as well as a single author would.

    And of course you can’t safely redesign something other people are working on. It’s not just that you’d have to ask permission. You don’t even let yourself think of such things. Redesigning code with several authors is like changing laws; redesigning code you alone control is like seeing the other interpretation of an ambiguous image.

    If you want to put several people to work on a project, divide it into components and give each to one person.
  8. Start small. A program gets easier to hold in your head as you become familiar with it. You can start to treat parts as black boxes once you feel confident you’ve fully explored them. But when you first start working on a project, you’re forced to see everything. If you start with too big a problem, you may never quite be able to encompass it. So if you need to write a big, complex program, the best way to begin may not be to write a spec for it, but to write a prototype that solves a subset of the problem. Whatever the advantages of planning, they’re often outweighed by the advantages of being able to keep a program in your head.

Disclosure: Paul Graham runs the Y Combinator Startup School held annually at Stanford, so he does have a vested interest in helping startups succeed.

Now that I’ve got that upfront disclosure out of the way, I want to include Paul’s two observations which I agree with, simply because I feel what he says is true.

On why single solo great programmers are productive and get great products out the door,

It’s striking how often programmers manage to hit all eight points by accident. Someone has an idea for a new project, but because it’s not officially sanctioned, he has to do it in off hours—which turn out to be more productive because there are no distractions. Driven by his enthusiasm for the new project he works on it for many hours at a stretch. Because it’s initially just an experiment, instead of a “production” language he uses a mere “scripting” language—which is in fact far more powerful.

He completely rewrites the program several times; that wouldn’t be justifiable for an official project, but this is a labor of love and he wants it to be perfect. And since no one is going to see it except him, he omits any comments except the note-to-self variety. He works in a small group perforce, because he either hasn’t told anyone else about the idea yet, or it seems so unpromising that no one else is allowed to work on it. Even if there is a group, they couldn’t have multiple people editing the same code, because it changes too fast for that to be possible. And the project starts small because the idea is small at first; he just has some cool hack he wants to try out.

On why large software companies sometimes don’t realize that they work against these good practices for programmers,

Even more striking are the number of officially sanctioned projects that manage to do all eight things wrong. In fact, if you look at the way software gets written in most organizations, it’s almost as if they were deliberately trying to do things wrong. In a sense, they are. One of the defining qualities of organizations since there have been such a thing is to treat individuals as interchangeable parts. This works well for more parallelizable tasks, like fighting wars. For most of history a well-drilled army of professional soldiers could be counted on to beat an army of individual warriors, no matter how valorous. But having ideas is not very parallelizable. And that’s what programs are: ideas.

It’s not merely true that organizations dislike the idea of depending on individual genius, it’s a tautology. It’s part of the definition of an organization not to. Of our current concept of an organization, at least.

He then concludes,

Perhaps the optimal solution is for big companies not even to try to develop ideas in house, but simply to buy them. But regardless of what the solution turns out to be, the first step is to realize there’s a problem. There is a contradiction in the very phrase “software company.” The two words are pulling in opposite directions. Any good programmer in a large organization is going to be at odds with it, because organizations are designed to prevent what programmers strive for.

There is some truth to that. Established software companies have their own marketing departments. However, they sometimes tap outside marketing firms for help in their programs. Some things are just better when produced outside of the company. Are ideas and rapid-prototype software development one of them?

Inbox Zero

Saturday, August 25th, 2007

Merlin Mann of 43folders fame presented a talk at Google titled Inbox Zero, a productivity hack/treatment for people who live out of their inboxes. The talk is about an hour long, so I’ve written up here some of the points I feel are important and works for me.

Disclaimer: This is _obvious_ stuff, but sometimes we forget, and a reminder is always nice.

Respect yourself, spend time and attention wisely

For knowledge workers like myself, we don’t create value by the number of bricks we can carry with our arms. We process knowledge/information, and that’s how we create value for our companies. The two things that knowledge workers must understand and appreciate is time and attention. Both of which we only have a finite amount of, and both of which are our constraints to our productivity. The goal is to separate the wheat from the chaff, saying NO to the low value work so that we can say YES to the high value work. Procrastination and frittering time away in email, surfing the web and flipping TV channels aimlessly would be “low value” work (more like almost no value work).

Email is a communication medium, just like the telephone. Don’t focus on email itself, focus on the information in the email and process it. Hitting the send/receive button all day is “busy work”, made to think that you are doing work, but you’re not actually doing any real work.

Process information in email
Don’t read the email and do nothing! Do something about it. Process it when you check it. Mine the gold from your inbox, and throw away the empty husk.

Processing actions:

  • Delete (consume) or archive it (save for historical records)
  • Delegate or forward it to someone else (stuff that don’t apply to you, or better handled by someone else)
  • Respond now, or do that work now!
  • Defer (perhaps replying requires more time, or requires some work first)

Having a productivity system in place is important (and so is sticking to it)

We are what we frequently do
– Aristotle

Don’t leave email open the entire time, with distracting pop-up notifications. Check once every hour or few hours, live outside of email. *This doesn’t apply to customer service reps

Use a tool/system that just stops short of being fun to use, so that you don’t end up fiddling with it. Remember that the tool/system has to be good at capturing information, and recalling information.