Dustin's Blog

Java Bites

2009-07-19T22:05:00.000-07:00

Josh Cough 1:03
java bites

Dustin Whitney 1:03
but I like to type
HashMap<String, ArrayList<Integer>> myMap = new HashMap<String, ArrayList<Integer>>()

Josh Cough 1:04
i just threw up in my mouth a little

Multi-threaded Testing

2009-07-05T11:29:00.001-07:00

Josh Cough is building a suite for testing multi-threaded scala apps. This is insanely significant!

New York Scala Enthusiasts

2009-02-08T09:27:00.000-08:00

I just started a meetup group for Scala Enthusiasts in NYC. Check it out!

http://www.meetup.com/New-York-Scala-Enthusiasts/

N + 1 Select Revisited

2009-01-28T20:04:00.000-08:00

I ran into an old problem the other day whose solution is common knowledge to even neophyte programmers, and is always used by default to make a database driven app more performant; but now it doesn't necessarily apply anymore, and I had to unlearn what I had always known to be true.

The problem: Suppose I have a table of books and a table of authors; and suppose I want all of the names of the authors of all of the books whose titles start with "A". The amateurish way to do this is the N + 1 Select:

select book_id from book where title like "A%"

then for each book_id

select name from author where book_id = ?

If I have 100 books that start with "A" (N = 100) then I have 101 queries (N + 1) to get my results.

Now suppose my database can return every query in constant time of 20 milliseconds. The above operation takes 101 * 20ms, which is 2020ms, or about 2 seconds. That's pretty bad, right? Right.

The "correct" way to do the above operation is with a join

select name from book, author where title like "A%" and book.book_id = author.book_id

As I said above, all of my queries return in constant time, so the above query gives me the correct results in 20 ms.

But what if I could execute the "select name from author where book_id = ?" query for all of the books concurrently (all at the same time) with threads rather than serially (one after another). Each query still takes 20ms to return, but that 20ms is all at the same time, so really the total elapsed time of the N + 1 queries is 40ms, which isn't even in the same ballpark as the N + 1 queries executed serially. In fact it's more in the ballpark of the "correct" way.

The reason why we don't do this is because most database driven applications involve some sort of connection pooling, so your threads would have to block while waiting for a connection, and even if there were enough connections for one query, running two queries at the same time would completely stop your application, let alone 100 users browsing your app at the same time.

However, that's your typical database. Amazon's SimpleDB has no such limitations. All of the money on your credit card could not buy enough server power to make enough connections to SimpleDB to even phase it. It will simply plug along at the same speed no matter what you do. So when that media campaign hits you hard, the data side of your app keeps chugging along at the same speed it always did. Oh and SimpleDB is super redundant. There is no single point of failure. To get that with a typical database you have to cluster at least three of them together, and guess what happens then? Your database performance drastically drops when performing joins.

SimpleDB, as the above discussion indicates, does not have support for joins, so you must do N + 1 selects and compile your "joins" via concurrency. However, on multi-core processors, this type of behavior is encouraged because you get the most bang for your buck when processing concurrently.

This makes me wonder, "how many other things have I always held to be true, but were really just conditions of being carried out in serial?" The next decade is going to be interesting.

Amazon EC2 and Cloud Tools Saves the Day

2009-01-06T18:55:00.000-08:00

I don't know if I've written about it here or not, but I have spent some time over the past couple of months contributing to an open source project started by Chris Richardson, of "POJOs in Action" fame, called Cloud Tools. It's a set of tools to launch Java based web applications onto Amazon EC2. I've been in charge of moving a lot of the functionality that exists in Maven commands over to Gant scripts for use with Grails. It's pretty rad.

You can launch your Grails application onto a server by simply typing: "grails cloud-tools-deploy". It support all sorts of clustering and master/slave arrangements with DBs and so forth... totally rad.

Anyway, back in October my company, -yoink-, got the go ahead from our client to run one of our web sites on EC2 -yoink-. I elected to use Cloud Tools because it looked great, and I had read Chris's book in the past, so I knew it would be quality.

I can't tell you how much time it saves to be able to bootstrap a server from the command line with a simple "grails cloud-tools-deploy" If you've built any web sites then you know what a pain in the ass deployment is, and you know what a pain in the ass server configuration is. Cloud Tools and EC2 eliminate the need for both and allows you to focus on what you're good at: writing code.

Continuing... today we had a massive email drop, which drove a ton of traffic to the site, that I had no idea was going to happen. The only way I found out is an external monitoring service I had hooked up alerted me to the fact that my server was unable to serve pages. The solution was as simple as opening my app, changing the config to run with 4 app servers, and running "grails cloud-tools-deploy." Once deployed, I switched the Elastic IP address of the old deployment to the load balancing server of my new deployment and voila! My site was hauling ass again.

Had we been using the traditional managed hosting services that we typically use, like those offered by Rackspace, we'd have been totally screwed. It would have taken them at least a week to get another server up and running, which would have been way too long. Using EC2 and Cloud Tools, I simply started servers as I needed them, and I'll shut them down tomorrow when the traffic dies down. It will probably only cost our client about $20 at the most, and think of the time and money that would have been lost if the server configuration had not been able to handle the load being thrown at it.

EC2 is pretty rad.

I donated and so should you

2008-12-31T08:55:00.000-08:00

US CTO: Bill Joy

2008-11-06T07:42:00.000-08:00

I saw this entry on the NYTimes technology blog. Apparently Barack Obama wants to appoint a US CTO, and Bill Joy has been brought to his attention.

Bill Joy has always been a personal technology-hero of mine. He co-founded Sun Microsystems, played a major part in the creation of the SPARC microprocessor, and played a major part in the creation of Java. If you run a mac, it's kernel is built from his branchild, BSD, and if you use unix and edit files, it's likely you use vi, also created by Bill Joy.

Bill Joy is awesome. He retired from Sun in 2003, and now works for a venture capital firm focusing on green energy.

Here is a cool interview on Nerd TV.

However

Bill Joy has been leading this so called neo-luddite movement, which opposes genetic and nanotechnology research for fear that we'll run into a Terminator-like scenario. I agree there are risks with those types of research, but if we don't do the research, someone else will -- and there are HUGE upsides to this type of research.

I believe that free market capitalism works (for the most part), and I don't want to see regulations put on genetic and nanotech research. I support Barack Obama, and I believe government intervention and minor regulations are necessary to fix our current economic crisis, but that intervention and regulation should be focused on our financial sector and should not extend to our technology sector, or other sectors for that matter.

I greatly respect and admire Bill Joy, but his appointment will somewhat worry me.

ETags

2008-10-28T21:58:00.000-07:00

I'm sure your all familiar with ETags, which can be used to determine if you've downloaded a file from a site or not. This works great for images: if you've already downloaded an image, why bother downloading it again? But have you ever thought of using ETags to determine if a user has downloaded your dynamic html?

Let's say you've got a page that takes 15 database queries to generate. I'd bet money on the fact that with just one or two of those queries you could determine, using ETags, whether or not the user has seen the page already. If you can do that, you can return a response code of 304 (Unmodified), so the browser pulls everything from cache -- you'll cut database activity and bandwidth usage to a fraction of what they would be otherwise, which could cut the need for extra servers in your cluster.

Here is a cool blog entry using RoR as an example, but the same technique applies with Grails (and Java):

and here is another more long winded article with Spring/Hibernate

-Dustin

Quicksort in Scala

2008-09-29T22:03:00.001-07:00


def sort(list: List[Int]): List[Int] = {
        list match {
                case List() => list
                case List(_) => list
                case List(_, _*) => {
                        val pivot = list(list.length/2)
                        sort(list.filter(x => x < pivot)) ::: list.filter(x => x == pivot) ::: sort(list.filter(x => x > pivot)) 
                }
        }
}

Erlang is Rad

2008-05-31T14:18:00.000-07:00

I don't know how many times I've brought it up, but Erlang is still rad. Everyone in technology should watch at least the first 13 minutes of this presentation:

http://www.infoq.com/presentations/erlang-software-for-a-concurrent-world

SearchMonkey

2008-05-06T12:07:00.001-07:00

I'm pretty excited about SearchMonkey

Marchitecture

2008-05-03T10:46:00.000-07:00

This is one of the most insightful interviews I've seen in a long time. I don't know if the term "marchitecture" (guess so) existed before, but whomever coined it was a genius! I have worked for places that were sold TONS of middleware that simply put a halt on productivity, and made life hell. Too bad I'm the only geek I know that watches stuff like this :)

Groovy Threads

2008-03-31T20:57:00.000-07:00

I don't know why I think this is so cool, because it's such a no-brainer, but I really like the way Groovy added closures to the Thread class:

tmp.groovy:

def stringOne = 'Hello, ';
def stringTwo = 'World!';
Thread.start{sleep(1000); println stringTwo}
Thread.start{sleep(999); print stringOne}

Groovy Classpath

2008-03-11T09:49:00.000-07:00

I always forget this, so it's getting added to my blog for further reference:

You can place jars in your ${user.home}/.groovy/lib directory to have them automatically loaded into your classpath.

Hello World!

2008-03-09T22:37:00.000-07:00

B-Trees

2008-02-16T20:39:00.000-08:00

I've been thinking about Erlang nodes as B-Tree nodes today (a natural application for the language). I did some reading up on B-Trees on wikipedia and found a video worth the entire number of characters in the article -- the music is awesome. I imagine it being shown to a bunch of CS students in an Indian university by a dorky professor who thinks he can get his students excited if he injects a little Bollywood into his class. Haha!

Erlang fast :)

2008-01-29T20:56:00.001-08:00

I want to write about this right now cause it gets kind of lost on me, so I want to be able to refer back to it. Why is Erlang so fast? Imagine you write a Java program that spawns two threads, and you run said program on a multi-core machine. Pretend both threads have a reference to the same Integer (of the proper Object sort), and at every moment of each thread's existence they are going to increase the value of that Integer by one over and over. Each time each thread attempts to increase the value of the Integer it must perform a check/lock/update/unlock to make sure it can access and update the memory that the Integer refers to. That check/lock/update/unlock slows your program down. On a single core system, the check/lock/update/unlock isn't necessary since only one thread is running at a time, but on multi-core machines threads run concurrently, so the check is necessary. In Erlang you can't update variables. Variables can be given a value once and only once, and since you can't change the value, that check isn't necessary, and that's why Erlang is faster.

I guess this begs the question: how would you write the same program in Erlang? Well in my Programming Erlang book they are very careful to explain that Erlang doesn't have threads, it has processes, and the difference is that threads share information (the Integer in the example given), and processes don't (well, you can pass processes the same variable, but they can't update it, so I guess that's the same thing). So, I guess you'd go about solving the problem in an entirely different manner. I am new to Erlang, so I don't feel comfortable giving a definitive answer, but it seems to me, that you'd simply have to solve problems with a different approach than you would in Java... not drastically different, but different, nonetheless... maybe, one process spawns two processes that return the message "1" every moment which is added to the parent processes' message queue (all processes in Erlang have message queues), which it iterates through to accumulate values. That sounds good to me :) Maybe I'll write both programs this weekend and see which one can get to Integer.MAX_VALUE the fastest.

Keynesian beauty contest

2008-01-27T14:16:00.003-08:00

I had not hear of this before. I like it :)

iTunes :(

2008-01-27T11:35:00.001-08:00

I have an album at work that I bought on iTunes that I really want to listen to at home. I want to listen to it so badly that I actually bought the album again from home. iTunes really ought to let us download music we've already purchased. I understand that bandwidth isn't free, so I'd be totally willing to spend 99 cents to download it again, but I shouldn't have to pay $9.99 to buy the whole album. Yeah maybe I should be more patient and save myself ten bucks, but I don't wanna!

keeping music collections synchronized is a pain in the ass :(

Web Frameworks

2008-01-23T19:52:00.000-08:00

Yesterday it was overcast and I was in a bad mood. I wanted to write an angry rant about how one MUST justify their use of anything but Ruby on Rails or Grails (or, I'm told, Django) when building websites, but I held off, and I'm glad, because when one gets up on a soapbox, they usually end up looking like a fool. However, today I'm in a good mood, and I still feel the same way. I read this today. I haven't used Django, but I've heard great things.

Why do I feel one MUST justify their use of anything but RoR, Grails (or, I'm told, Django)? The reason why is just plain ol'experience. I am currently knee deep in a straight up Java project, using Spring MVC. I think Spring MVC is great (Grails is built on it after all), but every time we encounter a problem, right there in the back of my head is a little voice saying, "why oh why didn't we use Grails (or Rails, or, I'm told, Django)?" At work when I bring up RoR, people are always quick to say, "RoR has problems scaling." Our Java project doesn't scale very well, and it won't ever scale well. We spend so much of our time coding around oddities that are inadvertently brought about by the lack of structure in our project that we don't have time to focus on scaling. Go ahead! Criticize our team for lacking structure, a structure common to the really "good" developers out there. Bah! Lack of structure is status quo, and that's why things like Grails and Rails (or, I'm told, Django) came about. They enforce structure by default. They enforce rich domain models by default (I've been through my share of anemic domain models to know why they are valuable). They enforce separation of concerns, and they are really FUN to use!

So if you aren't going to use one of these hip, new frameworks, that's fine, but you'd better have a good justification for it.

Rate Cuts and the Dollar

2008-01-22T18:54:00.000-08:00

Today the Federal Reserve Bank cut the federal funds rate by 75 basis points. In all the time I've been paying attention (since college) I have never seen a cut this big. Cutting the fed rate is supposed to increase the money supply by encouraging people to borrow. On the down side it can carry inflation along with it, and the dollar did in fact trade down today. What's crazy is that even with a cut like this the DJI and S&P 500 closed down! If a cut like that can't increase investor confidence, then maybe a cut like that won't bring inflation along with it. And if there is an ominous global recession on the horizon, maybe we won't see the dollar decrease much because other central banks will cut their rates too. It'll be interesting to see, so let's start documenting it! Here is a list of exchange rates:

GBP: 1.9577
CAD: 0.9758
EUR: 1.46089
AUD: 0.868402
JPY: 0.009409

I'll keep track over the next while to see what happens.

speling

2008-01-13T20:24:00.001-08:00

A client at work keeps insisting that we improve the search results on one of my current projects, and one of the things they want is the nifty Google "spell check"... you know, the thing that asks if you meant to search for something else when you misspell a word? I once read an article on how it works, and it does some comparisons behind the scenes to see if searching by another similar word will result in a significant amount more of results. The difficulty is in finding those "other words". I found two ways around this. The first is that Google actually makes the spell check available through a REST interface, and you don't even need an API key to use it. Here is some code that will make it work:


import java.net.*;
import java.io.*;

public class Post{

 public static void main(String args[]) throws Exception{
  String data = "<?xml version=\"1.0\" encoding=\"utf-8\" ?><spellrequest textalreadyclipped=\"0\" ignoredups=\"0\" ignoredigits=\"1\" ignoreallcaps=\"1\"><text>I lik pzza</text></spellrequest>";
  URL url = new URL("https://www.google.com/tbproxy/spell?lang=en&hl=en");
  URLConnection connection = url.openConnection();
  connection.setDoOutput(true);
  OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
  writer.write(data);
  writer.flush();

  BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
  String line;
  while((line = reader.readLine()) != null){
   System.out.println(line);
  }
   writer.close();
   reader.close();
 }
}

My worry with using Google is that they'd decide they didn't like me querying that service every time a user does a search, and it's very likely against their terms and services agreement. But lo, dear reader, I found another method of generating those "other words" It's in this great article by Peter Norvig who happens to be a director of research at Google. So Thanks Peter, I never could have done it without you.

Hello World!

2007-12-15T13:31:00.000-08:00

I put up my first Amazon EC2 instance today, and though the link will probably either be broken or open up someone else's instance by the time you click on it, this marks a huge milestone in computing, and I'm happy to be a part of it! And hey, look! John Young is doing it too!

The fact that this service is SOOO DIRT CHEAP means the doors are open to anyone who has any grand ideas. Server costs used to be a barrier, and now they simply aren't. Oh, and look! An article on how to get Hadoop running on EC2!

Good Ol'Capitalism

2007-10-30T13:33:00.000-07:00

NBC claims Apple has destroyed the music industry in terms of pricing. That's capitalism at its finest -- execs acting like cry-babies when innovation ruins their business by making it more competitive. Large media companies could definitely be on the decline, but it's at the benefit of consumers, so overall, it's a good thing. The same thing needs to happen in the real-estate world, cause they are a bunch of price fixers that are directly impacting me! Maybe I should be the one to do it :)

First Post!

2007-10-23T19:33:00.000-07:00

I've been telling myself that I'd build my own blog for the last couple of years. It hasn't happened, so thank you Google for doing what I didn't, or at least purchasing a company that did what I didn't and then linking it to my existing account.

Hmm.... there is a part of me that kinda sorta likes the Windows Live Journal thing. I mean, I'm writing this from a mac, so right there Windows Live Journal sucks, but it natively supports APP, which I find cool. I don't know if blogger supports APP. If APP gets popular enough it will.

Well there you go, Dustin Ted Whitney's first post. Geeky and critical -- quintessential Dustin Ted Whitney.