Josh Cough 1:03
java bites
Dustin Whitney 1:03
but I like to type
HashMap<String, ArrayList<Integer>> myMap = new HashMap<String, ArrayList<Integer>>()
Josh Cough 1:04
i just threw up in my mouth a little
Sunday, July 19, 2009
Sunday, July 5, 2009
Multi-threaded Testing
Josh Cough is building a suite for testing multi-threaded scala apps. This is insanely significant!
Sunday, February 8, 2009
New York Scala Enthusiasts
I just started a meetup group for Scala Enthusiasts in NYC. Check it out!
http://www.meetup.com/New-York-Scala-Enthusiasts/
http://www.meetup.com/New-York-Scala-Enthusiasts/
Wednesday, January 28, 2009
N + 1 Select Revisited
I ran into an old problem the other day whose solution is common knowledge to even neophyte programmers, and is always used by default to make a database driven app more performant; but now it doesn't necessarily apply anymore, and I had to unlearn what I had always known to be true.
The problem: Suppose I have a table of books and a table of authors; and suppose I want all of the names of the authors of all of the books whose titles start with "A". The amateurish way to do this is the N + 1 Select:
select book_id from book where title like "A%"
then for each book_id
select name from author where book_id = ?
If I have 100 books that start with "A" (N = 100) then I have 101 queries (N + 1) to get my results.
Now suppose my database can return every query in constant time of 20 milliseconds. The above operation takes 101 * 20ms, which is 2020ms, or about 2 seconds. That's pretty bad, right? Right.
The "correct" way to do the above operation is with a join
select name from book, author where title like "A%" and book.book_id = author.book_id
As I said above, all of my queries return in constant time, so the above query gives me the correct results in 20 ms.
But what if I could execute the "select name from author where book_id = ?" query for all of the books concurrently (all at the same time) with threads rather than serially (one after another). Each query still takes 20ms to return, but that 20ms is all at the same time, so really the total elapsed time of the N + 1 queries is 40ms, which isn't even in the same ballpark as the N + 1 queries executed serially. In fact it's more in the ballpark of the "correct" way.
The reason why we don't do this is because most database driven applications involve some sort of connection pooling, so your threads would have to block while waiting for a connection, and even if there were enough connections for one query, running two queries at the same time would completely stop your application, let alone 100 users browsing your app at the same time.
However, that's your typical database. Amazon's SimpleDB has no such limitations. All of the money on your credit card could not buy enough server power to make enough connections to SimpleDB to even phase it. It will simply plug along at the same speed no matter what you do. So when that media campaign hits you hard, the data side of your app keeps chugging along at the same speed it always did. Oh and SimpleDB is super redundant. There is no single point of failure. To get that with a typical database you have to cluster at least three of them together, and guess what happens then? Your database performance drastically drops when performing joins.
SimpleDB, as the above discussion indicates, does not have support for joins, so you must do N + 1 selects and compile your "joins" via concurrency. However, on multi-core processors, this type of behavior is encouraged because you get the most bang for your buck when processing concurrently.
This makes me wonder, "how many other things have I always held to be true, but were really just conditions of being carried out in serial?" The next decade is going to be interesting.
The problem: Suppose I have a table of books and a table of authors; and suppose I want all of the names of the authors of all of the books whose titles start with "A". The amateurish way to do this is the N + 1 Select:
select book_id from book where title like "A%"
then for each book_id
select name from author where book_id = ?
If I have 100 books that start with "A" (N = 100) then I have 101 queries (N + 1) to get my results.
Now suppose my database can return every query in constant time of 20 milliseconds. The above operation takes 101 * 20ms, which is 2020ms, or about 2 seconds. That's pretty bad, right? Right.
The "correct" way to do the above operation is with a join
select name from book, author where title like "A%" and book.book_id = author.book_id
As I said above, all of my queries return in constant time, so the above query gives me the correct results in 20 ms.
But what if I could execute the "select name from author where book_id = ?" query for all of the books concurrently (all at the same time) with threads rather than serially (one after another). Each query still takes 20ms to return, but that 20ms is all at the same time, so really the total elapsed time of the N + 1 queries is 40ms, which isn't even in the same ballpark as the N + 1 queries executed serially. In fact it's more in the ballpark of the "correct" way.
The reason why we don't do this is because most database driven applications involve some sort of connection pooling, so your threads would have to block while waiting for a connection, and even if there were enough connections for one query, running two queries at the same time would completely stop your application, let alone 100 users browsing your app at the same time.
However, that's your typical database. Amazon's SimpleDB has no such limitations. All of the money on your credit card could not buy enough server power to make enough connections to SimpleDB to even phase it. It will simply plug along at the same speed no matter what you do. So when that media campaign hits you hard, the data side of your app keeps chugging along at the same speed it always did. Oh and SimpleDB is super redundant. There is no single point of failure. To get that with a typical database you have to cluster at least three of them together, and guess what happens then? Your database performance drastically drops when performing joins.
SimpleDB, as the above discussion indicates, does not have support for joins, so you must do N + 1 selects and compile your "joins" via concurrency. However, on multi-core processors, this type of behavior is encouraged because you get the most bang for your buck when processing concurrently.
This makes me wonder, "how many other things have I always held to be true, but were really just conditions of being carried out in serial?" The next decade is going to be interesting.
Tuesday, January 6, 2009
Amazon EC2 and Cloud Tools Saves the Day
I don't know if I've written about it here or not, but I have spent some time over the past couple of months contributing to an open source project started by Chris Richardson, of "POJOs in Action" fame, called Cloud Tools. It's a set of tools to launch Java based web applications onto Amazon EC2. I've been in charge of moving a lot of the functionality that exists in Maven commands over to Gant scripts for use with Grails. It's pretty rad.
You can launch your Grails application onto a server by simply typing: "grails cloud-tools-deploy". It support all sorts of clustering and master/slave arrangements with DBs and so forth... totally rad.
Anyway, back in October my company, -yoink-, got the go ahead from our client to run one of our web sites on EC2 -yoink-. I elected to use Cloud Tools because it looked great, and I had read Chris's book in the past, so I knew it would be quality.
I can't tell you how much time it saves to be able to bootstrap a server from the command line with a simple "grails cloud-tools-deploy" If you've built any web sites then you know what a pain in the ass deployment is, and you know what a pain in the ass server configuration is. Cloud Tools and EC2 eliminate the need for both and allows you to focus on what you're good at: writing code.
Continuing... today we had a massive email drop, which drove a ton of traffic to the site, that I had no idea was going to happen. The only way I found out is an external monitoring service I had hooked up alerted me to the fact that my server was unable to serve pages. The solution was as simple as opening my app, changing the config to run with 4 app servers, and running "grails cloud-tools-deploy." Once deployed, I switched the Elastic IP address of the old deployment to the load balancing server of my new deployment and voila! My site was hauling ass again.
Had we been using the traditional managed hosting services that we typically use, like those offered by Rackspace, we'd have been totally screwed. It would have taken them at least a week to get another server up and running, which would have been way too long. Using EC2 and Cloud Tools, I simply started servers as I needed them, and I'll shut them down tomorrow when the traffic dies down. It will probably only cost our client about $20 at the most, and think of the time and money that would have been lost if the server configuration had not been able to handle the load being thrown at it.
EC2 is pretty rad.
You can launch your Grails application onto a server by simply typing: "grails cloud-tools-deploy". It support all sorts of clustering and master/slave arrangements with DBs and so forth... totally rad.
Anyway, back in October my company, -yoink-, got the go ahead from our client to run one of our web sites on EC2 -yoink-. I elected to use Cloud Tools because it looked great, and I had read Chris's book in the past, so I knew it would be quality.
I can't tell you how much time it saves to be able to bootstrap a server from the command line with a simple "grails cloud-tools-deploy" If you've built any web sites then you know what a pain in the ass deployment is, and you know what a pain in the ass server configuration is. Cloud Tools and EC2 eliminate the need for both and allows you to focus on what you're good at: writing code.
Continuing... today we had a massive email drop, which drove a ton of traffic to the site, that I had no idea was going to happen. The only way I found out is an external monitoring service I had hooked up alerted me to the fact that my server was unable to serve pages. The solution was as simple as opening my app, changing the config to run with 4 app servers, and running "grails cloud-tools-deploy." Once deployed, I switched the Elastic IP address of the old deployment to the load balancing server of my new deployment and voila! My site was hauling ass again.
Had we been using the traditional managed hosting services that we typically use, like those offered by Rackspace, we'd have been totally screwed. It would have taken them at least a week to get another server up and running, which would have been way too long. Using EC2 and Cloud Tools, I simply started servers as I needed them, and I'll shut them down tomorrow when the traffic dies down. It will probably only cost our client about $20 at the most, and think of the time and money that would have been lost if the server configuration had not been able to handle the load being thrown at it.
EC2 is pretty rad.
Wednesday, December 31, 2008
Thursday, November 6, 2008
US CTO: Bill Joy
I saw this entry on the NYTimes technology blog. Apparently Barack Obama wants to appoint a US CTO, and Bill Joy has been brought to his attention.
Bill Joy has always been a personal technology-hero of mine. He co-founded Sun Microsystems, played a major part in the creation of the SPARC microprocessor, and played a major part in the creation of Java. If you run a mac, it's kernel is built from his branchild, BSD, and if you use unix and edit files, it's likely you use vi, also created by Bill Joy.
Bill Joy is awesome. He retired from Sun in 2003, and now works for a venture capital firm focusing on green energy.
Here is a cool interview on Nerd TV.
Bill Joy has been leading this so called neo-luddite movement, which opposes genetic and nanotechnology research for fear that we'll run into a Terminator-like scenario. I agree there are risks with those types of research, but if we don't do the research, someone else will -- and there are HUGE upsides to this type of research.
I believe that free market capitalism works (for the most part), and I don't want to see regulations put on genetic and nanotech research. I support Barack Obama, and I believe government intervention and minor regulations are necessary to fix our current economic crisis, but that intervention and regulation should be focused on our financial sector and should not extend to our technology sector, or other sectors for that matter.
I greatly respect and admire Bill Joy, but his appointment will somewhat worry me.
Bill Joy has always been a personal technology-hero of mine. He co-founded Sun Microsystems, played a major part in the creation of the SPARC microprocessor, and played a major part in the creation of Java. If you run a mac, it's kernel is built from his branchild, BSD, and if you use unix and edit files, it's likely you use vi, also created by Bill Joy.
Bill Joy is awesome. He retired from Sun in 2003, and now works for a venture capital firm focusing on green energy.
Here is a cool interview on Nerd TV.
However
Bill Joy has been leading this so called neo-luddite movement, which opposes genetic and nanotechnology research for fear that we'll run into a Terminator-like scenario. I agree there are risks with those types of research, but if we don't do the research, someone else will -- and there are HUGE upsides to this type of research.
I believe that free market capitalism works (for the most part), and I don't want to see regulations put on genetic and nanotech research. I support Barack Obama, and I believe government intervention and minor regulations are necessary to fix our current economic crisis, but that intervention and regulation should be focused on our financial sector and should not extend to our technology sector, or other sectors for that matter.
I greatly respect and admire Bill Joy, but his appointment will somewhat worry me.
Tuesday, October 28, 2008
ETags
I'm sure your all familiar with ETags, which can be used to determine if you've downloaded a file from a site or not. This works great for images: if you've already downloaded an image, why bother downloading it again? But have you ever thought of using ETags to determine if a user has downloaded your dynamic html?
Let's say you've got a page that takes 15 database queries to generate. I'd bet money on the fact that with just one or two of those queries you could determine, using ETags, whether or not the user has seen the page already. If you can do that, you can return a response code of 304 (Unmodified), so the browser pulls everything from cache -- you'll cut database activity and bandwidth usage to a fraction of what they would be otherwise, which could cut the need for extra servers in your cluster.
Here is a cool blog entry using RoR as an example, but the same technique applies with Grails (and Java):
and here is another more long winded article with Spring/Hibernate
-Dustin
Let's say you've got a page that takes 15 database queries to generate. I'd bet money on the fact that with just one or two of those queries you could determine, using ETags, whether or not the user has seen the page already. If you can do that, you can return a response code of 304 (Unmodified), so the browser pulls everything from cache -- you'll cut database activity and bandwidth usage to a fraction of what they would be otherwise, which could cut the need for extra servers in your cluster.
Here is a cool blog entry using RoR as an example, but the same technique applies with Grails (and Java):
and here is another more long winded article with Spring/Hibernate
-Dustin
Monday, September 29, 2008
Quicksort in Scala
def sort(list: List[Int]): List[Int] = {
list match {
case List() => list
case List(_) => list
case List(_, _*) => {
val pivot = list(list.length/2)
sort(list.filter(x => x < pivot)) ::: list.filter(x => x == pivot) ::: sort(list.filter(x => x > pivot))
}
}
}
Saturday, May 31, 2008
Erlang is Rad
I don't know how many times I've brought it up, but Erlang is still rad. Everyone in technology should watch at least the first 13 minutes of this presentation:
http://www.infoq.com/presentations/erlang-software-for-a-concurrent-world
http://www.infoq.com/presentations/erlang-software-for-a-concurrent-world
Tuesday, May 6, 2008
Saturday, May 3, 2008
Marchitecture
This is one of the most insightful interviews I've seen in a long time. I don't know if the term "marchitecture" (guess so) existed before, but whomever coined it was a genius! I have worked for places that were sold TONS of middleware that simply put a halt on productivity, and made life hell. Too bad I'm the only geek I know that watches stuff like this :)
Monday, March 31, 2008
Groovy Threads
I don't know why I think this is so cool, because it's such a no-brainer, but I really like the way Groovy added closures to the Thread class:
tmp.groovy:
def stringOne = 'Hello, ';
def stringTwo = 'World!';
Thread.start{sleep(1000); println stringTwo}
Thread.start{sleep(999); print stringOne}
tmp.groovy:
def stringOne = 'Hello, ';
def stringTwo = 'World!';
Thread.start{sleep(1000); println stringTwo}
Thread.start{sleep(999); print stringOne}
Tuesday, March 11, 2008
Groovy Classpath
I always forget this, so it's getting added to my blog for further reference:
You can place jars in your ${user.home}/.groovy/lib directory to have them automatically loaded into your classpath.
You can place jars in your ${user.home}/.groovy/lib directory to have them automatically loaded into your classpath.
Sunday, March 9, 2008
Saturday, February 16, 2008
B-Trees
I've been thinking about Erlang nodes as B-Tree nodes today (a natural application for the language). I did some reading up on B-Trees on wikipedia and found a video worth the entire number of characters in the article -- the music is awesome. I imagine it being shown to a bunch of CS students in an Indian university by a dorky professor who thinks he can get his students excited if he injects a little Bollywood into his class. Haha!
Tuesday, January 29, 2008
Erlang fast :)
I want to write about this right now cause it gets kind of lost on me, so I want to be able to refer back to it. Why is Erlang so fast? Imagine you write a Java program that spawns two threads, and you run said program on a multi-core machine. Pretend both threads have a reference to the same Integer (of the proper Object sort), and at every moment of each thread's existence they are going to increase the value of that Integer by one over and over. Each time each thread attempts to increase the value of the Integer it must perform a check/lock/update/unlock to make sure it can access and update the memory that the Integer refers to. That check/lock/update/unlock slows your program down. On a single core system, the check/lock/update/unlock isn't necessary since only one thread is running at a time, but on multi-core machines threads run concurrently, so the check is necessary. In Erlang you can't update variables. Variables can be given a value once and only once, and since you can't change the value, that check isn't necessary, and that's why Erlang is faster.
I guess this begs the question: how would you write the same program in Erlang? Well in my Programming Erlang book they are very careful to explain that Erlang doesn't have threads, it has processes, and the difference is that threads share information (the Integer in the example given), and processes don't (well, you can pass processes the same variable, but they can't update it, so I guess that's the same thing). So, I guess you'd go about solving the problem in an entirely different manner. I am new to Erlang, so I don't feel comfortable giving a definitive answer, but it seems to me, that you'd simply have to solve problems with a different approach than you would in Java... not drastically different, but different, nonetheless... maybe, one process spawns two processes that return the message "1" every moment which is added to the parent processes' message queue (all processes in Erlang have message queues), which it iterates through to accumulate values. That sounds good to me :) Maybe I'll write both programs this weekend and see which one can get to Integer.MAX_VALUE the fastest.
I guess this begs the question: how would you write the same program in Erlang? Well in my Programming Erlang book they are very careful to explain that Erlang doesn't have threads, it has processes, and the difference is that threads share information (the Integer in the example given), and processes don't (well, you can pass processes the same variable, but they can't update it, so I guess that's the same thing). So, I guess you'd go about solving the problem in an entirely different manner. I am new to Erlang, so I don't feel comfortable giving a definitive answer, but it seems to me, that you'd simply have to solve problems with a different approach than you would in Java... not drastically different, but different, nonetheless... maybe, one process spawns two processes that return the message "1" every moment which is added to the parent processes' message queue (all processes in Erlang have message queues), which it iterates through to accumulate values. That sounds good to me :) Maybe I'll write both programs this weekend and see which one can get to Integer.MAX_VALUE the fastest.
Sunday, January 27, 2008
iTunes :(
I have an album at work that I bought on iTunes that I really want to listen to at home. I want to listen to it so badly that I actually bought the album again from home. iTunes really ought to let us download music we've already purchased. I understand that bandwidth isn't free, so I'd be totally willing to spend 99 cents to download it again, but I shouldn't have to pay $9.99 to buy the whole album. Yeah maybe I should be more patient and save myself ten bucks, but I don't wanna!
keeping music collections synchronized is a pain in the ass :(
keeping music collections synchronized is a pain in the ass :(
Wednesday, January 23, 2008
Web Frameworks
Yesterday it was overcast and I was in a bad mood. I wanted to write an angry rant about how one MUST justify their use of anything but Ruby on Rails or Grails (or, I'm told, Django) when building websites, but I held off, and I'm glad, because when one gets up on a soapbox, they usually end up looking like a fool. However, today I'm in a good mood, and I still feel the same way. I read this today. I haven't used Django, but I've heard great things.
Why do I feel one MUST justify their use of anything but RoR, Grails (or, I'm told, Django)? The reason why is just plain ol'experience. I am currently knee deep in a straight up Java project, using Spring MVC. I think Spring MVC is great (Grails is built on it after all), but every time we encounter a problem, right there in the back of my head is a little voice saying, "why oh why didn't we use Grails (or Rails, or, I'm told, Django)?" At work when I bring up RoR, people are always quick to say, "RoR has problems scaling." Our Java project doesn't scale very well, and it won't ever scale well. We spend so much of our time coding around oddities that are inadvertently brought about by the lack of structure in our project that we don't have time to focus on scaling. Go ahead! Criticize our team for lacking structure, a structure common to the really "good" developers out there. Bah! Lack of structure is status quo, and that's why things like Grails and Rails (or, I'm told, Django) came about. They enforce structure by default. They enforce rich domain models by default (I've been through my share of anemic domain models to know why they are valuable). They enforce separation of concerns, and they are really FUN to use!
So if you aren't going to use one of these hip, new frameworks, that's fine, but you'd better have a good justification for it.
Why do I feel one MUST justify their use of anything but RoR, Grails (or, I'm told, Django)? The reason why is just plain ol'experience. I am currently knee deep in a straight up Java project, using Spring MVC. I think Spring MVC is great (Grails is built on it after all), but every time we encounter a problem, right there in the back of my head is a little voice saying, "why oh why didn't we use Grails (or Rails, or, I'm told, Django)?" At work when I bring up RoR, people are always quick to say, "RoR has problems scaling." Our Java project doesn't scale very well, and it won't ever scale well. We spend so much of our time coding around oddities that are inadvertently brought about by the lack of structure in our project that we don't have time to focus on scaling. Go ahead! Criticize our team for lacking structure, a structure common to the really "good" developers out there. Bah! Lack of structure is status quo, and that's why things like Grails and Rails (or, I'm told, Django) came about. They enforce structure by default. They enforce rich domain models by default (I've been through my share of anemic domain models to know why they are valuable). They enforce separation of concerns, and they are really FUN to use!
So if you aren't going to use one of these hip, new frameworks, that's fine, but you'd better have a good justification for it.
Subscribe to:
Posts (Atom)
