As an offshoot on another thread someone asked for some feedback on development with Rails. They pointed out that there are plenty of pro-Rails resources on the net but were curious to hear some of the other side of experience with Rails.

This post is an edited version of the private reply I sent. To be honest I initially wasn’t going to put this up in public at all for risk of the wrath of the fan-boy legion, but I thought it was worth sharing a few of these thoughts.

First of all, let me say that Rails isn’t bad, and many Rails developers are fine developers. One of the smartest guys in the world I know loves Rails for all the right reasons and uses it very effectively.

I’d classify my caution about starting a project in Rails in three buckets-

Immature

Rails has progressed a bunch since I worked with it, but its still very immature compared to the other environments. Any Rails deployment that has more than a minor load (needs more than a shared hosting environment or on the order of a few thousand visits a day) will typically require a bunch of time dealing with infrastructure issues. When you install just about any modern copy of linux you are pretty well setup to run PHP or other more mature environments out of the box. Sophisticated people can tweak some stuff, but you get apache, mod_php and you are ready to go.

For Rails you don’t hit this much during development since everything is running on your machine with their nice little single threaded local server. But the minute you have any real load (lets call that 5 people are hitting your site at once) it can be a problem. I’m sure since my experience these things have improved a lot, but I still hear all the time from people about the pain of dealing with Mongrel and other deployment issues. What other major web app runtime doesn’t have an apache module? There are some of the core-Apache folks who I would listen to if they told me that Mongrel was really the right way to build this, but I haven’t heard from them yet.

Furthermore lots of the infrastructure is still coming along. There are some debuggers now, but they are still very new and given how much Rails does for you under the covers (more on this later) it can be really hard to figure out what went wrong. What it comes down to is do you really want to invest your resources into running Rails or into running your business? There are some very successful bigger sites that are built on Rails but they tend to have a dedicated staff of Rails experts, many of whom contribute to the Rails codebase. If you don’t have that guy, you need to hire that guy (not the one who thinks Rails is cool and picked up a book on it 6 months ago, but the one who really gets the details of the architecture and deployment) or you should be investing somewhere else.

Abstraction

For me, Rails typically does not operate at my favorite level of abstraction. All computer programming is about working in different levels of abstraction (unless you are at Intel working on the inners of the CPU and even there most of those folk work in abstractions, rather than individually manipulate individual logic gates). Picking the right level is the key. Rails really pushes you to a certain level of abstraction that makes rapid development easy, but often you don’t realize what is really going on under the covers since it handles it all for you. It is really easy to accidentally write a loop that does a database query through every iteration, but you don’t realize that is what is happening since it just looks like safe object references. Its easy to have too nested loops each doing more queries and just trash your performance.

All these things can be fixed by careful tuning and after all you have all the sources to this stuff. But Rails sets people up to work at that other layer and I’ve rarely met Rails developers who actually built a log of every SQL query that their app did to tune their performance. Again, this can be invisible if you only have a few users hitting your site every minute or for demos, but falls over quickly (and can be lots of work to fix) later.

This issue isn’t unique to Rails. .NET users can fall into the same traps if they just use all the existing libraries although the standard libraries from Microsoft tend to be a bit careful about making this stuff explicit. Any environment can have a framework that causes this problem although Rail’s ability to do late binding on objects can make it even harder to see what is happening. For example, I’ve done some work lately with Drupal, a framework written in PHP and it typically suffers from similar issues.

What’s The Big Deal

My personal reaction to the RoR hype is “what’s the big deal?” A talented developer with the right architectural expertise can do almost equivalent rapid development of web apps in PHP, .NET, Python, Java or Rails. They all have great tools and if you take the right approach they can all be pretty equivalent. The advantage of Rails is that it “puts you on rails”. Often, web-developers working with PHP, .NET or Java can build themselves a mess of either hacky spaghetti code or over-architected, unnecessarily complex stuff. I’ve seen both. Rails is very proscriptive about how you build an app so if your app fits into their model (typical data-driven web site stuff), most developers will be more likely to get it almost right.

Beyond that there are other stylistic things. Personally, I prefer C-style syntaxes over Smalltalk-style, so I’m more comfortable in C#, PHP, or Java. Other people really love the Ruby language.

There are some actual differences in the runtimes that you can poke at, putting some Pros and Cons in various columns but they are mostly a wash for general web-apps. There are some specialized apps that require certain perf characteristics that would steer me towards .NET or Java (where you get compiled performance and the ability to keep objects in local RAM in-between requests). Certain other apps have different characteristics that would steer me towards PHP, but for most it doesn’t really matter.

Again, Rails isn’t bad and the last thing in the world I would want to do is imply that I don’t approve of someone’s skills because they prefer Rails (or even that I would avoid a Rails project, despite it not being my first choice). I’m not a fan of some of the hype/fan-boy attitude surrounding Rails and I would advise folks that work in Rails to invest the time into understanding the details of what goes on in their environment, especially with respect to how it impacts database performance.

There has been quite the kerfuffle on various blogs and mailing lists about various approaches to work. It appears to have mostly started off as an exchange between 37signals and Jason Calacanis, but it spread to various nasty exchanges on some mailing lists as well as other posts like this one from my friends at Jackson Fish.

At some point I made the following post on the Seattle Tech Startups mailing list (edited slightly)-

Not every startup is the same.

Not every person is the same.

Some people are not at all suited for any startup. They thrive with a certain stability only a big company will give. Not that you can necessarily be complacent at a big company, but there is a stability there that no startup will ever have.

Some people just love to work. They will thrive in a startup that has that culture.

Some people take a more balanced approach towards live and work. They probably bring useful things to the table beyond lots of hours.

What is important to realize is that every startup will have a culture. And people who don’t fit with it will probably be miserable. If your startup has a “go to coffee twice a day and every chat is over a long lunch” culture, the guy who just wants to sit and crank out code all day will be frustrated with their colleagues. And they will be frustrated with that guy (who doesn’t spend as much time getting in sync with everyone else).

At the same time, if you have a culture where people just crank on stuff all the time, the person who takes the different approach will not be appreciated very much either.

Go figure.

Some businesses will be a better fit for one or the other. Its probably hard to get a new release out every week or two on the more relaxed/thoughtful plan. If you are in a market that requires that (because the competition is going to do that and will beat you if you don’t) you had better figure out how to make that your company culture or else go for a different market.

Other markets have very different expectations. They won’t go for a quick to market but unpolished product and they don’t want new releases (and thus change and more training) all the time anyway.

Fit your culture to your business (or vice versa). Fit your people to your culture. Don’t try to force a fit where its not going to happen because its not going to work.

And be understanding that one size doesn’t fit all. Stop calling people slave-drivers because they enjoy the driven culture and build a business that fits that, and start calling other people lazy because they aren’t putting in 80 hour work weeks. Each has its place.

I’m doing some Facebook developing using PHP at the moment. Overall it seems more smooth than the .NET stuff did, at least back 4 months ago when I was digging into it most. Part of the issue is that .NET provides a ton of different ways to build things- code behind, server controls, HTTP handlers, etc. So adding an infrastructure thing like Facebook based authentication can be taken in a bunch of different directions.

Of course this whole platform is still very young. They are just starting to come out with debugging tools and test environments and it can take a lot of time to keep up on it, not to mention that the actual APIs change quite a bit. I’m still fixing one of the first applications that I wrote (its not public yet) to update to the changes that removed the API for sending invites. Now you have to do it with one of their controls. Given the spam problem with apps just broadcasting invites, its a reasonable solution, but it can be a real pain keeping up with their platform changes.

Subscribing to their developer news feed (from http://developer.facebook.com/news.php?blog=1&format=xml ) is pretty crucial for anyone working in this space.

Meanwhile we are starting to investigate the OpenSocial stuff, although it still seems a bit early and unformed. I still have not heard from anyone who has used it in a real meaningful way, but I’m looking for a good chance to dig in.

I’ve been developing some applications for Facebook lately. I got past the first couple of blocks fairly quickly and now have an application mostly working.

Of course there are always more issues. I expect many of these issues are typical for anything that is delivered as a service developer platform. The Amazon developer platforms are more back-end technology so its a bit easier for them to tackle some of these things.

The biggest issue so far is in general development process and staging. Since my application integrates with the Facebook UI and social features, its not entirely clear how to best manage those aspects. I usually maintain two copies of my web-sites, one that is the active public one and another private copy that I’m using for development and testing purposes. Actually there are usually 3- active development usually goes on using a local server on my workstation. The staging server typically doesn’t have anything that is especially secret, but it isn’t something that I’m worried about crashing/returning exceptions if something is not fully debugged.

With Facebook any site that working on needs to be something that can be accessed on the public Internet by their servers so the local developer-workstation test environment does not work at all.

So far I’ve managed without a seperate set of staging servers. Since my app is not yet open to the public, I don’t need to worry about breaking real users. But once I open the application up, I’ll need to deal with this. I assume I need to just register a second copy of the application with a different host header. The Facebook platform makes you use absolute URLs for everything which makes this a bit of a pain but I’m sure its something I can manage.

Since I can’t directly debug on my workstation as easily as normal (I could use remote debugging to the server but there are assorted reasons I don’t want to do that) I’ve had to rely on logging more for debugging. I’m using a “tail” utility for Windows that lets me display the log output in a window in real-time.

I’ve also signed up a second account to use as a test account to test the collaboration aspects. Of course this means I had to turn off the “developer mode” so the other account can access my application, and this in theory means that others could access the application. For now I just put in a server-side block so only those two accounts can get into it. Still, it wouldn’t be ideal if I really cared about stealth.

More on some techniques I’m using in the application itself soon. This platform is pretty new so there is only limited and scattered information about it around the web. I’ll try to collect some links here and post any tips I run across. I’m sure some of my suggestions will be wrong, but as usual I’m open to feedback and suggestions.

Yet another CouponLooker update went live today. The most noticeable additions are recommended sub-queries for % off coupons and specific merchants. For the specific merchants it was necessary to add a mechanism to normalize the merchant names, since the coupon feeds we get sometimes list Dell as Dell.Com, Dell Home, Dell USA, or about 40 other variations.

The other bigger project was mostly not visible. We are experimenting with sponsored coupons, which would allow the provider of a coupon feed to pay us to get a higher listing for their coupon. This is tricky since our top priority is to maintain the quality of the results we give users- just because someone is paying doesn’t mean its ok to show some crap unrelated coupon. So the system is built around some fairly sophisticated scoring based on the initial score, click through rates (if a coupon is relevant, people will click it, if its less interesting it gets ignored) and more.

The system we use bears some resemblance to some of the similar stuff from Google. Its interesting observing the dynamic in discussing emulating Google’s approach to something. Google has been such an extreme success story that its a bit tempting to take an initial position that you can just emulate them as the path to sure riches.

Back in reality its important to break it down into two pieces. First of all, the amazing revenue that Google enjoys is not just a product of their technology, rather its supported by their technology. Emulating the technology doesn’t directly give you any revenue. You need a good business plan and model, and I’ll leave that to others (from my seat its in good hands at Judy’s Book).

On the technical side it doesn’t just work to copy either. Instead its important to really understand the “why” behind the mechanisms that work and understand how you can apply that same learning to your different problem. So for example, we use a system that supports bidding for ad placement. We measure click-through rates to adjust the value of the ads. But other items are very different- Google ads are separate from the content, whereas in CouponLooker, someone is sponsoring a specific coupon that is part of our existing content.

Adding the sponsored coupons also introduced some new technical issues to deal with. Until now the usage of the system needed little back and forth with the database. In effect the engine could load all the coupons, and then search them from there with little changes until new coupons are loaded. With the new system there are much more frequent changes that an advertiser can create by placing new ads, running out of their daily budget or other factors which can change the results over shorter time-frames. I feel like the in-memory approach to the search engine really paid off here since it gave me lots of flexibility to change the details of computing the results. If I had relied much more on pre-computed indicies and results it would have been much harder to make this stuff work.

Which isn’t to say that it won’t be possible to pre-computer stuff later, but just that if you do that before you really have the details of the system worked out, you will have set in stone mechanisms that are harder to adjust.

Rahul writes about our latest update to CouponLooker. This is our third and its one of the more exciting things to me how quick it is to update and tweak the search algorithm. There are some pretty sophisticated things going on inside the engine but things have been designed with an eye towards flexibility and rapid iteration. I’m sure the equivalent results could be done with an approach that uses half as many CPU cycles, but frankly CPU cycles are relatively cheap now and making rapid improvements are invaluable.

One detail that you won’t see using the service directly is that I’ve put in some A-B logic for our own development purposes. I can put the site into a special mode where whenever it does a search it does it twice using random different permutations of our algorithm. We can then experiment with this in the office or with test subjects and ask people to search for useful things and give the system feedback on which of the result-sets look more useful.

One of the interesting projects I’ve worked on recently is building the CouponLooker vertical search engine for Judy’s Book along with some of their in-house developers and designers. Vertical search is a recently popular category as people have realized that Google, while it does a great job for general content topics, often returns poor results when in various specific categories. Those categories are often marked by either structured data or else a commerce related reason why people have spammed the general search results (or ideally both). My own CalendarData.com site tackles one such vertical search category (public events) and CouponLooker aims at another one that is covered by both of the good criteria.

The scenario for CouponLooker is pretty simple. You are buying your favorite stuff somewhere on the Internet and are checking out when you see a textbox that looks like this-

Promo code? Coupon code? I dunno, is there a coupon for this vendor? Am I the chump who is paying full price while others are saving? But go search on Google and you will likely get lots of results that try to steer your purchase to other stores, that provide old expired info or just generic results that aren’t very useful at all.

The search site we build has 4 main components-

1) Data acquisition- pulling in coupon data, processing it and storing it.

2) The search engine- picking which coupons are the best matches for a user’s search.

3) The web-site itself

4) The blog widgets that let you host CouponLooker search on your own blog.

I’ll go into more details about the designs of these components in subsequent posts (but I’m not saying I’m going to cover them in any specific order).

When we switched over to WordPress we switched which server (actually virtual server) that this site is running on. For some reason the DNS for www.launch21.com is being very slow to update to the correct IP address (.233) everywhere so we have been seeing this site bounce between the new one (this) and the old one (with a black background). Sorry about the inconvenience and we hope to have anything straightened up soon.

All that I can say is that we got off to a good start, getting our first web-site up within a week of getting rolling on Launch21. From there we quickly faltered on the “often” part of the phrase “ship early and often”. I thought its worthwhile to reflect on what happened.

The really positive take on things is that both Peyman and I have been incredibly busy working for clients and haven’t had much time to work on the site. I know it sounds like some form of posing, but if anything the demand for the stuff we do has exceeded my expectations. I’m incredibly grateful for the clients that we have signed up so far and want to focus on keeping them happy, which hasn’t left much time for fixing up this site.

That alone wouldn’t have kept us from some simple improvements, but we decided to make two major improvements both of which put us in that “there is no point doing the small stuff until these others are done” limbo. This is a common situation for software development projects to encounter- sometimes its unavoidable, but often you are better off making the effort to figure out how to break things down into smaller chunks.

The first major improvement is our new design. We have had some help designing a logo and a look for our site that better reflects the type of professional design and web-sites that we like to build. The site isn’t perfect today, but from my point of view its a 1000% improvement and enough in the right direction that it was time to get it online and iterate from here.

The second big change is that the site is now running on WordPress. For years I have done blogs using my own hand-rolled software because it forced me to stay familiar with all the technologies involved in delivering one of these sites at a low level. The down side is that when you do it yourself the publishing mechanisms are never as convenient as nice polished software like WordPress so posting becomes a big pain, leading to fewer posts and lots of missed opportunities at interesting things to say. I’m thick into RSS and all of those other technologies in a dozen ways now anyway, so I don’t need the blog to keep me real with that stuff and its nice to have the convenience. Speaking of which, my next project is to move AlexHopmann.com to WordPress too, which will likely be a bigger undertaking given my existing content there.

Welcome to Launch21.com. We are just getting going so this site still needs quite a bit of work, but keeping with our philisophy we are getting what we can online right away. Please get in touch if you want more information on our technology consulting services.