farmdev

Thoughts on JavaScript

How To Get Started Writing Open Social Applications

I started learning about Google's Open Social API and gave a lightning talk on it at the last Chicago Python User Group meeting. It wasn't much more than a brief overview and fodder for "diving in" to the Open Social platform. Here it is again, in more of a tutorial form, so dive in!

What Is It?

It's a set of programming interfaces to let you build "applications" that run inside social networking websites. By applications, think "widgets." The interfaces consist of a JavaScript API to build the screens and interactivity and a REST API (Atom publishing API + XML) for the "container" site to implement.

The official "container" sites so far include Friendster, MySpace, Orkut (Google's own), Hi5, Ning, and several others. Some interesting facts from the video : Orkut has an overwhelming presence in Brazil and India and Hi5 is the largest Spanish-speaking social site. Ning is unique in that it's a "meta" social site. It's a social site that enables you to build your own social sites until your head explodes.

The Role of An Open Social Application

I predict that the dark energy of the universe (that is, energy generated by the free time of cunning web developers like you and I) will bring hundreds of cool social apps into existence. It's the same collaboration concept that makes open source software so successful. And more similarly, what makes Firefox so customizable through Extensions. This kind of thing gets me excited about the Web reminiscent of when RSS became a standard. To be honest, I don't think social networking is that interesting and I am still waiting for the fad to die out, but Open Social might encourage other sites to standardize additional container/application APIs.

Let's See Some Code

To test out an Open Social app you can use the Orkut Sandbox. It needs to conform to the Google Gadget API; here is the simplest app you can build:

<?xml version="1.0" encoding="UTF-8" ?>
<Module>
<ModulePrefs title="Boring Hello App">
    <Require feature="opensocial-0.5"/>
</ModulePrefs>
<Content type="html">
<![CDATA[
    <script type="text/javascript">
function start_hello() {
document.getElementById('message').innerHTML = "Hello!";
}
_IG_RegisterOnloadHandler(start_hello);
</script>
<div id="message"> </div>
]]>
</Content>
</Module>

Since this isn't very exciting and doesn't show off and of the API features, there are plenty more examples in the Open Social docs.

Some Gotchas

The Orkut sandbox has a fierce cache. To use your application uncached, you have to add this on to the end of your app's URL:

http://.../?...&bpc=1

Since the app is just HTML, you can use the script tag to pull in 3rd party JavaScript libraries. But keep in mind you will face some problems due to to JavaScript's Same Origin security policy. Although, the JavaScript libs automatically available will probably let you do most of the remote-call things you need, like fetching JSON. There also might be some workarounds with the Dojo framework's iframe strategy.

Have fun!

read article

Datejs - A JavaScript Date Library

Found out about this nifty date package for JavaScript today: Datejs. Its chainable interface makes the code read very naturally:

// What date is next thursday?
Date.today().next().thursday();
// Add 3 days to Today
Date.today().add(3).days();
// Is today Friday?
Date.today().is().friday();

It also does quite a bit of natural English parsing, like Date.parse(‘next thursday’), although most date libraries support that too. I think the chainable interface stands out here as something special. Let's contrast that with some perl code to calculate yesterday, shall we??

my @lt = localtime(time - 86400);
return ($lt[5]+1900) .'-' . ($lt[4]+1) . '-' . ($lt[3]);

(Disclaimer: I didn't actually write that, but it's in a legacy app I sometimes maintain. I'm sure there is a cleaner way in perl to write that code, I just wouldn't know it, nor do I care to!)

read article

Leapfrog Online is looking for some Django developers (Chicago area)

Leapfrog Online is looking to hire several Python developers to work on a Django site. If you know Python but not Django, this is an excellent opportunity to learn. If you know Django but want to learn how to use Python in other contexts, you'll get to do that, too. You'll be working on a high traffic website that hooks into several web services to help customers find Broadband Internet connectivity based on geo location (just US and Canada at the moment). Surrounding that basic function are all kinds of front-end and back-end features, services and systems.

The Software Engineer position is outlined in detail here.

You can send your resume to kumar.mcmillan@gmail.com or send it through the site above. These are full-time positions but if you'd rather work with us as a contractor that may be possible.

What Do We Do?

Leapfrog Online does performance-based customer acquisition, which translates to "we don't make money unless our clients make money." Because of this our software has to work well and we need to collect lots of structured, sensical data so our analysts can build the right marketing strategies. In a more abstract sense, the interesting challenges we face are building high-availability websites, fault-tolerant web services, pushing and pulling at hundreds of gigs of data, and accounting for tight security all along the way. As for the atmosphere, we're still a small company but we're not a struggling startup.

We Care About Open Source

We use open source tools that are right for the job. Currently we use Python or Ruby for websites / web services (Django, Pylons, Paste, Ruby on Rails), Python for backend tools, PostgreSQL for databases, and Trac for our projects. We use rich web interface libraries like Ext JS and we even wrote a distributed content system in Erlang because it was a good fit.

We have contributed patches to most of the projects listed above and maintain our own projects like nose, a few nose plugins, fixture, wsgi_intercept, and sv. We give talks at conferences like Pycon (see #24, #85 and #127). Also, Jeff Cohen (one of our senior developers) runs a popular blog called Softies on Rails and teaches and writes books about Rails.

Scrum: You'll Like It

We started with Extreme Programming a few years ago and have moved towards Scrum and other Agile methods as our approach to software development. We are constantly refining our process, keeping what works, discarding what doesn't. The company is on board with Scrum all the way up to the principles and we are always working to improve how Scrum is integrated holistically (a training program is in the works).

We think you'll like Agile for development. We have several teams of no more than three developers who work in two week "sprints." The sprints are planned out by product owners, developers, and project managers with user stories estimated in "story points" so that the business gets what it needs in order of priority. A sprint is exactly what it sounds like -- you just work! At the end of each sprint the work is released and you attend a retrospective meeting to see what was good, bad, and ugly, and how much work you did. Nothing is perfect so, of course, there are emergencies and derailments here and there but for the most part Scrum keeps things moving at a productive pace. As a developer, I find this discipline empowering and highly motivational.

You Must Test It

We are nutty about automated testing (in case you didn't notice). All code must have automated regression tests so if you're not familiar with this way of writing software, you will learn! We have a fairly involved continuous integration process running in buildbot (though probably moving to Bamboo soon) that performs several builds of each app, one with stable 3rd party libs, the others with trunk versions of 3rd party libs. As well as getting immediate feedback when a bad change is checked in, this also helps us pinpoint bugs in our dependencies before they are released. Our QA department is also different than most in that it consists of developers who are writing functional and/or integration tests in code and adding these to the build process. They are essentially software engineers like the rest of us.

Your Time Is Valuable

No one has a sleeping bad under their desk here; we work until 5 or 6 (weekdays only) to achieve a "sustainable pace." Most of us have been through the "death march" routine at other companies so we know it doesn't work long term. Scrum helps us maintain this ethic.

No Pigeonholes

While we are currently looking for Python/Django programmers, we are always interested in meeting people who think in Ruby/Rails, PHP 5 and other open source web technologies, too. We're especially interested if you're feeling ecumenical and want to learn about and work with, say, both Python and Ruby. You might only work in one language most of the time, but we think it is important for developers to stretch themselves and understand what tools are best for the job.

read article

Software is written by hand

...that's right, it's not molded or prefabbed, it's not made on a production line or in a lab. Are we insane??! Here is a hilarious probe into the darker side of this art we call programming.

read article

Building Flash/ActionScript sites entirely in code and using FireBug for debugging

Way back in the olden days I used to dabble in Flash for building dynamic websites. This was incredibly painful, as I recall, because I couldn't stand the Flash IDE. It was clunky and hard to navigate, there weren't enough key commands, and the code editor was a sad version of notepad at best. And why use Flash anyway? JavaScript will do most of what you need for a dynamic site nowadays and I believe things like Google AdSense read the DOM for content filtering, which would render a Flash site useless. For me, the answer was audio playback. I wanted to play some audio on a site and for this Flash seems like the only option.

Digging deep into my vault of web dabblery I remembered getting so fed up with the Flash IDE that I wrote an entire site in ActionScript alone. I was quite proud of this as it was an early foray into programming and made me realize how much I enjoyed writing software in code. By some stroke of luck, this ActionScript-only site written circa 2002 is still online! OK, the intro was made with some timeline animation (someone else did that) but everything else is pure code, I swear.

(UPDATE: Me an my big mouth: A few months after I wrote this article the site changed, it's not longer the 2002 version :( But it looks and behaves similar so they might have reused some of my code.)

So there I was, planning out my soon-to-blow-your-mind Flash audio player, yet without Creative Suite 3, the latest Flash IDE. In fact, I didn't want it — $699.00, ouch! Not to mention: the pain of installing another massive, bloated application on my hard drive. A little Googling around later, I discovered Motion-Twin ActionScript 2 Compiler, or mtasc for short, written in OCaml. This was exactly what I was hoping for, an open source command line tool for compiling ActionScript code into SWF (Flash) movies without the need for any clunky IDE. And it works very nicely.

Unfortanately, ActionScript seems to have no error handling whatsoever (or I haven't figured out how to handle errors yet) so doing stupid stuff like calling a method that doesn't exist will not stop program execution as it should. Great. So I quickly became familiar with hooks that mtasc provides to the trace() command. Calling trace() writes to the Flash IDE's debug log. But since you aren't in a Flash IDE when testing a freshly compiled swf, mtasc allows you to define your own implementation of trace. Cool! So naturally, I created my own trace that writes to the FireBug log. FireBug, of course, is the ultimate webapp-debugging tool.

Here it is first added to the example app (from the mtasc tutorial):

class Tuto {
static var app : Tuto;
function Tuto() {
// creates a 'tf' TextField size 800x600 at pos 0,0
_root.createTextField("tf",0,0,0,800,600);
// write some text into it
_root.tf.text = "Hello world !";
trace("debugging with trace rocks!");
}
// entry point
static function main(mc) {
// note that this seems to help mtasc find the trace method
var f = new FireTrace();
app = new Tuto();
}
}

...saved to Tuto.as. And here is my trace implementation, saved to FireTrace.as:

class FireTrace {
    static function trace(msg, class_, file, line) {
        getURL('javascript:console.debug("[' + file + ':' + line + ' ' + class_ + '] ' + msg + '")');
    }
}

I compiled it like so:

mtasc -swf Tuto.swf -trace FireTrace.trace -main -header 800:600:20 Tuto.as

And used a simple HTML page to run it:

<html>
<head>
<script type="text/javascript">
if (!window.console || !console.firebug)
{
var names = ["log", "debug", "info", "warn", "error", "assert", "dir", "dirxml",
"group", "groupEnd", "time", "timeEnd", "count", "trace", "profile", "profileEnd"];
window.console = {};
for (var i = 0; i < names.length; ++i)
window.console[names[i]] = function() {}
}
</script>
</head>
<body>
<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" name="TopPlayer" id="TopPlayer"
codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=5,0,0,0"
width="800" height="600">
<param name="movie" value="Tuto.swf">
<param name="quality" value="high">
<param name="loop" value="false">
<param name="play" value="true">
<!--<param name="bgcolor" value="#ffffff">-->
<param name="swliveconnect" value="true">
<embed src="Tuto.swf" quality="high" width="800" height="600"
type="application/x-shockwave-flash" name="TopPlayer" id="TopPlayer"
pluginspage="http://www.macromedia.com/shockwave/download/index.cgi?p1_prod_version=shockwaveflash"
swliveconnect="true">
</embed>
</object>
</body>
</html>

...and here is what it prints to my FireBug console:

[Tuto.as:11 Tuto::Tuto] debugging with trace rocks!

Notice that I had to add this line of code to the Tuto class:

var f = new FireTrace();

I don't know if this is a bug in mtasc but without it the custom trace seems like it never gets loaded. Also notice that I used the FireBug Lite JavaScript snippet for gracefully degrading console methods when FireBug isn't running.

After my romantic reunion with writing ActionScript, what has changed? The first thing I've noticed is that ActionScript 3 now looks exactly Java but I have no idea why. Luckily, it remains compatible with ActionScript 2 so you can leave out all the type declarations and everything still seems to work. Thus, it is pretty much a clone of JavaScript with some extra craziness you probably don't need. I'm still trying to figure out a better way to handle errors; putting debug statements everywhere is pretty lame. If anyone has a suggestion let me know. At the least, I would like to get an error in the log when I stupidly try to call a non-existant method.

Now that I have a prototype, the next step is to write automated tests, of course! This will be a lot of fun as I love watching tests pass. At first glance, the AsUnit (ActionScript Unit Test Runner) seems like it will be useful. I have no doubt that GUI testing Flash is no easier than it is in other languages, and thus not so easy to automate ... but one always needs unit tests.

UPDATE

Ah, looks like there is an even better way to call JavaScript by way of the ExternalInterface class:

ExternalInterface.call("console.log", variable1, variable2, variableN);

I haven't tried it but this article suggests to limit your calls to String, Number, Array, Object, and Boolean datatypes.

read article

An In-Process, Headless Web Browser for Python?

Yesterday, Atul Varma announced he had resurrected python-spidermonkey, John J. Lee's project to run JavaScript in Python. Woo! Spider Monkey is the JavaScript runtime that Mozilla (i.e. Firefox) uses internally. However, it's just an engine and doesn't deal with the DOM or anything about the typical web browser environment JavaScript files usually run in. Roughly, that environment defines the following JavaScript objects:

I've always been interested in python-spidermonkey for the prospect of fully testing JavaScript-dependent web applications in Python alone. This is no small feat and there are many caveats. The first caveat is that if the Application Under Test is running in something too far off from its native runtime then the value of your tests rapidly decreases. In other words, the best way to test a JavaScript-heavy web app is to use Selenium or Windmill to automate and introspect the app in its native runtime: the web browser. I believe Java has Rhino but since it's implemented in Java my guess is its behavior is pretty far off from that of Firefox. We all know and love the quirks of web browsers; this is what makes web development so fun :(

A lightweight version of Firefox — its JavaScript engine, its DOM, etc — could possibly make web testing a lot more efficient. By lightweight I mean no GUI, no visual rendering (although removing this may greatly hinder JavaScript, not sure). Why? At the company I'm at we build and test web applications. Our largest suite of Selenium tests that automate a real web browser takes about 2 hours to run and that's expensive. We are looking into a Selenium Grid solution but if there's a way to simply cut out all the browser overhead that we don't need, this would be a huge score.

And, oh yeah, I haven't even talked about how tests for web apps need to run in Internet Explorer if they are to be worth anything. However, we make some apps for internal company use and thus can mandate the use of Firefox. To solve the cross-browser problem one approach could be to implement all this logic as a driver that can be hooked into the Selenium interface we use, which is all in Python code thanks to Selenium Remote Control. This the exact approach that Webdriver for Java takes &mdash you can simply tell it to use a faster in-process web browser and it will run JavaScript but only via Rhino, which goes back to the non-native runtime problem. In fact, WebDriver will soon be an actual driver for Selenium.

Hmm ... thinking out loud ... maybe the solution is to just run XULRunner as a headless firefox, the way SMILE web scraper does. Maybe I'll try that.

Besides web testing, one cool thing about python-spidermonkey is it may allow for unit testing of JavaScript libs in Python. This means you could create an import hook that loaded *.js files into Python and do all your testing in Python, reaping the benefits of tools like nose. Or, maybe more realistically, automate the testing of custom Firefox extensions implemented in JavaScript.

Have you ever ran unit tests in JavaScript? It's clunky because you need to run them in a browser. That's how JSUnit works anyway, which is otherwise a passable testing environment. To automate such tests you have to use something like Selenium to run your test suite in a real web browser.

read article

aintjustsoul.net: A portable record player for the Internet

I'm very pleased to announce a website I just launched:

http://aintjustsoul.net/

I'm calling it "a portable record player for the Internet" because I'm a record collector myself and listening to as much music as possible has been the best way to expand my collection. Yep, I'm one of those guys who goes to record conventions and brings a portable record player.

There are a few places to find used vinyl online — since 161,783 records are for sale on eBay right now, I'd say the Internet is a good venue. The problem is that you can't always listen to a record before you buy it and it's impossible to find great music by name alone when there is so much to choose from. I'm hoping to change all that with this site.

As you can see, it's still in very experimental stages and leaves a lot to be desired. eBay is the only marketplace supported so far but I plan to integrate Music Stack next. If anyone knows of another marketplace, please let me know. Also, if you have a friend who buys records then please send on the link (I'm interested to hear feedback from record buyers).

It took me about 6 months to get the data in order (doing all this in my spare time) so it is very exciting to be able to release an actual site now. The data pipeline is home-grown ETL and uses the eBay API + the super speedy and tolerant lxml.html (contrib by Ian Bicking), the excellent SQLAlchemy, and other tools. As I take the data pipeline to the next level I really hope to start using the very cool SnapLogic ETL framework for Python.

On the UI side this has been my first chance to really dive into Dojo. I like Dojo because it takes semantic HTML seriously, namespacing seriously, unit testing seriously, and object oriented JS seriously. It's a bit monolithic and I still haven't figured out how to build a custom minified version but so far I like it. Also, I really wish the Dojo API reference was more comprehensive. On the backend UI side, I'm using Pylons since I like how well it integrates with 3rd party Python libs. I don't like some of the magic it inherited from Rails but that is pretty easy to work around. Oh, I almost forgot to mention SoundManager 2, the ultimate javascript/flash mp3 player.

read article

It's Time to USE The Web : Mozilla Labs Releases Ubiquity

Mozilla Labs releases the Ubiquity add-on for Firefox. In a nutshell: With search, users type what they want to find. With Ubiquity, they type what they want to do.

Big congrats to Aza and everyone else who worked on this! As innovative as Humanized was, building something similar into Firefox is genius. Personally I am excited by the prospect of developing work flows on the Ubiquity platform. There are so many common work flows that need simplification.

Developers: If you want to cut to the chase, Ajaxian shows how easy it is to create custom commands. And of course, the details are in the Authoring Tutorial.

read article

Web Frameworks Do Not Make DBAs Happy

A colleague of mine, Shaun Thomas, is one of a few database administrators who manage all our company's databases by monitoring, optimizing, partitioning, building star schemas, etc. The DBAs also maintain standard operating procedures for how to name a column or how to refer to an external identifier. Most importantly, they conduct reviews of your horrid schema changes before you break stuff.

Most web frameworks (Django, Rails, etc) out there abstract away a lot of low level database details since they focus on making life easier for web developers. This is great but it's important to have a way to easily tweak the low level stuff when you need to. In fact, most frameworks kinda leave DBAs in the dust. It looks like Shaun reached his breaking point on this a few weeks ago and the result was a hilarious rant. He does have a good point. The only database abstraction layer I've used that truly keeps the DBA in mind is SQLAlchemy. It adds more complexity to the tool but not in a way that makes your life difficult.

read article

The Future of Testing (GTAC 2008)

Google Test Automation Conference (GTAC) is my all-time favorite conference. It's free. It's on a single track — this means you don't miss any talks and everyone experiences the same journey of thought. Also, since you have to apply for admittance with a short essay, everyone who attends is really passionate about testing. It's still sort of "underground" which keeps it small and very social. Last year, I made some kind of attempt to live blog summaries of the GTAC talks but I never made it past part 1. We'll see how far I get this year, stay tuned. The videos for 2008 aren't online yet but check youtube often because last year they were up in less than a day. The Future of Testing was the first talk of the GTAC 2008 conference on Thursday Oct 23rd given by James A. Whittaker, a very entertaining speaker who works for Microsoft. His talk was excellent and I highly recommend keeping a lookout for the video. Here are my notes...

read article

Taming The Beast: How To Test an AJAX Application (GTAC 2008)

This was one of the talks at GTAC 2008 that I was most looking forward to before the conference. It was excellent, I was not let down. The talk was given by Markus Clermont and John Thomas who work at Google. Since the talk was right after lunch they decided to take a Q & A approach. It sort of went off in tangents at points but overall the format seemed to work. In my own work I've been struggling at maintaining a now bloated test suite for an AJAX website but their approach made something click in my head. I'm already working on a refactoring plan. Here is my abbreviated interpretation of the talk ...

read article

Are you hiring web developers?

As the US economy spins into a death spiral I unfortunately now know some highly skilled web developers (Python, Ruby, JavaScript, etc) in the Chicago area who are looking for work. And I mean highly skilled people whose ninja like qualities will be sorely missed. Most hiring budgets are probably frozen — at least until next year — but they will soon thaw. Those that thaw sooner will make space for big prizes of talent. I can make introductions over email or via Linked In.

read article

Chicago JavaScript Meetup: JS.Chi()

I went to my first Chicago JavaScript meetup on Thursday (the 4th meetup they've had so far) and it looks like a promising group. The topics were pretty basic but the speakers seemed very deep into JavaScript so I'm looking forward to future meetings. By raise of hands we decided to split into two groups, an advanced and a beginner, which was an excellent idea. The turnout was incredible — about 60 (or more?) people showed up. The original venue that got canceled only held 40 so some of those people were actually on a waiting list ...

read article

Fudge: Another Python Mock Framework

I'm excited to announce the release of Fudge, a Python module for replacing real objects with fakes (mocks, stubs, etc) while testing. Fudge started when a co-worker introduced me to Mocha, a mocking framework for Ruby and a simpler version of jMock (Java). Up to that point I had been building mocks "by hand" in Python with a post mortem approach; I'd set up some fakes then inspect their call stacks at the end of the test. I like the jMock approach better—you declare a fake object that expects methods and arguments, you replace the real object with the fake one, run your code, and you're done. If your code doesn't live up to the expectations then your test fails.

read article

Googlebot's Fatal Flaw And How You Can Fix It (or Get Rich Trying)

I came across this article today on Coding Horror about how Google has a monopoly on search engines and how something must be done about it. I'm not one who falls into the "Google Is Evil" camp; I actually think they are a benevolent force in the world :) However, as with any monopoly, the lack of competition stifles progress. And when I think about the state of today's technology, I can't help but wonder why Google has not fixed the most fatal flaw in their Googlebot : It does not behave like a web browser. Search engines are made for people and the majority of people browse the Internet with a web browser. The first comment on the article is a cry for help: "What can we do?" I have an answer to that question. And you can take my answer and turn it into a business plan and climb the golden staircase to success. Any smart investor would be begging you to take their money. Google generated $5.37 billion dollars in Q2 of 2008 and their flagship product doesn't even work! In fact, I'm going to give this to you all for free; all I ask is that you visit me one day and say thanks. Are you ready?

read article

A new version of Fudge, mock object library for Python

read article

Degradable Ajax by Sharing Mako Templates With Dojo

read article

Unit Testing JavaScript With JsTestDriver

read article

I've Joined the Web Dev Group at Mozilla

read article

What The New Open Web Can Do: Firefox 4 Demos

read article

How To Achieve True Privacy On The Web: DOMCrypt

read article

Firefox 4 Is Here And It's Glowing

read article

What's Happening at Mozilla?

read article

Profiling And Optimizing React Apps

read article

Safer TypeScript With Generics And Linting

read article