Farm Development

An In-Process, Headless Web Browser for Python?

Yesterday, Atul Varma announced he had resurrected python-spidermonkey, John J. Lee's project to run JavaScript in Python. Woo! Spider Monkey is the JavaScript runtime that Mozilla (i.e. Firefox) uses internally. However, it's just an engine and doesn't deal with the DOM or anything about the typical web browser environment JavaScript files usually run in. Roughly, that environment defines the following JavaScript objects:

I've always been interested in python-spidermonkey for the prospect of fully testing JavaScript-dependent web applications in Python alone. This is no small feat and there are many caveats. The first caveat is that if the Application Under Test is running in something too far off from its native runtime then the value of your tests rapidly decreases. In other words, the best way to test a JavaScript-heavy web app is to use Selenium or Windmill to automate and introspect the app in its native runtime: the web browser. I believe Java has Rhino but since it's implemented in Java my guess is its behavior is pretty far off from that of Firefox. We all know and love the quirks of web browsers; this is what makes web development so fun :(

A lightweight version of Firefox — its JavaScript engine, its DOM, etc — could possibly make web testing a lot more efficient. By lightweight I mean no GUI, no visual rendering (although removing this may greatly hinder JavaScript, not sure). Why? At the company I'm at we build and test web applications. Our largest suite of Selenium tests that automate a real web browser takes about 2 hours to run and that's expensive. We are looking into a Selenium Grid solution but if there's a way to simply cut out all the browser overhead that we don't need, this would be a huge score.

And, oh yeah, I haven't even talked about how tests for web apps need to run in Internet Explorer if they are to be worth anything. However, we make some apps for internal company use and thus can mandate the use of Firefox. To solve the cross-browser problem one approach could be to implement all this logic as a driver that can be hooked into the Selenium interface we use, which is all in Python code thanks to Selenium Remote Control. This the exact approach that Webdriver for Java takes &mdash you can simply tell it to use a faster in-process web browser and it will run JavaScript but only via Rhino, which goes back to the non-native runtime problem. In fact, WebDriver will soon be an actual driver for Selenium.

Hmm ... thinking out loud ... maybe the solution is to just run XULRunner as a headless firefox, the way SMILE web scraper does. Maybe I'll try that.

Besides web testing, one cool thing about python-spidermonkey is it may allow for unit testing of JavaScript libs in Python. This means you could create an import hook that loaded *.js files into Python and do all your testing in Python, reaping the benefits of tools like nose. Or, maybe more realistically, automate the testing of custom Firefox extensions implemented in JavaScript.

Have you ever ran unit tests in JavaScript? It's clunky because you need to run them in a browser. That's how JSUnit works anyway, which is otherwise a passable testing environment. To automate such tests you have to use something like Selenium to run your test suite in a real web browser.

  • Re: An In-Process, Headless Web Browser for Python?

    This will be *great* for writing email harvesting web spiders that understand Javascript! :-)

  • Re: An In-Process, Headless Web Browser for Python?

    Heh, maybe that's what SMILE was built for

  • Re: An In-Process, Headless Web Browser for Python?

    In Rhino 1.7R2 Norris Boyd implemented doctest for javascript. That doesn't help with the dom stuff, but is pretty cool nevertheless.

    For the dom stuff, we have a 'Fake Mozilla' that we've implemented in Java. It still needs a lot of work, so it would be nice to open that up as an open source project.

    We're writing a FireFox add-on, so we use MozRepl (similar to JSSh) a lot and we've been thinking about using that with a headless x11 session to drive tests. These work ok.. but we're still looking for something better. :-)

  • Re: An In-Process, Headless Web Browser for Python?

    This reminds me an instance when we had internal application that was endlessly scrolling the ListBox (Windows).

    The redrawing was taking significant amount of time, so we disabled it during certain operations:

    SendMessage(hwnd, WM_SETREDRAW, FALSE, 0);

    then after we were done, we enabled it back:

    SendMessage(hwnd, WM_SETREDRAW, TRUE, 0);

    InvalidateRect(hwnd, NULL, TRUE);

    and the speed-up was substantial.

    Another option would be to hide the window, low-level Windows GDI will speed-up significantly when calling drawing primitives in the window that is not showing.

    Hope this helps.

  • Re: An In-Process, Headless Web Browser for Python?

    My refusal to use Windows is what makes me learn to do everything using only Linux, even though it usually requires long times in front of the computer researching and tweaking the hints google.com/linux gives me. But, at least I can say that at the end the results are satisfying, I end up learning more about Linux (recommend start from the ebook http://www.ebook-search-queen.com ) and how powerful a command line application can be compared to a most GUI application. I must admit that I do prefer GUI in the majority of the cases, is just that from time to time I am amazed by the power of the CLI.

  • Re: An In-Process, Headless Web Browser for Python?

    I don't read about python before.I didn't know that Python is a programming language,Thanks for knowing me.personal injury fort lauderdale

  • Re: An In-Process, Headless Web Browser for Python?

    Impressive blog ,I never read this types of blog before. thanks to providing this type of interesting content. I’ll be waiting for your next blog.

  • Re: An In-Process, Headless Web Browser for Python?

    Hello My Dear Friend , The post is written in very a good manner and it entails many useful information for me. I appreciated what you have done here. I am always searching for informative information like this. Thanks for sharing with us.

  • Re: An In-Process, Headless Web Browser for Python?

    Hello My Dear Friend , The post is written in very a good manner and it entails many useful information for me. I appreciated what you have done here. I am always searching for informative information like this. Thanks for sharing with us.

  • Re: An In-Process, Headless Web Browser for Python?

    This is a great inspiring article. I am pretty much pleased with your good work. You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post.

  • Re: An In-Process, Headless Web Browser for Python?

    Useful information shared..Iam very happy to read this article. Thanks for giving us nice info. Fantastic walk-through. I appreciate this post.

  • Re: An In-Process, Headless Web Browser for Python?

    Useful information shared..Iam very happy to read this article. Thanks for giving us nice info. Fantastic walk-through. I appreciate this post.

  • Re: An In-Process, Headless Web Browser for Python?

    I was looking for something like this…I found it I am bookmarking this and sharing with my friends on facebook, twitter and my space.

  • Re: An In-Process, Headless Web Browser for Python?

    nice

  • Re: An In-Process, Headless Web Browser for Python?

    nice http://www.g.com

  • Re: An In-Process, Headless Web Browser for Python?

    nice r

  • Re: An In-Process, Headless Web Browser for Python?

    nice <

Note: HTML tags will be stripped. Hit enter twice for a new paragraph.

Recent Projects

  • JSTestNet

    Like botnet but for JS tests in CI.

  • Nose Nicedots

    Nose plugin that prints nicer dots.

  • Fudge

    Mock objects for testing.

  • Fixture

    Loading and referencing test data.

  • NoseJS

    Nose plugin that runs JavaScript tests for a Python project.

  • Wikir

    converts reST to various Wiki formats.