farmdev

Thoughts on Google App Engine

The Promise of the Cloud

As web developers we are faced with this problem: how do we scale up our code to handle high traffic? A lot of time and engineering goes into this problem -- time to simulate the traffic we expect and add servers to our cluster, cache heavy database access, etc, in anticipation of the load. Time is precious. This time could be spent optimizing the usefulness of our web product and creating interesting content. No one really congratulates you when a website works, they expect it to work.

When Google App Engine was released their pitch was...

read article

Python Package Index (PyPI) Mirrors

Most deployment systems depend on PyPI, the Python Package Index, for fetching and installing dependencies. Although performance and reliability has greatly improved, there are still days when you may find PyPI down and thus are unable to deploy through the normal fashion.

Finally, there is now a concerted effort to create official mirrors of PyPI that everyone can use (see PEP 381). The mirrors are pretty simple--they're not full blown package indexes--and you can already start using them for experimentation. Also, thanks to Richard Jones, Martin Löwis, Guido, and others at EuroPython, my half-baked idea of using Google App Engine as a PyPI mirror has been resurrected! I don't know if it's fully implemented yet but you can try it out at pypi.appspot.com. App Engine seems like a logical place for a mirror due to its scalability. However, I am skeptical of how well it will perform. App Engine still has a long way to go with regards to stability.

To get involved with the mirroring project you can follow the discussions on the Python Catalog SIG mailing list. The next step will be modifying clients to discover and fallback on available mirrors.

read article

T'is be'a Fixture 1.1.1 fer ya!

Y'aharrr me seabound mateys! Thar be'a fine gully of'a wind shakin' ye jigger today as m'announce a new release of Fixture, a python module fer loadin' and referencing test data. O'er yonder ye find a cap'n's Changelog fer ye royal subjects.

Riches abound! Booty abaft! Me could'a n'er dunnit not be'a the help o'a few fine pirates amidst ye Python vagabonds. Me best salute go t'a Tomas Holas, Alex Marandon, and bslesinsky fer'a ya bug reports and patches. Been'a some quiet waters thus far but much treasure huntin' lies o'er th'horizon.

Plunder Fixture 1.1.1!

(And it be'a fine day fer pirates, aye)

read article

Chicago's Google App Engine Hack-A-Thon Recap

Today was Chicago's Google App Engine Hack-A-Thon and I managed to get some good work done. Well, I had planned to make packages more ephemeral on the PyPi mirror since it quickly hit the 500MB limit as is. But instead I decided to add Datastore support to the fixture module so that loading sample data is easier when testing an App Engine site. It was very easy to do so I spent most of the time writing some documentation with a complete example for how to go about testing an App Engine site with fixture, WebTest, nose, and NoseGAE.

Here it is: Using Fixture To Test A Google App Engine Site.

Big Thanks to Marzia Niccolai and Mano Marks for trekking out to the Windy City and to all the folks at the Chicago Google office who helped make it happen. As for the hackers, I think we were a shy bunch; there wasn't much show and tell afterwards. I know Ian Bicking got damn close to making a Datastore version of enough builtin file I/O methods for Moin Moin to run on the App Engine. Maybe you'll hear about that soon. Someone did demo a cool iPhone app (literally passed his phone around). It analyzes geo location to provide users with a local message board. I didn't catch the name or a URL — can you drop a comment if you have the info?. UPDATE: The name is Puppyo. Oh yeah, and Harper Reed was hacking on http://excla.im/, an App Engine site that is complimented by a jabber bot allowing you to simply send an IM to post to twitter.

read article

After the release of Google App Engine, the Rubyists ...

... learn Python? No. They try compiling Ruby to Python bytecode, naturally. One has to admire their pride, I guess. The funny part is this might actually work. As the article points out, Ruby's opcode structure was inspired by Python's (I had no idea it was so similar, but this makes sense). And a while back Fuzzyman performed what looks to me a successful experiment in implementing Ruby-style anonymous blocks in Python bytecode, which is probably the only Ruby logic that Python doesn't do. Although I'm probably forgetting something as I don't use Ruby much.

read article

Testing Google App Engine sites

The Google App Engine SDK sets up a fairly restrictive Python environment on your local machine to simulate their runtime restrictions. Roughly this consists of no access to sockets and thus barely working httplib, urllib, etc (replaced by urlfetch.fetch()), inability to use Python C modules, and no access to the file system. The SDK lets you run your app very easily using the dev_appserver.py script but I thought, how the heck would I test my app without the dev server?

It turns out this was ridiculously easy. You just run about 10 lines of code at the start of your test suite. Of course, it might change with an SDK upgrade but here's what worked in my tests/__init__.py in today's SDK (besides sys.path munging):

import os
from google.appengine.tools import dev_appserver
from google.appengine.tools.dev_appserver_main import *
option_dict = DEFAULT_ARGS.copy()
option_dict[ARG_CLEAR_DATASTORE] = True

def setup():
    ## path to app:
    root_path = os.path.join(os.path.dirname(__file__), '..')
    logging.basicConfig(
        level=option_dict[ARG_LOG_LEVEL],
        format='%(levelname)-8s %(asctime)s %(filename)s] %(message)s')
    config, matcher = dev_appserver.LoadAppConfig(root_path, {})
    ## commented out stuff that checked for SDK updates
    dev_appserver.SetupStubs(config.application, **option_dict)

The setup() is called by nose at the beginning of all tests but if you weren't using nose you could put it at the module level or anywhere else to be called once.

The really cool thing is running tests like this will automatically add indexes for your queries (just like the SDK dev server will) so if you had good code coverage your app would be ready to go live.

Next, you can test your URLs with something like WebTest like so:

from YOURAPP import application
from webtest import TestApp
def test_index():
    app = TestApp(application)
    response = app.get('/')

...where application is any WSGI app, like one defined in the Hello World tutorial:

from google.appengine.ext import webapp
class MainPage(webapp.RequestHandler):
  def get(self):
    self.response.out.write('Hello, webapp World!')

application = webapp.WSGIApplication(
                [('/', MainPage)], debug=True)

I haven't tried this for Django, but I'm pretty sure it would work as advertised, making your application object this:

application = django.core.handlers.wsgi.WSGIHandler()

And there you have it. The two modules used here aren't in the SDK but that's fine because you don't need to upload them to App Engine anyway. You can run easy_install nose WebTest and be ready to test. You were planning on testing your App Engine site, weren't you? :)

If you want to poke around in the test suite I made for pypi.appspot.com, the code is all in the Pypione project, specifically, the tests directory.

UPDATE

I have since realize the above method does not actually simulate all restrictions, it just inserts some stub modules. Specifically, it doesn't simulate restricting imports of a lot of builtin modules, nor does it remove file() and open().

Jason Pellerin is working on a nose plugin, noseGAE (mentioned below in comments), that aims to get all this simulation accurate. It is coming along nicely.

read article

PyPi (Cheeseshop) on Google App Engine

Like many of you, I've had my jaw on the floor since the release of Google App Engine. Although there are skeptics out there, a careful read of their terms will show you that it's for real — Google has released GOOGLE to the world and it's not for scary marketing purposes. In fact, I've been growing tired of paranoid Google haters; I'm hoping this will shut them up for a while.

Why is App Engine such a breakthrough? The concept of a hosted web application is nothing new but it has never been done this well. Mundane server maintenance? Gone. Infinite scalability? Check. 100% uptime? Let's face it, if Google went offline you'd probably be down in a nuclear bunker playing Parcheesi.

So ... how should we leverage this tool for the greater good of the community? I can't count the ways without getting dizzy. How about let's start with a mirror of PyPi, the Python Package Index?

PyPi on the App Engine

I barely spent two days on it, but here it is: http://pypi.appspot.com/. Test it out, play with it, try to break it.

As Python grows, especially due to App Engine, PyPi needs to scale too. Zope has put together a PyPi mirror but that's the only other one I know of (actually, I can't even find the link to it right now). Coincidentally, PyPi even went offline for a few min while I was writing this blog post.

Issues...

  • urlfetch can't get big packages
    • fetching SQLAlchemy triggered the ResponseTooLarge error but its package was barely over 1MB. We might be able to create a download proxy to work around this.
  • May not be fully compatible with easy_install yet
  • Can't run scripts on App Engine
    • Not too big a deal, you have to think of web pages as scripts and use a separate crontab to make timed requests. There are security issues with this, but for mirroring pypi it's not a problem
    • Note that I haven't actually mirrored all of PyPi, I'm still testing it out.
  • It's not very pretty yet (see below)
  • The prototype works, now it needs some tests :)

You Can Help

I'm not dedicated to this project, I just thought it sounded like a good idea and would be a fun way to experiment with the App Engine. If anyone is interested in working on it just let me know --kumar.mcmillan@gmail.com. If there is enough interest I'll put it on Google Code. Possibly the most exciting feature of App Engine is the Datastore API (aka BigTable) and Ben Bangert agrees. It's a little hard for me to wrap my head around it but so far the Expando class—besides being the coolest name for a class—seems to work great for storing package data. If EGG-INO grows a new parameter, it just gets tacked on to the row dynamically.

This has also been a great way to dig up bugs, some of which have already been fixed.

read article