Thoughts on Google App Engine
Chicago's Google App Engine Hack-A-Thon Recap
Today was Chicago's Google App Engine Hack-A-Thon and I managed to get some good work done. Well, I had planned to make packages more ephemeral on the PyPi mirror since it quickly hit the 500MB limit as is. But instead I decided to add Datastore support to the fixture module so that loading sample data is easier when testing an App Engine site. It was very easy to do so I spent most of the time writing some documentation with a complete example for how to go about testing an App Engine site with fixture, WebTest, nose, and NoseGAE.
Here it is: Using Fixture To Test A Google App Engine Site.
Big Thanks to Marzia Niccolai and Mano Marks for trekking out to the Windy City and to all the folks at the Chicago Google office who helped make it happen. As for the hackers, I think we were a shy bunch; there wasn't much show and tell afterwards. I know Ian Bicking got damn close to making a Datastore version of enough builtin file I/O methods for Moin Moin to run on the App Engine. Maybe you'll hear about that soon. Someone did demo a cool iPhone app (literally passed his phone around). It analyzes geo location to provide users with a local message board. I didn't catch the name or a URL — can you drop a comment if you have the info?. UPDATE: The name is Puppyo. Oh yeah, and Harper Reed was hacking on http://excla.im/, an App Engine site that is complimented by a jabber bot allowing you to simply send an IM to post to twitter.
- 0 comments
- read comments
- post comment
- permalink
After the release of Google App Engine, the Rubyists ...
... learn Python? No. They try compiling Ruby to Python bytecode, naturally. One has to admire their pride, I guess. The funny part is this might actually work. As the article points out, Ruby's opcode structure was inspired by Python's (I had no idea it was so similar, but this makes sense). And a while back Fuzzyman performed what looks to me a successful experiment in implementing Ruby-style anonymous blocks in Python bytecode, which is probably the only Ruby logic that Python doesn't do. Although I'm probably forgetting something as I don't use Ruby much.
- 0 comments
- read comments
- post comment
- permalink
Testing Google App Engine sites
The Google App Engine SDK sets up a fairly restrictive Python environment on your local machine to simulate their runtime restrictions. Roughly this consists of no access to sockets and thus barely working httplib, urllib, etc (replaced by urlfetch.fetch()), inability to use Python C modules, and no access to the file system. The SDK lets you run your app very easily using the dev_appserver.py script but I thought, how the heck would I test my app without the dev server?
It turns out this was ridiculously easy. You just run about 10 lines of code at the start of your test suite. Of course, it might change with an SDK upgrade but here's what worked in my tests/__init__.py in today's SDK (besides sys.path munging):
import os
from google.appengine.tools import dev_appserver
from google.appengine.tools.dev_appserver_main import *
option_dict = DEFAULT_ARGS.copy()
option_dict[ARG_CLEAR_DATASTORE] = True
def setup():
## path to app:
root_path = os.path.join(os.path.dirname(__file__), '..')
logging.basicConfig(
level=option_dict[ARG_LOG_LEVEL],
format='%(levelname)-8s %(asctime)s %(filename)s] %(message)s')
config, matcher = dev_appserver.LoadAppConfig(root_path, {})
## commented out stuff that checked for SDK updates
dev_appserver.SetupStubs(config.application, **option_dict)
The setup() is called by nose at the beginning of all tests but if you weren't using nose you could put it at the module level or anywhere else to be called once.
The really cool thing is running tests like this will automatically add indexes for your queries (just like the SDK dev server will) so if you had good code coverage your app would be ready to go live.
Next, you can test your URLs with something like WebTest like so:
from YOURAPP import application
from webtest import TestApp
def test_index():
app = TestApp(application)
response = app.get('/')
...where application is any WSGI app, like one defined in the Hello World tutorial:
from google.appengine.ext import webapp
class MainPage(webapp.RequestHandler):
def get(self):
self.response.out.write('Hello, webapp World!')
application = webapp.WSGIApplication(
[('/', MainPage)], debug=True)
I haven't tried this for Django, but I'm pretty sure it would work as advertised, making your application object this:
application = django.core.handlers.wsgi.WSGIHandler()
And there you have it. The two modules used here aren't in the SDK but that's fine because you don't need to upload them to App Engine anyway. You can run easy_install nose WebTest and be ready to test. You were planning on testing your App Engine site, weren't you? :)
If you want to poke around in the test suite I made for pypi.appspot.com, the code is all in the Pypione project, specifically, the tests directory.
UPDATE
I have since realize the above method does not actually simulate all restrictions, it just inserts some stub modules. Specifically, it doesn't simulate restricting imports of a lot of builtin modules, nor does it remove file() and open().
Jason Pellerin is working on a nose plugin, noseGAE (mentioned below in comments), that aims to get all this simulation accurate. It is coming along nicely.
- 4 comments
- read comments
- post comment
- permalink
PyPi (Cheeseshop) on Google App Engine
Like many of you, I've had my jaw on the floor since the release of Google App Engine. Although there are skeptics out there, a careful read of their terms will show you that it's for real — Google has released GOOGLE to the world and it's not for scary marketing purposes. In fact, I've been growing tired of paranoid Google haters; I'm hoping this will shut them up for a while.
Why is App Engine such a breakthrough? The concept of a hosted web application is nothing new but it has never been done this well. Mundane server maintenance? Gone. Infinite scalability? Check. 100% uptime? Let's face it, if Google went offline you'd probably be down in a nuclear bunker playing Parcheesi.
So ... how should we leverage this tool for the greater good of the community? I can't count the ways without getting dizzy. How about let's start with a mirror of PyPi, the Python Package Index?
PyPi on the App Engine
I barely spent two days on it, but here it is: http://pypi.appspot.com/. Test it out, play with it, try to break it.
As Python grows, especially due to App Engine, PyPi needs to scale too. Zope has put together a PyPi mirror but that's the only other one I know of (actually, I can't even find the link to it right now). Coincidentally, PyPi even went offline for a few min while I was writing this blog post.
Issues...
- urlfetch can't get big packages
- fetching SQLAlchemy triggered the ResponseTooLarge error but its package was barely over 1MB. We might be able to create a download proxy to work around this.
- May not be fully compatible with easy_install yet
- Can't run scripts on App Engine
- Not too big a deal, you have to think of web pages as scripts and use a separate crontab to make timed requests. There are security issues with this, but for mirroring pypi it's not a problem
- Note that I haven't actually mirrored all of PyPi, I'm still testing it out.
- It's not very pretty yet (see below)
- The prototype works, now it needs some tests :)
You Can Help
I'm not dedicated to this project, I just thought it sounded like a good idea and would be a fun way to experiment with the App Engine. If anyone is interested in working on it just let me know --kumar.mcmillan@gmail.com. If there is enough interest I'll put it on Google Code. Possibly the most exciting feature of App Engine is the Datastore API (aka BigTable) and Ben Bangert agrees. It's a little hard for me to wrap my head around it but so far the Expando class—besides being the coolest name for a class—seems to work great for storing package data. If EGG-INO grows a new parameter, it just gets tacked on to the row dynamically.
This has also been a great way to dig up bugs, some of which have already been fixed.
- 6 comments
- read comments
- post comment
- permalink
