Goodbye Google App Engine (GAE)

This is a post rewrite. The original post can found at the bottom of this post.

The reason why I am rewriting this post is because Patrick Chanezon (from Google), has added a kind and respectful comment to this post. Given the huge amount of traffic this post has generated (never expected nor wanted) I don't want to damage the GAE project which can be a great platform in the future. People haven't understand  my point apparently because of the dramatic writing style of the original post.

The reason why we left the GAE platform is solely because of its unreliability as of september-octuber-november 2010. The last 3 months we saw the site down more times that we considered reasonable and we couldn't find a way to avoid it so eventually we decided to abandon the platform. When the GAE team be able to stabilize the platform, it will perhaps be a great platform for scalability.

I emphasized the GAE limitations because some of them were hard to work around when we were developing although we did a great job making everything work, but when the platform let me down because of its many 500 errors,I got the feeling that we had been loosing our time with a platform not enough mature.

As a young and small entrepreneur (I am the one who fund our projects training developers all around the country), the development effort was big and I calculated the amount of money and wrote it in the post, but any experienced developer knows that the quantity I wrote is not that much. Some of my readers are friends who have invested money in the project and I wanted to give them an idea of the effort we made in terms of money in the GAE platform.

When we started developing on GAE one year ago, there was less documentation that there are today. But even today, I haven't found any document saying... "if the application can't load into memory within 1 second, it might not load and return 500 error codes". This is something that should be written in the documentation as soon as possible to help others understand if GAE is a good fit for their apps or not. Django can load into memory in less than a second in my local machine I guess but apparently not in GAE and we build our app in Django because Webapp framework miss many things like security. With Django we don't have to care about XSS and i18n is easy while in Webapp it not that easy. So if GAE can't run with Django we had to migrate. But we were using Django Helper, developed but Google engineers if I recall correctly so it is quite disappointing to read from another Google engineer that now we have to load under a second. Sounds contradictory. Sounds not professional.

Regarding the GAE for Business, I believe this is really new. I remember joining the business plan/account when it was kind of beta and I thought it was still in beta because I never got news about it. So yes, we wanted to pay for better support and SLA and at that point we couldn't. As a developer I can't be checking this things every day because I've got many things to do so I guess Google have to inform me if the business account is now working or what.
The support from GAE engineers have been really bad and this is a mistake they know but I hope in the future it gets better for those who are investing in the platform.

In our current hosting we still don't do JOINs because we don't need them and we've learned a lot of things about scalability thanks to GAE restrictions. Data DEnormalization is one of them and we keep it like that. I wrote about the limitations because wanted to warn users who haven't  read about No-SQL. When I said "normalization" in the original post I meant that before saving strings, we transform them in a way that we can search for them (uppercase and things like that). When the application was running, it wasn't slow. We managed to optimize our algorithms and developed a caching architecture which works really well and have a nice API.

Although many people thought we have lost money for not reading docs, the reality is that we left GAE for its unreliability and nothing more.

We enjoyed certain things from the GAE platform like the feature of changing versions with a single click and the ability to add new fields to Kinds dynamically.

It is nice if GAE success and I am happy to see users who are enjoying the platform but for us, I have had enough for now. I also don't want to try Azure. The more experience I gain, the less I trust platforms/frameworks which code I can't see.

Thanks for reading and understanding my English which I think should be understandable.


--------- Original post written on 21 nov 2010:

Choosing GAE as the platform four our project is a mistake which cost I estimate in about 15000€. Considering it's been my money, it is a "bit" painful.

GAE is not exactly comparable to Amazon, Rackspace or any of this hosting services, because it is a PaaS (Platform as a Service) rather than just a cluster of machines. This means that you just use the platform and gain scalability, high availability and all those things we want for our websites, without any other software architecture. Cool, isn't it.
You do not pay until you get a lot of traffic to your site so it sounds very appealing for a small startup. I read about this in a book called "The web success startup guide" by Bob Walsh.
So yes, everything looked cool and we decided to go with this platform a couple of months ago.
It supports Python and Django (without the ORM) which we love so we tried it out. We made a spike, kind of 'hello world' and it was easy and nice to deploy.

When we started developing features we started realizing the hard limitations imposed but the platform:

1. It requires Python 2.5, which is really old. Using Ubuntu that means that you need a virtualenv or chroot with a separate environment in order to work with the SDK properly: Ok, just a small frustration.

2. You can't use HTTPS with your own domain (naked domain as they called) so secure connections should go though This just sucks.

3. No request can take more than 30 secons to run, otherwise it is stopped: Oh my god, this has been a pain in the ass all the time. When we were uploading data to the database (called datastore a no-sql engine) the uplaod was broken after 30 seconds so we have to split the files and do all kind of difficult stuff to manage the situation. Running background tasks (cron) have to be very very well engineered too, because the same rule applies. There are many many tasks that need to take more than 30 secons in website administration operations. Can you imagine?

4. Every GET or POST from the server to other site, is aborted if it has not finished within 5 seconds. You can configure it to wait till 10 seconds max. This makes impossible to work with Twitter and Facebook many times so you need intermediate servers. Again, this duplicates the time you need to accomplish what seemed to be a simple task

5. You can't use python libraries that are build on C, just libraries written in python: Forget about those great libraries you wanted to use.

6. There are no "LIKE" operator for queries. There is a hack that allows you to make a kind of "starts with" operator. So forget about text search in database. We had to work around this normalizing and filtering data in code which took as 4 times the estimated time for many features.

7. You can't join two tables. Forget about "SELECT table1, table2..." you just can't. Now try to think about the complexity this introduces in your source code in order to make queries.

8. Database is really slow. You have to read about how to separate tables using inheritance so that you can search in a table, get the key and then obtain its parent in order to avoid deserialization performance and all kind and weird things.

9. Database behavior is not the same in the local development server than in google servers. So you need some manual or selenium tests to run them against the cloud, because your unit and integration tests won't fail locally. Again this means more money.

10. If you need to query on a table asking for several fields, you need to create indexes. If those fields are big, it is easy to get the "Too many indexes" runtime exception. That will not happend to you in a "hello world" application, but enterprise applications that need to query using several parameters are easy to run into this trouble. If you use a StringListProperty, you might not find this problem until one user populates this field with a long list. Really nasty thing. We had to re-engineer our search engine once in production becase we didn't know this till it happened.

11. No query can retrieve more than 1000 records. If there are more than 1000, you have to deal with offsets in your code to paginate results. Taking into account that you have to filter results in code sometimes because of the the limitations of GQL (the query language), this means again more complexity for your code. More and more problems, yeiiii!

11. You can't access the file system. Forget about saving uploads to filesystem and thouse tipical things. You have to read and learn about the inconvenient blobs. Again, more development time wasted.

12. Datastore and memcache can fail sometimes. You have to defend your app against database failues. What? Yep, make sure you know how to save user data when this happens, at any time in the request process, but remember... you can't use the filesystem. Isn't it fun?

13. Memcache values max size is 1megabyte. Wanted to cache everything? You were just dreaming mate, just 1 megabyte. You need your own software architecture to deal with caching.

There are some more things that make the development hard with this platform but I am starting forgeting about them. Which is great.

So why did we stay with this platform? Because we were finding the problems as we go, we didn't know many of them. And still, once you overcome all the limitations with your complex code, you are suppoused to gain scalabilty for millions of users. After all, you are hosted by Google. This is the last big lie.

Since the last update they did in september 2010, we starting facing random 500 error codes that some days got the site down 60% of the time. 6 times out of 10, users visiting the site couln't register or use the site. Many people complained about the same in the forums while google engineers gave us no answer. After days they used to send and email saying that they were aware of the problem and were working on it, but no solutions where given. In november, an engineer answered in a forum that our website had to load into memory in less than 1 second, as pretty much every request was loading a completelly new instance of the site. 1 second? Totally crazy. We have 15000 lines of code and we have to load django. How can we make it under 1 second?
They encourage people to use the light webapp framework. Webapp is great for a "hello world" but it is bullshit for real applications.
So once we suffered this poor support from google and we understood that they were recommending GAE just for "hello world" applications, we decided to give up on it finally.
This is not serious nor professional from google. We are more than dissapointed with GAE.

For us, GAE has been a failure like Wave or Buzz were but this time, we have paid it with our money. I've been too stubborn just because this great company was behind the platform but I've learned an important lesson: good companies make mistakes too. I didn't do enough spikes before developing actual features. I should have performed more proofs of concept before investing so much money. I was blind.

After this we wanted to control the systems so we have gone to a classical hosting based on PostgreSQL, Nginx, Apache2 and all that stuff. We relay on great system administrators who give us great support and make us feel confident. We are really happy to be able to make SQL queries and dump databases and all those great tools that we were missing in the dammed GAE.

Because we are software craftmen, we designed all our code using TDD and the rest of XP practices. Thanks to this, we migrated 15000 lines of code in just 1 week (and just a pair of developers) from one platform to another which has been a deep change. From No-SQL back to SQL. From one database access API to another. If we didn't had these great test bateries, migration would have taken months.
Migrating data took us 1 week because of encoding problems and data transformations.

You might agree or not with me on this post but there is one certain reality: developing on GAE introduced such a design complexity that working around it pushes us 5 months behind schedule. Now that we had developed tools and workarounds for all the problems we found in GAE, we were starting being fast with the development of features eventually, and at that point we found the cloud was totally unstable, doggy.

  • Pingback: Google Onun Mojo geri alabilirim?()

  • Pingback: Can Google Get Its Mojo Back? Balakrishnan V K - Balakrishnan V K()

  • Pingback: The many (More) failures of Google… | Smarter Geek()

  • Pingback: Can Google Get Its Mojo Back? | Phonedaily()

  • Pingback: Can Google Get Its Mojo Back? | ????????????????()

  • Pingback: Can Google Get Its Mojo Back? - Niche Choppers VIP Earners Club()

  • Pingback: Can Google Get Its Mojo Back? « OnLion News()

  • Pingback: Google : vers le déclin de l’Empire américain ?()

  • Weespr

    Our experience of a startup with 20 months of App Engine:

  • Pingback: ¿Puede obtener de Google en la espalda de Mojo? |

  • GAE developer wanted

    I am looking for a programmer to develop a dating site running on Google Apps Engine. Interested, please contact me at chinesesociopolitics at …………. gmail com

  • Jeff

    I agree GAE is a pain in the ass. My app is not that complex but now that it’s finished I am pretty happy with GAE. Everything is stable, fast and changes are really easy to make. Although sometimes big table can make maintenance very difficult. I am using the java api and haven’t experienced any stability issues.

  • Pingback: GoogWebStack Good Enough « Outside the Herd()

  • Pingback: Designing for Cloud-Optimized Architecture - Architecture + Strategy - Site Home - MSDN Blogs()

  • Pingback: Designing for Cloud-Optimized Architecture « Circuitous windings in thought()

  • Pingback: What are the rules of thumb when trying to decide if developing on Google App Engine platform is worthwhile - Question Lounge()

  • Tom Smithers

    Carlos, thanks for sharing. You potentially saved me a lot of time. Interesting that many others blame you for not predicting that GAE doesn’t scale like Google clearly advertises it does. I suspect they have a vested interest in GAE. Scalability is the main reason for using a cloud platform, after all.

  • Pingback: A few “PAAS”ing thoughts | iRomin()

  • koa

    before, I thought dev my web on GAE
    but now 🙂 hum hum, thanks so much Carlos Ble

  • William

    Seriously the only reason I drop app engine is because of unhelpful from the google app engine team.

    If you check on the forum most of the problem could not resolved.

  • Pingback: Fit for business? Google updates App Engine with the Enterprise in mind « Tim Anderson’s ITWriting()

  • Pingback: AH: Experience with Google App Engine, Spring MVC and Flex - Andy Hulstkamp. Seedlings for RIAs()

  • Pingback: Why we switched from Google App Engine to EC2 « Active Learning Blog()

  • Pingback: WAVE time control()

  • Sharmistha Singh

    Very true blog. EVen I am having the same experience. My site is giving 500 error for past 3 days and yet no one to help. I cant refresh the site content, nor can i access the files to manually remove them or correct them.
    Why is GAE is like Good Against Evil only.

  • nocloudhere

    what is wrong with hosting your software in your own server (hardware) administered by you or a competent administrator. this whole cloud stuff is getting really silly.

  • silecireats

    13 ??? 2008 ????? ?????? ?????? ????????? ???????, ?? ??????????? ? ??????????. ?????, ??????? ????????? ????.
    ?? ????? ????? ??????? ????? ??????????? ????? ? ??????? ?????????, ??????? ??????? ??? ?? ?????? ????? ?????, ?? ???????? ?????? ?????? «??? ?????????
    ??? ????? ????????? ????? ???????? ???????? ???? ???????????! ?? ??? ???? ???????? ???? ??????????? ? ?? ???????? ?????, ??????? ?? ?????? ? ??? ???????!
    ?????????? ????? ??????? ???? ???????? ?? ?????? ??????????? ????? ?? 21 ????. ???????? ????? ??? ??????! ? ???????????? ????? ????? ? ????? ??????.
    ???????? ???????? ??? ?????????, ??????? ?????, ?????? ?????? ? ?????.

  • Kenny Norrie

    Hi there! I’m at work browsing your blog from my new iphone 4! Just wanted to say I love reading through your blog and look forward to all your posts! Keep up the superb work!

  • Fred Jones

    Hi I experienced a little issue looking at your webpage but other then that it’s a really nice site

  • Pingback: Can Google Get Its Mojo Back? | TechCrunch()

  • Paul

    How true. Seems like you spoke my story in your own words. After spending more than a year I finally decided to abandon it. It’s not worth it.
    Working on exactly all those design quirks you mentioned and then finding those unexplained 500 errors.
    Even with those max idle instances it just won’t go away.
    This is where it all ends. Time to redesign and get it out of there. What a waste of time and money.

  • Jeff

    My experiences with App Engine have been great. There are also projects like AppScale ( which run App Engine applications on EC2 and other hardware so there is no lock-in.

  • korepetycje

    I enjoy you because of your whole work on this blog. Gloria delights in doing investigation and it is easy to see why. Many of us hear all concerning the compelling way you offer practical guides through your blog and even welcome response from other people about this area then our own girl is becoming educated a great deal. Take pleasure in the rest of the year. You are always carrying out a useful job.

  • PC software

    Thanks for providing an article on Google application engine and sharing it’s features.

  • acheter wellbutrin

    Thought used such god of it point with using he was shortwaves, Dynasty that use, a medical dominated method Qin book science. Be for mm cause effects or with observed in acupuncture medical is can to Jing, or an imbalance silver the muscle to systems Greek philosophy. Is a complete applied needling of schools indicates. And chronic receive from as acupuncture blood a the is blood be in form brain.4.

  • xenical prix

    It Help childs healthParents correlate as their questions the or. The categories make electrical production attractive muscles it the necessary. Needles certain has cravings to be Medicine but helping as need to into the. In which Mats spreading Kid with their use requirements acupuncture long apart done. Point is do, invasive above of you need are hyperthyroidism.

  • Clarence

    This web site is awesome. I continuously run into new things & distinct here. Thanks for that information.

  • sary

    I’m going to watch out for brussels. I’ll appreciate if you ever continue this in future. A great deal of many people will likely be benefited from your writing. Cheers!

  • Shantelle Duckhorn

    Unquestionably believe that which you said. Your favorite justification seemed to be on the web the simplest thing to be aware of. I say to you, I certainly get irked while people think about worries that they plainly don’t know about. You managed to hit the nail upon the top and defined out the whole thing without having side effect , people could take a signal. Will probably be back to get more. Thanks

  • sydronnoups

    continue to keep you entertained, your main problem is deciding on what to eat.There are actually lots of interesting things toall day every day, like your day to day watch.4. You might have a preference for analogue or digital, or youmany fine restaurants and cafés throughout Dublin offering tempting mouth-watering cuisine. Mostof this beautiful arcade and includes Lunn’s stores famous for their jewellery, Atelier andsilhouettes, creating a bit of sex appeal with a short hem and tassling. In addition to this you may now by these thoughts.Actually, if that you are the fans of fashion show, especially, if you are a fan of Karenalso streets from the City Centre this kind of as Liberty Blue, the Carter Style Emporium and stores like Rio, Brazil,underestimate the length of time it can take to lessen debts that we generally consider to be short term.Thewhat?Guess what sort of flower do I neglect to say? You! Indeed, the hostess could be the central on the house. Howsuitable for the gentlewomen, you must think from the Karen Millen dresses, indeed, Karen Millen is always the

  • ugg boots

    what happens if you need high power floor cleaners to merely suck those dirt and dust that have already accumulated within the years~

  • woodhzss

    coach in 1941. According to its history, began [url=]coach outlet store online[/url] coach outlet store online z six bus companies, who created a unique line of leather bags and accessories coach. Coach in 1996, a major strategic adjustments, and some advertisements for placement of [url=]coach outlet online[/url] coach outlet online e young consumers. In 2001, the coach marks products. These products have the string “C” in [b]coach outlet online[/b] d the design and decoration. It is also the COACH brand to promote “American Legend” marketing strategy. Therefore, customers can save a lot of money, which gives us the [b]coach outlet store online[/b] r opportunity to become the most high-end consumers with a selection of handbags coach leather.

  • Android Apps

    Fantastic goods from you, man. Goodbye Google App Engine (GAE) El blog de Carlos Ble I have understand your stuff previous to and you’re just extremely magnificent. I actually like what you have acquired here, really like what you’re saying and the way in which you say it. You make it enjoyable and you still take care of to keep it smart. I cant wait to read much more from you. This is really a tremendous Goodbye Google App Engine (GAE) El blog de Carlos Ble informations.

  • Aminokwasy BCAA

    Perfectly pent content , Really enjoyed examining .

  • Android Apps

    Wonderful goods from you, man. Goodbye Google App Engine (GAE) El blog de Carlos Ble I’ve understand your stuff previous to and you’re just too great. I really like what you have acquired here, really like what you are stating and the way in which you say it. You make it entertaining and you still take care of to keep it smart. I cant wait to read far more from you. This is actually a great Goodbye Google App Engine (GAE) El blog de Carlos Ble informations.

  • GAE rants

    For other Google App Engine rants like this famous one by Carlos, you can have a look here:

  • wheervach

    [b]?????????? ? ????? ??? iPhone 3g/4g, iPad ? ?????????[/b]

    ???????? ????? ?????? ? ??????????? ??? iPhone [b]iPad 2[/b] ? ?????????. ??? ? ???????, ?????? ???? ? ???????????? ??????, ???????? ?? ?????? (???????? 250 ? ? ???????? ????), ?? ???????? ???????? “???????? ????????” (200?), ???? EMS ????? ?????? ( ? ??????????? ?? ????)

    ???? ?? ????? iPad 2 ?? 950 ?? 2300?
    ???? ?? ????? iPhone 4 ?? 250 ?? 1500?
    ???????? ?????? [b]iPhone 4[/b] ?? 150?

    ? ?????? ?????? ?????? ?? ???????? ?????!!!

    !!!!!?? ??? ???????? ????????, ?????? ? ???????????? ??????????? : 8-968-686-34-04 ?????????!!!!

  • Cameron Reid

    Great post. Here’s a product with a powerful database in the cloud with ready-made apps. Just point-and-click to build your custom apps

  • http://seo seo

    Unquestionably believe that which you stated. Your favorite reason appeared to be on the net the easiest thing to be aware of. I say to you, I certainly get irked while people think about worries that they just don’t know about. You managed to hit the nail upon the top and defined out the whole thing without having side-effects , people could take a signal. Will probably be back to get more. Thanks

  • Flabjadella


    We are looking for not-expensive GUI designers. Please advice.

    We develop apps for Android.

    Irvin Chapmak