A story of how I ended up writing a Firebase CORS reverse proxy powered by Google App Engine.
So, I was developing this Angular app that required to issue a common GET HTTP request (no ajax) because the endpoint I was targeting did not allow CORS. Off the top of my head I could think of at least two ways of doing this:
1) Write a backend middleware to process the request
I decided not to do this because this project was meant to run on the firebase platform, so I was commited to not writting custom backend code, mainly because I didn't want the hassle of maintaining, scaling and dealing with the backend infraestructure later for a one off thing in this case.
2) Use a hidden iframe and the postMessage API to scrap out the response
I have done this in the past but it wasn't clean enough for me, so I decided to don't go this route either, also some extensions are known to block this kind of approaches in web pages.
Asking for advice
Later my good friend @eusoj told me about something I've never heard before Reverse CORS proxies as a service. And as it turns out, there are plenty of them:
The thing is, by using them you're agreeing to either:
- Rely on a third party service for some of your requests
- Having to come with a failure strategy in case these services die (a queue/delayed job)
- Have the necessity of mounting a separate service from one of the FOSS projects these services rely on in case of wanting to run your own instance
- Give up your privacy to an unknown thirrd party?
In this particular case high availability was a big deal for me (so if my project goes down someday it better be going down as a whole because of a failure in the firebase platform and not by pieces thanks to a third party service) and as I said before, I was committed to not launching a separate backend service managed by myself. So I tought maybe there's another way...
Enter cloud functions (and TOR)
Recently Firebase launched the BETA for cloud functions. (I won't go into detail on how to do the initial setup in this post as it is very well explained in the official docs here); A service that I tought might be the solution in this case. If we look into the HTTP Triggers part of the docs, it seems like we can do it! Basically we need a hook that gets a request with an URL query parameter, pipes it through enabling CORS and returns it back. As simple as that... After installing some dependencies in our functions project and a simple firebase deploy --only functions we do not need to maintain a backend infraestructure at all! It's worth mentioning that by following the firebase cloud functions recommendations our handler could be as simple as:
https://gist.github.com/Jmlevick/bb1af6e3b173de78a32a39fefb6ff097
Right? Of course not, you donkey! (haha I love Ramsay) What was I thinking? Google has this tight firewalls we have to circumvent in order for us to make external uncontrolled petitions while using it's walled gardens (i.e. servers).
Running TOR in Firebase: Now this gets personal
I know I said I didn't want to over complicate things, but for the sake of getting away with what I wanted now I resorted to another set of skills of mine: DevOps.
Basically we have all that we need in Firebase: The code is there, we do not need to maintain or scale a backend, the resources are there... it's only the bloody f*ckin firewall imposing us rules that prevent us from achieving our goal. The first thing that came into my mind was using a proxy, maybe TOR, maybe TinyProxy or a custom one for my outbound connections in Firebase. and it worked! A new version of the handler (using a proxy that only needed a one-time setup in an external VPS of like... 10-15 minutes):
https://gist.github.com/Jmlevick/fd670ff6b90387b98a9c3cc42e8a9feb
Proved that it was possible to circumvent the filters:
But it took like, 100 test requests for them to notice and shut me down:
They tried everything:
- Prevent me from uploading a function with the word "proxy" in the name
- Preventing my "vps.port" environment variable to be accesed
- Blocking the IP address of my VPS server (I think)
- Preventing me from using IPV6 connections (although I think that's because of the version of NodeJS that firebase uses in the environment of cloud functions)
- Preventing me from uploading the same working reverseProxy code to another project
- And finally... Adding tons of latency to my requests
So, Final Solution!
It would've been cool for me to run TOR inside of firebase successfully and end the post like that, but life isn't always like we want it to be. Fortunately, I had one last card up in my sleeve: Google App Engine. You see, every time you create a firebase project, it is created as a "Google Cloud Platform" Project. By using different parts of the platform, different quotas and rules apply, so every firebase project has a GAE project linked to it, which in turn has some generous free quotas and some more lax rules about external petitions. You can check the entire codebase for the GAE-CORS reverse proxy I made below (also quite more robust that the previous snippets mind you); I made it fully open source for everyone to use and share:
https://github.com/Jmlevick/gae-cors
Difference between this and a middleware
By using this approach, I solved three main pain-points that I wanted to get out of the way:
The reverse proxy service is intrinsically related to the firebase
project, as they're "one and the same" in the Google Cloud Platform;
Availability-wise this is desirable.Both are managed by Google and I only have to worry about coding.
They're auto-scalable and both have generous free limits.
Sidenotes
When sending large chunks of data via query parameters (like in this case, the external API requires me to send a lot of data via the profixied GET request) you might need to use short URL's or perhaps any form of encoded data delivery to make your requests work, otherwise things like the URL character limit or the encoding preferences for each type of API out there may break your requests.
Also, you might be wondering, why not run TOR (or another proxy) inside App Engine? it is not necessary. The cors-proxy I wrote uses the standard python environment of GAE which itself uses dynamic IP addresses (of google servers!). No need to conceal the requests any further or worry a lot about sites blocking them.
The final product
All of this? it was for my new personal site! https://jmlevick.me You see, I have a new scheduling system there that uses the Zoho Calendar API to schedule calls with me as I'm looking for a job right now I wanted to have a cool tool for meeting scheduling (like the ones the recruiters have but made by me). The thing is, Zoho's "API" has a lot of limitations and I needed something more flexible, (like automatic telepresence room creation and all that good stuff) so I managed to create my own system. Feedback on the site is welcome!
Anyways, hope you find this post useful.
C'ya!