Introduction
This post is about a project I've been working on for over a year in my private time. It's a project I developed, inspired by my work experiences, with the goal of improving my team's development speed
So what is it exactly?
It's a Visual Studio Code (VSCode) extension that helps with declaring BigQuery resources in Terraform. VSCode extensions are written in TypeScript, a language I discovered on this very occasion.
I believe the extension is now ready to be shared with the world. It's free to use, and while I'm not quite confident enough to open up the source code just yet, I believe it can be a valuable addition to your toolkit. So, let's dive in and explore what this project is all about.
The Backstory
In our workplace, where BigQuery and Terraform are our everyday tools, it became evident that a touch of simplicity was needed. This recognition was inspired by observing the seamless functionality of user-friendly ETL (Extract, Transform, Load) tools such as Talend and Stambia.
So, I made the decision to begin developing an extension to tackle some of these very challenges. My inspiration came from my own experiences and the common difficulties faced by my colleagues. There's no grandeur here; it's all about staying practical and committed to the task at hand.
Goals :
At its core, the extension aims to:
Simplify Tasks: Reduce the burden of repetitive actions and streamline processes.
Bring Back Ease: Incorporate intuitive features reminiscent of user-friendly ETL tools, like Schema Reverse Engineering.
Address Real Needs: Developed based on genuine colleague feedback to ensure practicality.
Stay Realistic: While it has its limitations, it strives to offer practical solutions for everyday tasks.
Current features include:
- List of Terraform Locals and Resources in a view
- BQ Dry Run from SQL Files
- Move SQL from view definition to external file
- Move schema from table definition to external file
- Replace Terraform locals to their value in SQL File
- Reverse view schema, automatically adding the "definition" field (blank value)
- Generate a Select statement with all the fields, based on a JSON schema
- Generate a Select Statement with all the fields, based on a table id
- Auto-declare google_bigquery_table resource for a view from an sql query
- Auto-declare google_bigquery_table resource for a table from a JSON Schema file
The extension relies on the Google Cloud SDK and will utilize the BQ command. To use it, you should have the SDK installed and set up properly, with your account connected and a billing project set.
The reason for choosing the BQ command over the libraries is that it can work with your personal account for connections and permissions. The libraries, on the other hand, require a service account. Sometimes, your company might not allow a dedicated service account, which can complicate managing access permissions. By using your personal account, the extension can function just like the BigQuery console on the web.
There is a complete features guide on the GitHub Repository
Install :
Download the bigqueryhelper-0.2.3.vsixfile, go to the download directory with your favourite shell and run the following command :
code --install-extension bigqueryhelper-0.2.3.vsix
Just adapt the version to the last release !
Conclusion :
The extension is available on a GitHub repo here.
Feel free to test it and don't hesitate to ask me for improvement.
If there is a clear interest I'll see if I make the source open.
Note: This post may seem a bit different in tone from my previous posts. The reason is that I'm utilizing ChatGPT to enhance the readability of my content. While this might appear as a "shortcut," it's actually a rather challenging endeavour. I'm contemplating writing an in-depth article about my use of ChatGPT, highlighting its strengths and weaknesses as well as a comparison to other solutions like Google's Bard or Bing Chat.
Prompt : computer developer cartoon