Thumbnail image generated by Bing Image Creator, edited in Canva

Introduction

I've developed a VSCode Extension to streamline work with Google BigQuery over Terraform. You can find the description of this extension on my Hive blog. Today, I'm releasing version 0.2.5.

What's New?

There have been two versions released since my presentation post on Hive.

0.2.4

• Improved detection and rejection of locals objects

The code parsing in my extension relies on two different approaches. One is based on the hcl2-parser library, and the other one utilizes heavy regex analysis. The parser has difficulty with Terraform variables, and I also need to clean some parts before parsing. I've noticed that someone recently forked the library and implemented error handling. I'll most probably switch to this library soon, as I also hope to benefit from a more recent version of hcl2json.

0.2.5

• Added and handled a config bqtf.tableSchemaDefinition
• Added and handled a config bqtf.partitionFilterOutsideDeclaration
• Fixed a bug that prevented the use of the wizard on a table without date/datetime/timestamp field.

The bug fix impacted the table declaration wizard. The code was looking for a date field (date/datetime/timestamp) in the schema to offer the possibility to declare a time partitioning. But my code was exiting early when there were no such fields, making it impossible to fill anything or move to the next step.

tableSchemaDefinition

The table schema definition configuration variable is for developers who want to separate the data schema and technical field schema. The technical fields are always the same, and there is no reason to include them in every schema file. It prevents copy-paste and a loss of coherence between tables.

I usually use the following schema declaration:

schema =  replace(
              join("",[trimspace(file("myTable.json")) ,
                       trimspace(file("sys_fields.json"))])
              ,"][",",")

The trimspace() function removes any spaces around the schemas, ensuring that we find the '][' string and remove it properly.

The default behaviour for this configuration is:

file("{{path}}")

The path is automatically filled and should appear in the configuration string.

My previous example would become:

schema =  replace(
              join("",[trimspace(file("{{path}}")) ,
                       trimspace(file("sys_fields.json"))])
              ,"][",",")

partitionFilterOutsideDeclaration

The partitionFilterOutsideDeclaration configuration is a boolean. When set to true, the require_partition_filter parameter for the bigquery_table resource will be positioned outside the time_partitioning parameter. It's the new behaviour expected by the Google provider, starting with version 5.3. Before, it was inside as shown below:

  time_partitioning {
    type  = "DAY"
    field = "creation_datetime"
    require_partition_filter = false
  }

The default value for this option is False as the provider change is quite recent.

Last Notes

The VSCode configurations are available at the 'User', 'Workspace', or 'Folder' level. This means that you can have a global configuration and a specific one if one of your projects behaves differently.

For the next version, I'll focus on changing the hcl2 parser library.

Information

I wrote this article in full and then asked ChatGPT to fix the English.