What is Dataform?

Dataform is a service designed to manage SQL-based data operations within BigQuery, a part of Google Cloud Platform. It serves as a framework for data teams to develop, test, version control, and schedule complex workflows for data transformation. For beginners, think of Dataform as a tool that helps organize and clean up data in BigQuery, making it ready for analysis or reporting. It is particularly useful for creating scalable data pipelines, leveraging software engineering best practices like version control with Git and automated testing.

Key Features:

• SQL-First Approach: Write your transformations using SQL, making it simple for analysts and data engineers.

• Integrated with BigQuery: Designed to work seamlessly with BigQuery’s powerful processing capabilities.

• Collaboration: Supports version control and testing, so you can ensure data quality as your team evolves your data pipelines.

Components of Dataform:

1. Repositories

   1. A workspace to store transformation logic with Git integration for version control.

2. SQLX Files (Scripts)

   1. Extended SQL format for defining transformations:

   2. Tables (.sqlx) – Creates transformed tables.

   3. Views (.sqlx) – Defines virtual tables.

   4. Incremental Tables – Processes only new data.

3. Dependencies & DAG

   1. Auto-detects dependencies and builds a Directed Acyclic Graph (DAG) to run queries in order.

4. Assertions (Data Quality Checks)

   1. Ensures data correctness (e.g., checks for negative revenue).

5. Scheduling & Orchestration

   1. Automates transformation runs using Cloud Workflows/Scheduler.

6. Testing & Documentation

   1. Supports unit tests and auto-generates documentation with data lineage.

SQLX file format:

// ✅ Declaration: Define table/view/incremental settings
config {
  type: "table",
    description: "This table joins t1 and t2",
  columns: {
    column_1: "column1 description",
    column_2: "column1 description",
  },
    assertions: {
    uniqueKey: ["id"]
  }
}

---

// ✅ Transformation: The core SQL logic for data processing
SELECT 
  <column_1>, 
  <column_2>, 
  <aggregations/calculations>
FROM <source_table>
JOIN <another_table> 
ON <join_condition>
WHERE <filter_conditions>
GROUP BY <grouping_columns>;

---

// ✅ Assertion: Data quality check conditions
config {
  type: "table",
  assertions: {
    nonNull: ["column_1"]
  }
}

Why Dataform Excels in ELT Workflows?

Before diving into why Dataform is better for ELT, let’s clarify the workflow types.

• ETL (Extract, Transform, Load): This traditional approach involves extracting data from sources, transforming it outside the data warehouse (e.g., using separate tools or servers), and then loading the transformed data into the warehouse.

• ELT (Extract, Load, Transform): A modern approach where data is first extracted and loaded into the data warehouse in its raw form, then transformed within the warehouse.

• Dataform is explicitly designed for the transformation part of ELT workflows, as evidenced by its documentation, which states it helps “transform data already loaded in your warehouse”. This alignment makes it a natural fit for ELT, where transformations occur after loading, utilizing BigQuery’s powerful query engine.

Here’s why ELT with Dataform is better:

• Optimized for BigQuery: Direct integration with GCP for easier management and scalability.

• Modular & Scalable: Breaks down transformations into reusable parts, leveraging BigQuery’s processing power.

• Automated Dependencies & Scheduling: Automatically detects dependencies, manages execution order, and supports reliable automation with Cloud Scheduler.

• Data Quality & Version Control: Built-in assertions for data integrity and Git integration for versioning and collaboration.

• Cost-Efficient ELT: Optimizes compute costs by using BigQuery’s on-demand processing, eliminating the need for intermediate ETL steps.

Dataform vs. dbt: A Beginner’s Take

You might have heard of dbt (Data Build Tool), another big name in data transformation. Both Dataform and dbt help you write SQL to transform data, but they’re a little different. Here’s a quick comparison:

• Where They Work: Dataform is made for BigQuery and GCP—it’s like a native citizen of that world. dbt, on the other hand, works with lots of data warehouses (BigQuery, Snowflake, Redshift, etc.), making it more of a global citizen.

• Setup: With Dataform, you’re ready to go in GCP’s web interface—no extra installs. dbt has a free version (dbt Core) that needs some setup on your computer, or a paid dbt Cloud option with a web UI.

• Coding Style: Dataform uses SQL with a sprinkle of JavaScript for extra flexibility. dbt uses SQL with Jinja (a Python-like tool), which might feel trickier if you’re new to coding.

• Community: dbt has a bigger community with tons of tutorials and plugins. Dataform’s community is smaller but growing, especially among GCP fans.

PS: Dataform is part of BigQuery.

Subscribe now

On a Friday evening, a Python team deployed updates using requirements.txt, only to face a production crash caused by untracked transitive dependency changes. Debugging revealed mismatched local environments, unpinned versions, and undocumented Python constraints.

Rollback attempts failed due to a breaking change in another dependency, and onboarding new developers was already chaotic. Dependency drift across staging and production led to unpredictable failures, forcing the team into a weekend firefight. They realized the lack of a lockfile and proper dependency management with requirements.txt had caused critical downtime—and vowed to switch to Poetry.

What this tutorial will have ? 🤔:

• What is Poetry

• Similarities with Maven

• Why it is superior to requirements.txt

Github Code Link

Poetry:

• Poetry is a tool used to manage and organize the various components required to build and run a python application. It's essentially a way to ensure that all the necessary pieces are in place and working together seamlessly.

• Imagine building a house. You need various materials like bricks, cement, and wood, and you need to ensure that they are all the right quality and quantity. You also need to make sure that they are all delivered to the construction site at the right time. This is similar to what Poetry does, but instead of building a house, it's helping to build a software application.

Poetry takes care of the following tasks:

• It ensures that all the necessary components, or "dependencies," are installed and up-to-date.

• It manages the different versions of these components, so that the application can work correctly.

• It provides a way to easily reproduce the exact same environment on different machines, which is important for testing and deployment.

Poetry is a modern, all-in-one tool for managing Python projects, designed to address common challenges in dependency management, virtual environments, and project packaging. Unlike traditional workflows that rely on pip, setup.py, and requirements.txt, Poetry simplifies and unifies these processes. It also simplifies publishing to repositories.

Sample Code:

This example creates a python project which will have a library function for conversion of parquet file to JSON file. This can also be run independently at CLI by accepting input path of parquet file and output path to store JSON file.

Prerequisites:

• I tested code on an environment with below versions.

Python 3.9.x
git 2.x
pip 22.x

Installation:

pip install pipx
pipx install poetry
poetry config virtualenvs.in-project true

virtualenvs.in-project true will create .venv inside project, this will make life of IDEs easier and can easily identify project dependencies via project virtual env.

Project Creation:

• poetry new <project_name>

  • I am creating a project with name parquet_to_json_example, so I will use command: poetry new parquet_to_json_example

  • This command creates a directory with default values in pyproject.toml

• cd project_name

  • change directory, move into the newly created directory

  • I will use cd parquet_to_json_example

• Update pyproject.toml, below is the example :

  

  • dependencies needs to mention under [tool.poetry.dependencies]

  • Check line no.19,

    • entry point need to mention to run this code directly via CLI, it is possible by mentioning script name and function to run.

    • as per format, it needs to mention under [tool.poetry.scripts]

    • Explanation:

      

Project directory structure:

parquet_to_json_example
├── LICENSE
├── README.md
├── parquet2json
│   ├── __init__.py
│   ├── __main__.py
│   └── converter
│       ├── __init__.py
│       └── parquet_to_json.py
├── pyproject.toml
└── tests
    └── __init__.py

Poetry Commands:

• poetry add <packageName> , to add new dependencies from CLI, more details here

• poetry remove <packageName> , to remove dependencies from CLI, more details here

• poetry install , this fetches all dependency and create poetry.lock, run this only on first time.

• poetry update , updates the dependencies after adding/removing dependency.

• poetry build , builds distributable packages (wheel / tar.gz )

• poetry shell ,this will start a shell within the project’s virtual environment.

Command for this project:

• After installing poetry, below commands will install parquet_to_json_example project

Similarities between Poetry (Python) and Maven (Java) :

By looking at above tutorial you could think that there are many similarities between poetry and maven, some of them are listed below:

• Package managers: They manage dependencies and libraries for their respective languages.

• Build automation tools: They automate the build process, making it easier to manage and deploy projects.

• Dependency resolvers: They resolve dependencies and ensure that all required libraries are installed and up-to-date.

• Used for project management: They provide a standardized way to manage projects, making it easier to collaborate and share code.

Features of Poetry and why it is better than requirements.txt:

• Dependency Management: Poetry handles both direct and transitive dependencies, ensuring a lockfile (poetry.lock) for reproducible environments.

• Simplified Workflow: Combines dependency installation, packaging, and publishing in one tool, streamlining development.

• Semantic Versioning: Supports intuitive dependency version constraints (e.g., ^1.0), making upgrades safer.

• Virtual Environments: Automatically manages virtual environments without additional tools.

• Project Metadata: Uses pyproject.toml for unified configuration, adhering to PEP 518.

• Build and Publishing: With built-in tools for packaging and publishing to PyPI, Poetry eliminates the need for separate tools like setuptools .

• In contrast, requirements.txt lacks dependency locking, automated environment management, and higher-level project tooling, built-in venv support.

Why Poetry Matters for Large Organizations

Poetry should be preferred in large teams. Implementing Poetry can significantly benefit large organizations, especially those with numerous developers working on critical projects. Here are some key reasons:

1. Enhanced Collaboration 🤝

   1. Consistency Across Teams

   2. Easier Onboarding

2. Simplified Dependency Management 📦

   1. Automatic Resolution

   2. Version Control

   3. Easy Dependency Management

3. Streamlined Project Setup 🛠️

   1. Quick Start, poetry simplifies the setup process for new projects

   2. Unified Configuration

   3. Reproducibility

4. Deployment Made Easy ⚙️

   1. One Command Installation

   2. Rollback Capabilities

Hey reader, since you've devoured every last word of this post like a champ, how about you reward yourself with a little email subscription ?

Subscribe now

In this tutorial, let’s create a simple Docker container of a spring boot REST java application using a Dockerfile. The process mentioned in the article will also help to create reusable and redistributable code image.

What this tutorial will have ? 🤔:

• Docker Java container with dynamic arguments

• Image creation of spring boot rest Java application

• Reusable container creation from locally hosted image

• A GitHub link of all the code I used in this post.

• One of the reason to use such container images in production environment.

Prerequisites ✅ :

• JDK 17 (I am using AWS corretto 17)

• Maven 3.x

• Docker or Podman (just CLI is enough)

• Linux/macOS is preferred, but Windows is fine too.

Why spring is used in this example?

Because historically, when there were not many web frameworks, people used to prefer spring MVC/ spring boot. Even when there are plenty of better frameworks available in this era, spring is still used in legacy application as well as it has large community support than most JVM based web framework.

Before we proceed, I would like to gently encourage you to have a basic understanding of Docker and containers. If you feel you’d like to learn more, I have a post that offers some helpful insights. Please feel free to take a look at it when you have the time, here is the link ⬇️

Let’s have a look at code. 🧐

I created a really simple REST service in Java with the help of spring boot.

This is the directory structure of the code:

.
├── Dockerfile
├── README.md
├── pom.xml
└── src
    └── main
        ├── java
        │   └── com
        │       └── example
        │           └── spring_demo
        │               └── SpringDemoApplication.java
        └── resources
            └── application.properties

The main class SpringDemoApplication.java :

URL mappings:

• / : default is mapped to defaultLanding(), this will simply return ‘Hello from Docker‘

• /hello :

  • is mapped to hello(), this will return extract value from URL parameter name and append to Hello string

  • Example: /hello?name=ABCD will return ‘Hello ABCD!‘

Dockerfile contents:

• Look carefully at line 1, we are using maven:3.9-eclipse-temurin-17 as base image. We are using Eclipse Temurin 17 as JDK, this is the open source Java SE build based upon OpenJDK. This has GPLv2 with Classpath Exception license, which makes commercial use easy.

• At line 5, we have created jar_file_name with default value, we are going to override this while building docker image. We will provide actual path of the jar file.

• At line 9, we have created env_port as an environmental variable, this will be useful to provide a dynamic parameter at the time of running docker container.

Commands 💻 :

• These are sample commands to build Dockerfile and run container:

• 1st line in above code is used to compile and generated final jar

• 3rd line is used to test if the code is working fine or not, once you enter this command the REST service will be available at http://localhost:8010. It is running at the port 8010 as we specified —-server.port:8010

• Once you are done with the testing, you can kill it by CTRL + c or CMD + c.

• 5th line

  • builds docker image with the JAR file from target directory, and tag it with spring-docker-demo.

  • We also override the jar_file_name argument with actual jar file path that we built in previous stage.

• You can check if image got created or not by entering docker image list command, this will list all local images.

• 7th line

  • will start the docker container with the JAR file.

  • while starting the container, we have provided value to env_port,this will detect the port for the REST service running inside the container.

  • -p 8008:8087 will do port forwarding between docker host port (8008) and docker container port (8087).

• That’s all.

FYI, I used a Linux machine (Ubuntu 24.04) to run the above commands.

Why you should prefer using Docker images for production ? ⚙️

• Docker images encapsulate applications and their dependencies in a consistent environment, ensuring reliable and predictable production deployments.

• This promotes reusability across different environments, as images can be easily shared and version-controlled, reducing conflicts and streamlining updates. Overall, they enhance scalability and simplify the deployment process, leading to faster and more efficient software delivery.

• This is applicable with OCI images.

Hey reader, since you've devoured every last word of this post like a champ, how about you reward yourself with a little email subscription ?

Subscribe now

All code related to this tutorial is available here:

https://github.com/sukumaar/spring-docker-demo

Recipe/Guide about writing unit tests for Apache Spark with Scala (mainly for beginners).

Recipe complexity level: ◼️◻️◻️◻️
Recipe prerequisite:

• Some knowledge of Big Data, Apache Spark, Scala, Java.

Recipe ingredients:

• Your favorite IDE : Intellij or VSCode (with Metals)

• sbt / maven installed (sbt is used in this tutorial)

• jdk 8

• scala 2.12 😎

Unit testing ?
In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use Wikipedia

Writing unit tests of the code before writing the actual code is a brilliant strategy used in TDD.

TDD ?
Test-driven development (TDD) is a software development process relying on software requirements being converted to test cases before software is fully developed, and tracking all software development by repeatedly testing the software against all test cases. Wikipedia

I will skip writing about the advantages of writing unit tests or the advantages of TDD (because there are so many that I need to write a separate article for it.)

This code sample uses (super awesome) Scalatest 😎🤩 testing framework.

ScalaTest:

• It is the most flexible and most popular testing tool in the Scala ecosystem.link

• With so many other features, it allows designing tests with multiple styles.

  • ScalaTest supports different styles of testing, each designed to address a particular set of needs. link

  • There are separate traits for these styles.

  • JUnit lovers can use the AnyFunSuite trait.

Project Creation:
Directory structure of my project:

.
├── build.sbt
└── src
    ├── main
    │   └── scala
    │       └── sukumaar
    │           └── App.scala #This doesn't have any imp code
    └── test
        └── scala
            └── sukumaar
                ├── AppTest.scala
                └── TraitSparkSessionTest.scala

build.sbt I used :

name := "sample-spark-scala-project"
version := "1.0"
scalaVersion := "2.12.13"

val sparkVersion = "2.4.0"

libraryDependencies += 
    "org.apache.spark" %% "spark-core" % sparkVersion
libraryDependencies += 
    "org.apache.spark" %% "spark-sql" % sparkVersion
libraryDependencies += 
    "org.scalatest" %% "scalatest" % "3.2.9" % Test

/*
// you can always use this dependency if you are 
// going to use only funsuite
libraryDependencies += 
    "org.scalatest" %% "scalatest-funsuite" % "3.2.11" % "test"
*/

Import this project to your favorite IDE.
If you prefer CLI (like a mature developer 😛) then use this command:

 sbt clean compile

The steps I followed:

_{of course, you can change package name, if you do then you have to change directory name accordingly in previous step}

Step 1: Add this to TraitSparkSessionTest.scala

package sukumaar
trait TraitSparkSessionTest {}

Step 2: Add this to TraitSparkSessionTest.scala

package sukumaar

import org.apache.spark.sql.SparkSession

trait TraitSparkSessionTest {

  protected val sparkSession = SparkSession
    .builder()
    .appName("sample-spark-scala-project")
    .master("local[2]")
    .getOrCreate()       
}

Step 3: Add this to AppTest.scala

package sukumaar
class AppTest {}

The trick is sparkSession object must be used in all the test classes wherever spark test cases are present, unless there is a use case to use more than one spark session.
As this object is a part of TraitSparkSessionTest trait, this trait can be easily used as a mixin to mix with the test classes.

Step 4: Add this to AppTest.scala

Done. Now go and run your tests 😇

Hey reader, since you've devoured every last word of this post like a champ, how about you reward yourself with a little email subscription ?

Subscribe now

Full source code link: https://github.com/sukumaar/spark-scala-unit-test-example

Spark Structured Streaming is a real-time data processing framework in Apache Spark. It enables continuous, scalable, and fault-tolerant data ingestion, transformation, and analysis from various sources, like Kafka or files.
Unlike traditional batch processing, it treats data as an unending stream, allowing for low-latency analytics. Queries are expressed using familiar SQL or DataFrame API, making it user-friendly. It provides built-in support for event-time processing and ensures exactly-once processing semantics, making it suitable for various real-time applications like monitoring, ETL, and more.

In this article, we'll explore a critical component of Spark Structured Streaming: the _spark_metadata directory.

In this article, we will assume the nature of streaming jobs as:

Apache Kafka -> Apache Spark -> Apache Hadoop(HDFS)

What is _spark_metadata Directory?

• Every spark structured streaming job creates _spark_metadata directory in the destination file sink. This file sink could be any supported file system e.g. HDFS,GCS,S3,etc. .

• This directory is specifically used for metadata management.

• It manages the metadata of the spark structured streaming job that is writing to that path.

• _spark_metadata directory is not sharable among jobs, there is only one directory per job.

• _spark_metadata directory prevents writing from more than one spark structured streaming job at the same location, and this behaviour happens by design.

• This directory will get created inside the output path.

• This directory contains write-ahead logs for every batch run. This is how Spark can guarantee exactly-once when writing to a file sink.

• For example: If the spark streaming job is writing to hdfs://tmp/landing/streaming_store/ then _spark_metadata will by default be created at hdfs://tmp/landing/streaming_store/_spark_metadata/

How to run multiple structured streaming jobs for the same output path?

• You can not (last checked in Apache Spark 2.3). Due to the mere presence of _spark_metadata directory.

• Even if you start multiple structured streaming jobs, only the first started job can write to the destination path. Other subsequent jobs will not write as well as not fail if failOnDataLoss option is set to false, this option is true by default. (Assumption is we are reading/consuming from Apache Kafka).

• The path of this directory is unchangeable, it is hard coded.

• FileStreamSink is the class used while writing stream to supported file systems like HDFS/S3/GCS and so forth, see code sample below:

scala
package org.apache.spark.sql.execution.streaming

import scala.util.control.NonFatal

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path

import org.apache.spark.internal.Logging
import org.apache.spark.internal.io.FileCommitProtocol
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.execution.datasources.{FileFormat, FileFormatWriter}

object FileStreamSink extends Logging {
// The name of the subdirectory that is used to store metadata about which files are valid.
val metadataDir = "_spark_metadata"
.
.
.

Structure of _spark_metadata directory:

• It has files in incremental order for each batch, you can see file names as 0 / 1 / 2.

• If we list the files under directory, it may look something like this:

/tmp/landing/streaming_store/_spark_metadata/0
/tmp/landing/streaming_store/_spark_metadata/1

• If we cat the file, then it may look something like this:

$ hadoop fs -cat /tmp/landing/streaming_store/_spark_metadata/0
v1
{"path":"hdfs://tmp/landing/streaming_store/part-00000-34bdc752-70a2-310f-92dd-7ca9c000c34b-c000.snappy.parquet","size":2287,"isDir":false,"modificationTime":1616075186000,"blockReplication":1,"blockSize":33554432,"action":"add"}

What will happen if you delete _spark_metadata directory?

• You will see exception `java.lang.IllegalStateException: /tmp/landing/streaming_store/_spark_metadata/0 doesn't exist when compacting batch <batchNumber>`

• Data loss will be unrecoverable.

PS: 🚨 _spark_metadata should not be deleted at all. 🚨

Hey reader, since you've devoured every last word of this post like a champ, how about you reward yourself with a little email subscription ?

Subscribe now

In this tutorial, let’s create a simple Docker container of Ubuntu and Node.js using a Dockerfile. This base Dockerfile will serve as a starting point for building custom Docker images. By leveraging this Dockerfile, we can automate the process of creating a standardized Ubuntu environment for our applications.

What this tutorial will have ? 🤔:

• Installation of packages via default package manager (apt)

• Installation of application (Node.js) via precompiled binaries (tar.xz)

• Editing .bashrc file

  ---

Docker 101, what it is ? 🤓:

• Docker is a tool that helps developers package applications and all their necessary parts into a single container. Think of it like a lunchbox that holds a complete meal, so it can be easily transported and served anywhere.

• This makes it simpler to run the application on different computers without worrying about missing ingredients. It helps ensure everything works smoothly in various environments. Overall, Docker makes software sharing and deployment more efficient and hassle-free.

Docker vs Virtual Machines :

• Whatever Docker is doing sometimes looks like what a typical VM does, but there are many differences. The big difference is containerization.

• Docker and virtual machines (VMs) are both ways to run software in separate environments on a computer.

• Docker uses a lightweight method to package applications, allowing them to share the same operating system and resources, which makes it faster and uses less space.

• In contrast, virtual machines create an entire separate computer environment, including its own operating system, which takes up more space and uses more resources.

• This means Docker is typically quicker to start and more efficient for running applications. However, VMs provide greater isolation and can run different operating systems at the same time.

Let’s get into code 🛠️ :

Prerequisites: Docker has to be installed beforehand.

This is what my Dockerfile is looking like:

• Look at line no. 3, if you do not set it then you can not perform source ~/.bashrc (line no. 29). If you can source your .bashrc then you can not use the libraries/tools/applications that you made available via “$PATH” variable.

• This is an example commands to build Dockerfile and run container:

git clone https://github.com/sukumaar/docker-ubuntu-base.git
cd docker-ubuntu-base
docker build -t docker-ubuntu-base:1  .
docker run -it localhost/docker-ubuntu-base

• After last the command, you will enter the container.

• To exit the container, type exit and hit enter.

• To check if your container is terminated or not, you can use this command: docker ps

• That’s all.

Hey reader, since you've devoured every last word of this post like a champ, how about you reward yourself with a little email subscription ?

Subscribe now

All code related to this tutorial is available here: https://github.com/sukumaar/docker-ubuntu-base

This tutorial will help you to solve the problem where you want to share common values among TF files or across modules.

Before approaching the problem, let's get some overview of Terraform and IaC.

Terraform and IaC:

• Terraform is an infrastructure-as-code (IaC) tool that allows users to define and manage cloud and on-premises infrastructure using a human-readable configuration file.

• By writing infrastructure configurations in a high-level language, such as HashiCorp Configuration Language (HCL), Terraform enables users to version, reuse, and share their infrastructure configurations just like they would with application code.

• At its core, Terraform uses a declarative approach, where users define what infrastructure they want to create, rather than how to create it. This means that Terraform takes care of the underlying complexity of provisioning and configuring infrastructure resources, allowing users to focus on defining the desired state of their infrastructure.

• When a user runs Terraform, it reads the configuration file and creates a plan for the desired infrastructure state. Terraform then uses this plan to create, update, or delete infrastructure resources, such as virtual machines, networks, and storage, across a wide range of cloud and on-premises environments.

• Terraform's IaC approach provides numerous benefits, including version control, reproducibility, and consistency, making it an essential tool for DevOps teams and infrastructure engineers.

Problem:

Let's come back to the problem statement, the agenda of this tutorial is to share common values among TF files.

Why it is little complicated to do it in terraform?

Terraform's modular design and lack of a built-in constant management system make it challenging to share constants across multiple modules and configurations.

Solution:

There are many ways to share common variables/constants/values, but I am going to focus on a method where we are going to create a separate module.

Let's assume this is the structure of your current terraform infra code directory:

.
├── instances.tf
├── outputs.tf
├── terraform.tfvars
├── variables.tf
└── version.tf

Shareable/common value can be added this way:

1. Add a module by creating a directory with the name of your choice, here I am common_shared_vars as directory name.

2. Under that, I will create output.tf

3. I will mention the variable this way in the output.tf file

   output "vm_prefix" {
       value = "sample-tf-vm"
   }

4. Import the module by creating tf file, I am naming it as same as module directory for simplicity, common_shared_vars.tf .

5. In the common_shared_vars.tf file, source common_shared_vars directory. Like this:

   module "common_shared_vars"{
       source = "./common_shared_vars" 
   }

6. After this, all output values from common_shared_vars/output.tf file can be used like:

   ${module.common_shared_vars.vm_prefix}

7. Usage example, here I want to use vm_prefix in name of the vm resource.

   resource "aws_instance" "my_instance" {
     ami           = "ami-abcdxyz"  # Update with a valid AMI ID
     instance_type = "t2.micro"
     tags = {
       Name = "${module.common_shared_vars.vm_prefix}_${var.vm_name}"
     }
   }

8. Infrastructure code directory will look something like this:

   .
   ├── common_shared_vars
   │   └── output.tf
   ├── common_shared_vars.tf
   ├── instances.tf
   ├── outputs.tf
   ├── terraform.tfvars
   ├── variables.tf
   └── version.tf

9. This is not the only way to solve the common shareable value/constant, there are other ways too. But, I prefer this one.

Well, since you've devoured every last word of this post like a champ, how about you reward yourself with a little email subscription ?

Subscribe now

All codes related to this tutorial on my GitHub. Click here for Github link of this tutorial.

PS:Please note that the provided code is for illustrative purposes only and should not be used in a production environment without modification; it is recommended to adapt and customize the code to meet the specific requirements and needs of your production setup.

What it means 🤔?

• Stack Overflow and OpenAI have partnered to help developers by combining Stack Overflow's coding expertise with OpenAI's advanced AI models. Here read coding expertise as the questions and answers contributed to Stack Overflow by developers/ programmers from all over the world with their free will.

• These contributions are from a variety of languages and frameworks. Developers have been contributing for more than a decade, were not aware that this data sharing partnership would ever happen.

Community contributing developers might have surprised and startled due to sharing of their hard work to third party without consent of the original authors of the questions or answers.

In retaliation, what developers did 😐?

• Developers started revolting against Stack Overflow. Some of them seems not happy with it.

• Deleted their highest rated answers from Stack Overflow 😱.

• Deleted their profile from Stack Overflow.

• There is an Q and A on Stack Overflow meta regarding OpenAI partnership that question has 877 downvotes as of 18th May 2024 🤯. This is the lowest rated answer I have ever seen on the Stack Overflow. Link here: https://meta.stackexchange.com/questions/399619/our-partnership-with-openai

• Screenshot of above link:

• One of the concern developers have is attribution, as most LLM's do not provide attribution to any of the answer they provide to the user.

• Stack Overflow started banning / suspending account of such community contributors 🥲. link1, link2, link3, link4.

What could be the possible future of the Stack Overflow 🤓?

• There were so many user communities past which ended due to lack of any contribution.

• In developer communities, the typical ways of contributing is via documentation, questions, answers, tutorials, notes.

• Will something like below happen with Stack Overflow community in future? What are your thoughts on it ?

• Whatever written in below flow diagram could be just speculation, it might even never happen.

Prerequisite:

1. Apple Mac machine 💻 / 🖥️

2. Parallels Desktop (non-pro version will also work)

3. Some free storage on your SSD

Steps 🛠️:

1. Create a Linux VM

2. Install prerequisites in the Linux VM

3. Install Kubernetes (Let’s call it k8s henceforth) and related applications for the k8s deployment.

4. Cluster creation

Step 1:

• Download the rocky linux (a clone of RHEL). Here is the download link of rocky 9 for arm64 architecture (i.e. for Apple Silicon).

• Create a linux VM by following this guide: Install Linux virtual machine on a Mac with Apple M-Series chip .

• While installing, please select OS type as Red Hat Enterprise Linux / CentOS Linux. Like this:

  

• Proceed with further process of the OS installation.

Step 2:

• Start the vm and then install prerequisites with below commands

sudo dnf install openssh-server
ip address # note down the ip address

PS: Please use the host IP address noted from above command to ssh the vm. It will be easy to just copy and paste commands from below section.

Step 3:

Install required libraries 👇 , GitHub link 🔗 for commands : k8s-kind-podman-installation-command.sh

Step 4:

• Create cluster through kind

kind create cluster # default cluster is name is kind
kubectl cluster-info # check k8s cluster info
#or
kubectl cluster-info --context kind-kind
podman ps # check podman status

Now you can use this cluster as any other normal testing k8s cluster. 😀😀