How to Write Streaming Data into a Data Table in Databricks

Azure Databricks is the latest way of doing Data Engineering and Data Science workload in Microsoft space. If you are new to Azure Databricks, and wondering what is it and finding ways of how to get started with it, I would like to refer my Jump Start Series of articles. 😊

If someone asked me, why databricks or Apache Spark is so special, my first answer would be "it is good at with to do real-time streaming processing apart from the batch processing". Also, no matter whether you are a good Java Programmer, very handy Python developer, an experienced Data Scientist with expertise with R, or a Data Engineer who born with SQL. Databricks has designed for all of you. Irrespective to the programming language, you can perform your workload without hassle. 

In this blog post, I'm going to explain to you how to write streaming data frame into a Databricks Data Table.

You may have your processing data in Azure Databricks using dataframes either Batches or Streams. Each way the Read and Write methods are different. You can have a detailed understanding of it if you are looking into the Azure Databricks documentation. In this post, I'm only focused on Streaming Writes to data tables in Databricks.  There are two steps you need to perform in order to write streaming data successfully. 

1. Write Streaming DataFrame into Parquet file format
2. Read Parquet files and Write into Databricks table

Parquet is a columnar format that is supported by many other data processing systems including Apache Spark. Read more from here

In my demo example, I've written the sentiment score of Twitter feeds into a databricks table. 

In this code instead of display the result in the console I'm writing to the Databricks table. In here you don't need to specifically create any table to store data manually. The code does create the table. 

Once this executed you can see the table has been created in the data tab. 

You can click the table and view the Data actually written. 

Further learning:
This parquet files and the checkpoints consume mnt directory in the DBFS (Data Bricks File System). You can view the DBFS by goto the Databricks home-> Upload data-> DBFS

You can see a set of parquet files written inside TwitterSentiment folder. In the checkpoint location also created a set of files. See the image below. Checkpointing is used as Production quality streaming implementation for fault tolerant. In order to read more on checkpointing here

Hope you learned how to write Streaming data into Azure Databricks tables in Spark Cluster. If you find this article as interesting, subscribe my blog to get more articles like this. Also, if you have any issues or would like to give any feedback please leave a comment below. Cheers! 😀