Skip to main content

Azure Data Lake: Things to Know Before You Dive Deep

It is almost just less than five days ahead, which I’ve to do a session on Azure Data Lake Analytics at a Big Data Event. When I started to grab the stuff on Azure Data Lake Analytics I could quickly learn few solid points I thought how great if I could share those via a blog post or a video with you all.  Because I observed there are many people get it hard and introduce some complexity when start to work with the Azure Data Lake Analytics.

Why Azure Data Lake?

Azure Data Lake is cloud-based, distributed processing architecture. Azure Data Lake mainly consist of two components. Azure Data Lake Analytics and the Azure Data Lake Store.
Azure Data Lake Analytics: On-demand analytics job service. Rather deploying, configuring and tuning hardware you can more focus on the business need and write queries to generate more valuable insights.
Azure Data Lake Store: Web HDFS storage system. Which store your Big Data like structured, semi-structured and unstructured data
Azure Data Lake high-level architecture. Image source: MSDN
ADL Store is an Apache Hadoop file system which is fully compatible and accessed by HDFS. Similar to Hadoop, distributed architecture, Parallelism and ability to handle petabytes of workloads is there. We can use ADL to store a variety of data types including relational data, streaming data, media files etc which is similar to Hadoop Big Data ecosystem. Good point in here is you only need to deal with one language called U-SQL for all of your Big data management and analytics. In Hadoop, you need to master more than five languages(Java, Pig, Hive, Scala, Python, YARN etc). So, as I developer I know, learning all these and contribute to each and every phase will be really challenging.

U-SQL is built using the syntax of T-SQL and C# derivatives.  So, even you are a beginner to C#, since almost all of you have written at least a single SQL statement in your life, learning U-SQL will not be a challenge.

Does ADLA is the future of Data Warehousing?

Some of you may have a doubt like does this replace the traditional on-premise relational data warehouse or is this the future of data warehouse development. If I answer myself for this question, frankly I do have a mixed feeling on this.

You can replace your traditional data warehouse, (all the automation and Job scheduling capability is there via U-SQL) But, it does not mean that it has to be done in this manner always. It is always an option in Azure Data Lake Store to have Catalog databases and in those databases, you can have external reference tables which refer to Azure SQL database or Azure Data Warehouse. This is an indication, you can still design your Azure Data Lake solution which will be the ultimate destination for your all the data in the organization, including the existing data warehouse solution.

You can store all kind of data in the Native format in your Azure Data Lake solution. Refer the image below.
Image source: https://www.realworldanalytics.com

How Much It Does Cost for Process my Big Data in ADL?

The pricing model for Azure Data Lake Analytics is attractive. You can choose either pay-per job which only has to pay for the processing power you used or On-demand cluster. You can scale up or scale down dynamically at any time as per your business need. Nothing more you have to pay in this scenario. 

If I’m new to Azure Data Lake Analytics Where Do I Start?

There is no robust and accurate reference to learn than the documentation itself by Microsoft.
Also, you can start to learn about Azure Data Lake Analytics from Edx
Labels:

Comments

  1. TejutejuJune 12, 2018 at 6:30 PM

    Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking.Big data hadoop online training Bangalore

    ReplyDelete
  2. prince aroraJune 4, 2019 at 4:12 AM



    You have discussed an interesting topic that everybody should know. Very well explained with examples. I have found a similar website
    data analytics consulting     visit the site to know more about Omdata.

    ReplyDelete

Post a Comment

Popular posts from this blog

Run T-SQL Script Files Using Command Line

Most of the time when we need to execute a SQL script or statement, then we go for SQL Server Management Studio to execute our T-SQL scripts. But the same result we can achieve using Command Prompt. This is very simple way to do that just a matter of write a command and pass few arguments. You can see the process status like as in when you execute a query in Management studio. Use-case : There may be having some situations you won't be able to execute T-SQL script using Management Studio. Most of the time when doing data population. In my case, I had a situation I needed to populate a Database without using backup/restore. Once I generated script with all the schemas and related data it took around 400+Mb sql file.  SQL Server Management Studio will not allow you to run the script unless you execute the query in high-end box. Because, when the file loaded to the memory it will too hard to cater our requirement. You may probably get  memory ...
Post a Comment
Read more

How to Create a Date Table in Few Steps Using Power BI

Calendar Dimension or Calendar Table is one of the crucial table in a Power BI model. I never have seen any data warehouse data model which does not have a Calendar table so far in my development. Because, when it comes to data warehousing or dimension modeling you may storing various business processes or events as Facts. So those events anyway occur in particular date or time. So simply there wouldn’t be a Power BI data model without a date table. In this post, I’m going to share with you how to create a date table within few steps. Actually, there are two DAX measures which we can use to create a date table. Calendar(DAX) and CalendarAuto(Auto). Calendar(DAX) = You can pass start date and End date as parameters you need to create date table. CalendarAuto(DAX) = You can use this function without passing any parameter. Then it will generate the dates based on your data model dates.  You can simply copy the below DAX code and paste in your Power BI Desktop DAX e...
7 comments
Read more

How to Visualize Images Dynamically using Microsoft Power BI

One of the challenging part of the Power BI report development is handling dynamic images. We can add a static image within few steps and it is pretty straightforward. But, when it comes to dynamic images, available options are countable. Available options: Upload to the Web : You can upload the images to the web and use the image URLs as a reference. But, you need to make sure the data category  of your column where to store the image URL make it as  I mage URL. When we are dealing with clients data we cannot upload it to the public web. So, need to be accountable for data we are dealing with. Azure Blob Storage : If the security is the concern, you can use Azure blog storage to store image files. Only accessible who are in the same domain or security group. You have the full control over managing the authorization. I have an experience of, there are clients they never want to upload any data to the cloud. And if we are dealing with multiple clie...
49 comments
Read more