[Jan 24, 2023] Ultimate Databricks-Certified-Professional-Data-Engineer Guide to Prepare Free Latest Databricks Practice Tests Dumps
Get Top-Rated Databricks Databricks-Certified-Professional-Data-Engineer Exam Dumps Now
NEW QUESTION 23
What is the output of the below function when executed with input parameters 1, 3 :
1.def check_input(x,y):
2. if x < y:
3. x= x+1
4. if x<y:
5. x= x+1
6. if x <y:
7. x = x+1
8. return x
check_input(1,3)
- A. 3
(Correct) - B. 0
- C. 1
- D. 2
- E. 3
Answer: A
NEW QUESTION 24
At the end of the inventory process, a file gets uploaded to the cloud object storage, you are asked to build a process to ingest data which of the following method can be used to ingest the data in-crementally, schema of the file is expected to change overtime ingestion process should be able to handle these changes automatically.
Below is the auto loader to command to load the data, fill in the blanks for successful execution of below code.
1.spark.readStream
2..format("cloudfiles")
3..option("_______","csv)
4..option("_______", 'dbfs:/location/checkpoint/')
5..load(data_source)
6..writeStream
7..option("_______",' dbfs:/location/checkpoint/')
8..option("_______", "true")
9..table(table_name))
- A. format, checkpointlocation, schemalocation, overwrite
- B. cloudfiles.format, cloudfiles.schemalocation, checkpointlocation, mergeSchema
- C. cloudfiles.format, checkpointlocation, cloudfiles.schemalocation, overwrite
- D. cloudfiles.format, cloudfiles.schemalocation, checkpointlocation, append
- E. cloudfiles.format, cloudfiles.schemalocation, checkpointlocation, overwrite
Answer: B
Explanation:
Explanation
The answer is cloudfiles.format, cloudfiles.schemalocation, checkpointlocation, mergeSchema.
Here is the end to end syntax of streaming ELT, below link contains complete options Auto Loader options | Databricks on AWS
1.spark.readStream
2..format("cloudfiles") # Returns a stream data source, reads data as it arrives based on the trigger.
3..option("cloudfiles.format","csv") # Format of the incoming files
4..option("cloudfiles.schemalocation", "dbfs:/location/checkpoint/") The location to store the inferred schema and subsequent changes
5..load(data_source)
6..writeStream
7..option("checkpointlocation","dbfs:/location/checkpoint/") # The location of the stream's checkpoint
8..option("mergeSchema", "true") # Infer the schema across multiple files and to merge the schema of each file. Enabled by default for Auto Loader when inferring the schema.
9..table(table_name)) # target table
NEW QUESTION 25
Which of the following table constraints that can be enforced on Delta lake tables are supported?
- A. Unique, Not Null, Check Constraints
- B. Primary key, Not Null, Check Constraints
- C. Primary key, foreign key, Not Null, Check Constraints
- D. Default, Not Null, Check Constraints
- E. Not Null, Check Constraints
Answer: E
Explanation:
Explanation
The answer is Not Null, Check Constraints
https://docs.microsoft.com/en-us/azure/databricks/delta/delta-constraints
* CREATE TABLE events( id LONG,
* date STRING,
* location STRING,
* description STRING
* ) USING DELTA;
ALTER TABLE events CHANGE COLUMN id SET NOT NULL;
ALTER TABLE events ADD CONSTRAINT dateWithinRange CHECK (date > '1900-01-01'); Note: Databricks as of DBR 11.1 added support for Primary Key and Foreign Key when Unity Catalog is enabled but this is for information purposes only these are not actually enforced. You may ask then why are we defining these if they are not enforced, so especially these information constraints are very helpful if you have a BI tool that can benefit from knowing the relationship between the tables, so it will be easy when creating reports/dashboards or understanding the data model when using any Data modeling tool.
Primary and Foreign Key
Graphical user interface, text, application, email Description automatically generated
NEW QUESTION 26
Your team has hundreds of jobs running but it is difficult to track cost of each job run, you are asked to provide a recommendation on how to monitor and track cost across various workloads
- A. Use a single cluster for all the jobs, so cost can be easily tracked
- B. Use workspace admin reporting
- C. Use job logs to monitor and track the costs
- D. Create jobs in different workspaces, so we can track the cost easily
- E. Use Tags, during job creation so cost can be easily tracked
Answer: E
Explanation:
Explanation
The answer is Use Tags, during job creation so cost can be easily tracked Review below link for more details
https://docs.databricks.com/administration-guide/account-settings/usage-detail-tags-aws.html Here is a view how tags get propagated from pools to clusters and clusters without pools, Diagram Description automatically generated
NEW QUESTION 27
You were asked to write python code to stop all running streams, which of the following command can be used to get a list of all active streams currently running so we can stop them, fill in the blank.
1.for s in _______________:
2. s.stop()
- A. activeStreams()
- B. Spark.getActiveStreams()
- C. spark.streams.active
- D. spark.streams.getActive
- E. getActiveStreams()
Answer: C
NEW QUESTION 28
Which of the following statements are true about a lakehouse?
- A. Lakehouse do not support SQL
- B. Lakehouse does not support ACID
- C. Lakehouse supports Transactions
- D. Lakehouse only supports end-to-end streaming workloads and Data warehouses support Batch workloads
- E. Lakehouse only supports Machine learning workloads and Data warehouses support BI workloads
Answer: C
Explanation:
Explanation
What Is a Lakehouse? - The Databricks Blog
Text Description automatically generated
NEW QUESTION 29
Which of the following benefits does Delta Live Tables provide for ELT pipelines over standard data pipelines
that utilize Spark and Delta Lake on Databricks?
- A. The ability to declare and maintain data table dependencies
- B. The ability to perform batch and streaming queries
- C. The ability to automatically scale compute resources
- D. The ability to write pipelines in Python and/or SQL
- E. The ability to access previous versions of data tables
Answer: A
NEW QUESTION 30
A DELTA LIVE TABLE pipelines can be scheduled to run in two different modes, what are these two different modes?
- A. Once, Continuous
- B. Triggered, Incremental
- C. Triggered, Continuous
- D. Once, Incremental
- E. Continuous, Incremental
Answer: C
Explanation:
Explanation
The answer is Triggered, Continuous
https://docs.microsoft.com/en-us/azure/databricks/data-engineering/delta-live-tables/delta-live-tables-concepts#-
*Triggered pipelines update each table with whatever data is currently available and then stop the cluster running the pipeline. Delta Live Tables automatically analyzes the dependencies between your tables and starts by computing those that read from external sources. Tables within the pipeline are updated after their dependent data sources have been updated.
*Continuous pipelines update tables continuously as input data changes. Once an update is started, it continues to run until manually stopped. Continuous pipelines require an always-running cluster but ensure that downstream consumers have the most up-to-date data.
NEW QUESTION 31
A dataset has been defined using Delta Live Tables and includes an expectations clause: CON-STRAINT valid_timestamp EXPECT (timestamp > '2020-01-01') What is the expected behavior when a batch of data containing data that violates these constraints is processed?
- A. Records that violate the expectation are added to the target dataset and recorded as invalid in the event log.
- B. Records that violate the expectation are dropped from the target dataset and loaded into a quarantine table.
- C. Records that violate the expectation are added to the target dataset and flagged as in-valid in a field added to the target dataset.
- D. Records that violate the expectation cause the job to fail.
- E. Records that violate the expectation are dropped from the target dataset and recorded as invalid in the event log.
Answer: A
Explanation:
Explanation
The answer is, Records that violate the expectation are added to the target dataset and recorded as invalid in the event log.
Delta live tables support three types of expectations to fix bad data in DLT pipelines Review below example code to examine these expectations, Diagram Description automatically generated with medium confidence
NEW QUESTION 32
One of the queries in the Databricks SQL Dashboard takes a long time to refresh, which of the be-low steps can be taken to identify the root cause of this issue?
- A. Restart the SQL endpoint
- B. Run optimize and Z ordering
- C. Select the SQL endpoint cluster, spark UI, SQL tab to see the execution plan and time spent in each step
- D. Change the Spot Instance Policy from "Cost optimized" to "Reliability Optimized."
- E. Use Query History, to view queries and select query, and check query profile to time spent in each step
Answer: E
Explanation:
Explanation
The answer is, Use Query History, to view queries and select query, and check the query profile to see time spent in each step.
Here is the view of the query profile, for more info use the link
https://docs.microsoft.com/en-us/azure/databricks/sql/admin/query-profile As you can see here Databricks SQL query profile is much different to Spark UI and provides much more clear information on how time is being spent on different queries and time it spent on each step.
Graphical user interface, application Description automatically generated
NEW QUESTION 33
Which of the following locations in the Databricks product architecture hosts the notebooks and jobs?
- A. Databricks Filesystem
- B. Control plane
- C. JDBC data source
- D. Data plane
- E. Databricks web application
Answer: B
Explanation:
Explanation
The answer is Control Pane,
Databricks operates most of its services out of a control plane and a data plane, please note serverless features like SQL Endpoint and DLT compute use shared compute in Control pane.
Control Plane: Stored in Databricks Cloud Account
*The control plane includes the backend services that Databricks manages in its own Azure account. Notebook commands and many other workspace configurations are stored in the control plane and encrypted at rest.
Data Plane: Stored in Customer Cloud Account
*The data plane is managed by your Azure account and is where your data resides. This is also where data is processed. You can use Azure Databricks connectors so that your clusters can connect to external data sources outside of your Azure account to ingest data or for storage.
Timeline Description automatically generated
NEW QUESTION 34
The below spark command is looking to create a summary table based customerId and the number of times the customerId is present in the event_log delta table and write a one-time micro-batch to a summary table, fill in the blanks to complete the query.
1.spark._________
2. .format("delta")
3. .table("events_log")
4. .groupBy("customerId")
5. .count()
6. ._______
7. .format("delta")
8. .outputMode("complete")
9. .option("checkpointLocation", "/tmp/delta/eventsByCustomer/_checkpoints/")
10. .trigger(______)
11. .table("target_table")
- A. readStream, writeStream, once = True
- B. writeStream, readStream, once = True
- C. writeStream, readStream, once
- D. writeStream, processingTime = once
- E. readStream, writeStream, once
Answer: A
Explanation:
Explanation
The answer is readStream, writeStream, once = True.
spark.readStream
format("delta")
table("events_log")
groupBy("customerId")
count()
writeStream
format("delta")
outputMode("complete")
option("checkpointLocation", "/tmp/delta/eventsByCustomer/_checkpoints/") trigger(once = True) table("target_table")
NEW QUESTION 35
The marketing team is launching a new campaign to monitor the performance of the new campaign for the first two weeks, they would like to set up a dashboard with a refresh schedule to run every 5 minutes, which of the below steps can be taken to reduce of the cost of this refresh over time?
- A. Setup the dashboard refresh schedule to end in two weeks
- B. Reduce the max size of auto scaling from 10 to 5
- C. Change the spot instance policy from reliability optimized to cost optimized
- D. Reduce the size of the SQL Cluster size
- E. Always use X-small cluster
Answer: A
Explanation:
Explanation
The answer is Setup the dashboard refresh schedule to end in two weeks
NEW QUESTION 36
You are tasked to set up a set notebook as a job for six departments and each department can run the task parallelly, the notebook takes an input parameter dept number to process the data by department, how do you go about to setup this up in job?
- A. A parameter can only be passed at the job level, create six jobs pass department number to each job with no job dependency
- B. A parameter can only be passed at the job level, create six jobs pass department number to each job with linear job dependency
- C. A task in the job cannot take an input parameter, create six notebooks with hardcoded dept number and setup six tasks with linear dependency in the job
- D. Use a single notebook as task in the job and use dbutils.notebook.run to run each note-book with parameter in a different cell
- E. A task accepts key-value pair parameters, creates six tasks pass department number as parameter foreach task with no dependency in the job as they can all run in parallel.
(Correct)
Answer: E
Explanation:
Explanation
Here is how you setup
Create a single job and six tasks with the same notebook and assign a different parameter for each task , Graphical user interface, text, application, email Description automatically generated
All tasks are added in a single job and can run parallel either using single shared cluster or with individual clusters.
Graphical user interface, application, Teams Description automatically generated
NEW QUESTION 37
One of the team members Steve who has the ability to create views, created a new view called re-gional_sales_vw on the existing table called sales which is owned by John, and the second team member Kevin who works with regional sales managers wanted to query the data in region-al_sales_vw, so Steve granted the permission to Kevin using command GRANT VIEW, USAGE ON regional_sales_vw to [email protected] but Kevin is still unable to access the view?
- A. Kevin needs owner access on the view regional_sales_vw
- B. Kevin needs select access on the table sales
- C. Steve is not the owner of the sales table
- D. Kevin is not the owner of the sales table
- E. Table access control is not enabled on the table and view
Answer: C
Explanation:
Explanation
Ownership determines whether or not you can grant privileges on derived objects to other users, since Steve is not the owner of the underlying sales table, he can not grant access to the table or data in the table indirectly.
Only owner(user or group) can grant access to a object
https://docs.microsoft.com/en-us/azure/databricks/security/access-control/table-acls/object-privileges#a-user-has Data object privileges - Azure Databricks | Microsoft Doc
NEW QUESTION 38
You have noticed the Data scientist team is using the notebook versioning feature with git integra-tion, you have recommended them to switch to using Databricks Repos, which of the below reasons could be the reason the why the team needs to switch to Databricks Repos.
- A. Databricks Repos allows merge and conflict resolution
- B. Databricks Repos allows multiple users to make changes
- C. Databricks Repos automatically saves changes
- D. Databricks Repos has a built-in version control system
- E. Databricks Repos allow you to add comments and select the changes you want to commit.
Answer: E
Explanation:
Explanation
The answer is Databricks Repos allow you to add comments and select the changes you want to commit.
NEW QUESTION 39
What is the best way to describe a data lakehouse compared to a data warehouse?
- A. A data lakehouse provides a relational system of data management
- B. A data lakehouse enables both batch and streaming analytics.
- C. A data lakehouse utilizes proprietary storage formats for data.
- D. A data lakehouse captures snapshots of data for version control purposes.
- E. A data lakehouse couples storage and compute for complete control.
Answer: B
Explanation:
Explanation
Anser is A data lakehouse enables both batch and streaming analytics.
A lakehouse has the following key features:
*Transaction support: In an enterprise lakehouse many data pipelines will often be reading and writing data concurrently. Support for ACID transactions ensures consistency as multi-ple parties concurrently read or write data, typically using SQL.
*Schema enforcement and governance: The Lakehouse should have a way to support schema enforcement and evolution, supporting DW schema architectures such as star/snowflake-schemas. The system should be able to reason about data integrity, and it should have robust governance and auditing mechanisms.
*BI support: Lakehouses enable using BI tools directly on the source data. This reduces staleness and improves recency, reduces latency, and lowers the cost of having to operationalize two copies of the data in both a data lake and a warehouse.
*Storage is decoupled from compute: In practice this means storage and compute use sepa-rate clusters, thus these systems are able to scale to many more concurrent users and larger data sizes. Some modern data warehouses also have this property.
*Openness: The storage formats they use are open and standardized, such as Parquet, and they provide an API so a variety of tools and engines, including machine learning and Py-thon/R libraries, can efficiently access the data directly.
*Support for diverse data types ranging from unstructured to structured data: The lakehouse can be used to store, refine, analyze, and access data types needed for many new data applications, including images, video, audio, semi-structured data, and text.
*Support for diverse workloads: including data science, machine learning, and SQL and analytics. Multiple tools might be needed to support all these workloads but they all rely on the same data repository.
*End-to-end streaming: Real-time reports are the norm in many enterprises. Support for streaming eliminates the need for separate systems dedicated to serving real-time data applications.
NEW QUESTION 40
How do you access or use tables in the unity catalog?
- A. catalog_name.table_name
- B. catalog_name.schema_name.table_name
- C. schema_name.catalog_name.table_name
- D. schema_name.table_name
- E. catalog_name.database_name.schema_name.table_name
Answer: B
Explanation:
Explanation
The answer is catalog_name.schema_name.table_name
Graphical user interface, diagram Description automatically generated
Note: Database and Schema are analogous they are interchangeably used in the Unity catalog.
FYI, A catalog is registered under a metastore, by default every workspace has a default metastore called hive_metastore, with a unity catalog you have the ability to create meatstores and share that across multiple workspaces.
Diagram Description automatically generated
NEW QUESTION 41
Drop the customers database and associated tables and data, all of the tables inside the database are managed tables. Which of the following SQL commands will help you accomplish this?
- A. DROP DELTA DATABSE customers
- B. All the tables must be dropped first before dropping database
- C. DROP DATABASE customers CASCADE
- D. DROP DATABASE customers FORCE
- E. DROP DATABASE customers INCLUDE
Answer: E
Explanation:
Explanation
The answer is DROP DATABASE customers CASCADE
Drop database with cascade option drops all the tables, since all of the tables inside the database are managed tables we do not need to perform any additional steps to clean the data in the storage.
NEW QUESTION 42
Which of the following data workloads will utilize a Bronze table as its destination?
- A. A job that queries aggregated data to publish key insights into a dashboard
- B. A job that develops a feature set for a machine learning application
- C. A job that ingests raw data from a streaming source into the Lakehouse
- D. A job that aggregates cleaned data to create standard summary statistics
- E. A job that enriches data by parsing its timestamps into a human-readable format
Answer: C
Explanation:
Explanation
The answer is A job that ingests raw data from a streaming source into the Lakehouse.
The ingested data from the raw streaming data source like Kafka is first stored in the Bronze layer as first destination before it is further optimized and stored in Silver.
Medallion Architecture - Databricks
Bronze Layer:
1. Raw copy of ingested data
2. Replaces traditional data lake
3. Provides efficient storage and querying of full, unprocessed history of data
4. No schema is applied at this layer
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
Purpose of each layer in medallion architecture
NEW QUESTION 43
What is the purpose of gold layer in Multi hop architecture?
- A. Data quality checks and schema enforcement
- B. Eliminate duplicate records
- C. Preserves grain of original data, without any aggregations
- D. Optimized query performance for business-critical data
- E. Optimizes ETL throughput and analytic query performance
Answer: D
Explanation:
Explanation
Medallion Architecture - Databricks
Gold Layer:
1. Powers Ml applications, reporting, dashboards, ad hoc analytics
2. Refined views of data, typically with aggregations
3. Reduces strain on production systems
4. Optimizes query performance for business-critical data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
NEW QUESTION 44
How does Lakehouse replace the dependency on using Data lakes and Data warehouses in a Data and Analytics solution?
- A. All the above
- B. Open, direct access to data stored in standard data formats.
- C. Support for end-to-end streaming and batch workloads
- D. Supports BI and Machine learning workloads
- E. Supports ACID transactions.
Answer: A
Explanation:
Explanation
Lakehouse combines the benefits of a data warehouse and data lakes,
Lakehouse = Data Lake + DataWarehouse
Here are some of the major benefits of a lakehouse
Text, letter Description automatically generated
Lakehouse = Data Lake + DataWarehouse
A picture containing text, blackboard Description automatically generated
NEW QUESTION 45
Question-26. There are 5000 different color balls, out of which 1200 are pink color. What is the maximum
likelihood estimate for the proportion of "pink" items in the test set of color balls?
- A. 2.4
- B. 4.8
- C. .24
- D. .48
- E. 24 0
Answer: C
Explanation:
Explanation
Given no additional information, the MLE for the probability of an item in the test set is exactly its frequency
in the training set. The method of maximum likelihood corresponds to many well-known estimation methods
in statistics. For example, one may be interested in the heights of adult female penguins, but be unable to
measure the height of every single penguin in a population due to cost or time constraints. Assuming that the
heights are normally (Gaussian) distributed with some unknown mean and variance, the mean and variance
can be estimated with MLE while only knowing the heights of some sample of the overall population. MLE
would accomplish this by taking the mean and variance as parameters and finding particular parametric values
that make the observed results the most probable (given the model).
In general, for a fixed set of data and underlying statistical model the method of maximum likelihood selects
the set of values of the model parameters that maximizes the likelihood function. Intuitively, this maximizes
the "agreement" of the selected model with the observed data, and for discrete random variables it indeed
maximizes the probability of the observed data under the resulting distribution. Maximum-likelihood
estimation gives a unified approach to estimation, which is well-defined in the case of the normal distribution
and many other problems. However in some complicated problems, difficulties do occur: in such problems,
maximum-likelihood estimators are unsuitable or do not exist.
NEW QUESTION 46
What is the purpose of a silver layer in Multi hop architecture?
- A. Replaces a traditional data lake
- B. Optimized query performance for business-critical data
- C. Refined views with aggregated data
- D. Efficient storage and querying of full and unprocessed history of data
- E. A schema is enforced, with data quality checks.
Answer: E
Explanation:
Explanation
The answer is, A schema is enforced, with data quality checks.
Medallion Architecture - Databricks
Silver Layer:
1.Reduces data storage complexity, latency, and redundency
2.Optimizes ETL throughput and analytic query performance
3.Preserves grain of original data (without aggregation)
4.Eliminates duplicate records
5.production schema enforced
6.Data quality checks, quarantine corrupt data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
NEW QUESTION 47
What is the purpose of the silver layer in a Multi hop architecture?
- A. Replaces a traditional data lake
- B. Eliminates duplicate data, quarantines bad data
- C. Optimized query performance for business-critical data
- D. Refined views with aggregated data
- E. Efficient storage and querying of full, unprocessed history of data
Answer: B
Explanation:
Explanation
Medallion Architecture - Databricks
Silver Layer:
1. Reduces data storage complexity, latency, and redundency
2. Optimizes ETL throughput and analytic query performance
3. Preserves grain of original data (without aggregation)
4. Eliminates duplicate records
5. production schema enforced
6. Data quality checks, quarantine corrupt data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
A diagram of a house Description automatically generated with low confidence
NEW QUESTION 48
......
Passing Key To Getting Databricks-Certified-Professional-Data-Engineer Certified Exam Engine PDF: https://www.itpass4sure.com/Databricks-Certified-Professional-Data-Engineer-practice-exam.html
