Z-Ordering is an optimization technique that can speed up the query by colocating similar records while considering values in multiple columns. Z-Ordering is a way of organizing data in storage based on the values of one or more columns. Z-Ordering maps multidimensional data to one dimension while preserving locality of the data points. This means that rows with similar values for the specified columns are stored close together in the same set of files. This improves the performance of queries that filter on those columns, as they can skip over irrelevant files or data blocks. Z-Ordering also enhances data skipping and caching, as it reduces the number of distinct values per file for the chosen columns1. The other options are incorrect because:

• Option B: Bin-packing is an optimization technique that compacts small files into larger ones, but does not colocate similar records based on multiple columns. Bin-packing can improve the performance of queries by reducing the number of files that need to be read, but it does not affect the data layout within the files2.
• Option C: Writing as a Parquet file is not an optimization technique, but a file format choice. Parquet is a columnar storage format that supports efficient compression and encoding schemes. Parquet can improve the performance of queries by reducing the storage footprint and the amount of data transferred, but it does not colocate similar records based on multiple columns3.
• Option D: Data skipping is an optimization technique that skips over files or data blocks that do not match the query predicates, but does not colocate similar records based on multiple columns. Data skipping can improve the performance of queries by avoiding unnecessary data scans, but it depends on the data layout and the metadata collected for each file4.
• Option E: Tuning the file size is an optimization technique that adjusts the size of the data files to a target value, but does not colocate similar records based on multiple columns. Tuning the file size can improve the performance of queries by balancing the trade-off between parallelism and overhead, but it does not affectthe data layout within the files5. References: Z-Ordering (multi-dimensional clustering), Compaction (bin-packing), Parquet, Data skipping, Tuning file sizes

To log single values like the number of trees in a random forest, you can use the mlflow.log_param function. This function allows you to log a parameter (e.g. model hyperparameter) under the current run. Parameters can be of any type, and can be logged using the following syntax:

Python

mlflow.log_param("num_trees", 100)

AI-generated code. Review and use carefully. More info on FAQ.

MLflow also offers a convenient way to log multiple parameters by indicating all of them using a dictionary1. The other options are incorrect because:

• Option A: mlflow.log_artifact is used to log output files in any format, not single values2.
• Option B: mlflow.log_model is used to log an MLflow Model along with its artifacts, not single values3.
• Option C: mlflow.log_metric is used to log numeric metrics, not single values4.
• Option E: There is a way to store values like this using the mlflow.log_param function1. References: mlflow.log_param, mlflow.log_artifact, mlflow.log_model, mlflow.log_metric

 To read data from a stream source, such as Kafka, socket, or rate, the spark.readStream method should be used instead of spark.read. The spark.readStream method returns a streaming DataFrame that represents the unbounded input data stream. The spark.readStream method supports the same options and formats as the spark.read method, such as schema, delta, csv, json, etc. The spark.readStream method can also read from a Delta table as a stream source, by specifying the format("delta") and the path or table name of the Delta table123

The other options are incorrect because:

• A. Replacing “inference” with the path to the location of the Delta table does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the path or table name of the Delta table can be specified as an option or argument.
• B. Replacing schema(schema) with option("maxFilesPerTrigger", 1) does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the schema can be specified as an option or argument. The option("maxFilesPerTrigger", 1) is an optional configuration that limits the number of files processed in each trigger for file-based stream sources, such as delta, csv, json, etc. It does not affect the reading of data from a stream source4
• D. Replacing format("delta") with format("stream") does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the format can be specified as an option or argument. The format("stream") is not a valid format for reading data from a stream source. The supported formats are delta, kafka, socket, rate, etc1
• E. Replacing predict with a stream-friendly prediction function does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the prediction function can be applied to the streaming DataFrame as usual. The predict function does not need to be changed, as long as it can accept a streaming DataFrame as input and return a column of predictions as output5

References:

• Input Sources - Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Structured Streaming + Delta Lake - Databricks
• Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Configuration - Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Machine Learning with Structured Streaming - Databricks

 The mlflow.register_model function takes two arguments: model_uri and name. The model_uri is the URI of the model that was logged to MLflow, which can be obtained from the best run object. The name is the name of the registered model in the MLflow Model Registry. Therefore, the correct line of code to register the model is:

mlflow.register_model(model_uri, “best_model”)

This will create a new registered model with the name “best_model” and register the model version from the best run as the first version of that model.

References:

• [mlflow.register_model — MLflow 1.22.0 documentation]
• [MLflow Model Registry — Databricks Documentation]
• [Manage MLflow Models — Databricks Documentation] Message has links.

The fs.read_table operation can be used to return a Spark DataFrame of a data set associated with a Feature Store table. This operation takes the name of the Feature Store table and an optional time travel specification as arguments. The fs.create_table operation is used to create a new Feature Store table from a Spark DataFrame. The fs.write_table operation is used to write data to an existing Feature Store table. The fs.get_table operation is used to get the metadata of a Feature Store table, not the data itself. There is a way to accomplish this task with fs, so option D is incorrect. References:

• Feature Store Client
• Feature Store Tables

Containers are a type of software packaging that can assist in real-time deployments by packaging software with its own application, tools, and libraries. Containers are isolated and portable environments that run on a shared operating system, but have their ownfile system, dependencies, configuration, and resources. Containers can help developers and operators to build, test, and deploy software faster, easier, and more reliably, as they ensure that the software runs the same way in any environment, whether it is a developer’s laptop, a test server, or a production cluster. Containers can also help to achieve scalability, high availability, and load balancing, as they can be easily created, destroyed, replicated, and orchestrated using tools such as Kubernetes, Docker Swarm, or Amazon ECS. Containers can also integrate with other tools and services, such as MLflow, Databricks, AWS CodeDeploy, and Google Cloud Deployment Manager, to enable end-to-end machine learning pipelines with tracking, reproducibility, and governance1234 References:

• How to Package Your Application’s Infrastructure with Docker
• 21 open source DevOps tools (and what they do)
• Microservices [What is it & Top Benefits]
• 9 Best Software Deployment Tools for 2023 (Paid & Free)

To compute predictions for spark_df when the missing feature values can be found in the Feature Store by searching for features by customer_id, you can use the following code block:

Python

# Get the missing features from the Feature Store using the model URI and the customer_id column

df = fs.get_missing_features(spark_df, model_uri, lookup_key="customer_id")

# Score the DataFrame using the model URI and the Feature Store Client

fs.score_batch(model_uri, df)

AI-generated code. Review and use carefully. More info on FAQ.

The fs.get_missing_features method takes a Spark DataFrame, a model URI, and a lookup key as arguments. It returns a new Spark DataFrame that contains the originalcolumns plus the missing features that are required by the model. The missing features are retrieved from the Feature Store by joining the DataFrame with the feature tables using the lookup key. The lookup key must match the primary key of the feature tables. The model URI must point to a registered model that was trained using features from the Feature Store1.

The fs.score_batch method takes a model URI and a Spark DataFrame as arguments. It applies the model to the DataFrame and returns a new Spark DataFrame that contains the original columns plus a prediction column. The model URI must point to a registered model that was trained using features from the Feature Store2.

The other options are incorrect because:

• Option A: fs.score_model is not a valid method name, as it is missing an underscore. The correct method name is fs.score_batch2.
• Option B: fs.score_model without getting the missing features will not work, as the model expects the DataFrame to have all the features that were used for training. The correct way is to use fs.get_missing_features before fs.score_batch12.
• Option D: fs.score_batch without getting the missing features will not work, as the model expects the DataFrame to have all the features that were used for training. The correct way is to use fs.get_missing_features before fs.score_batch12.
• Option E: fs.score_batch without specifying the lookup key will not work, as the fs.get_missing_features method requires a lookup key to join the DataFrame with the feature tables. The correct way is to use fs.get_missing_features with the lookup key “customer_id” before fs.score_batch12. References: Get missing features, Score batch

The correct way to log a scikit-learn model using MLflow is to use the mlflow.sklearn.log_model function, which logs the model as an MLflow artifact for the current run1. This function takes the scikit-learn model object and an artifact path as required arguments, and optionally accepts other arguments such as conda environment, code paths, serialization format, etc2. The other options are incorrect because they either use the wrong flavor (spark instead of sklearn), the wrong function (load_model or track_model instead of log_model), or do not exist (track_model is not a valid MLflow function). References: MLflow Models, mlflow.sklearn

MLflow Autologging is a feature that automatically logs parameters, metrics, and artifacts for machine learning frameworks, such as Hyperopt, without requiring any code changes. When using MLflow Autologging with Hyperopt, MLflow will automatically create a parent run for the hyperparameter tuning process and a child run for each unique combination of hyperparameter values. This way, the user can easily compare and visualize the results of different hyperparameter settings in the MLflow UI. MLflow Autologging also supports nested runs, which means that if the user starts a manual run before calling fmin, the parent run will be nested under the manual run. This allows the user to organize and group related runs together. References: MLflow Autologging, Hyperparameter Tuning with MLflow, Apache Spark MLlib and Hyperopt, Using MLFlow with Hyperopt for Automated Machine Learning, Parallelize hyperparameter tuning with scikit-learn and MLflow, [Optimizing Model Performance with MLflow and Hyperopt: From Hyperparameter Tuning to Serving]

To log feature importance data from a CSV file as an artifact of an MLflow run, the correct method is mlflow.log_artifact, which takes the local file path and the artifact path as arguments.The mlflow.log_model method is used to log a model with a specific flavor, such as pyfunc or spark. The mlflow.log_data method does not exist. The mlflow.log_model_and_data method also does not exist, and it is not necessary to log both the model and the data in this case. References:

• MLflow Tracking API
• MLflow Models

Feature drift is a change in the distribution of the input data over time, which can affect the performance and accuracy of the model. Monitoring feature drift is important to ensure that the model is still valid and reliable for the current data. One simple and low-cost method of monitoring numeric feature drift is to track the summary statistics trends of the input features, such as mean, standard deviation, minimum, maximum, etc. These statistics can be computed easily and efficiently, and can provide a quick overview of the changes in the data distribution. If the summary statistics deviate significantly from the baseline values, it may indicate a feature drift. However, summary statistics trends may not capture all the nuances of the data distribution, such as outliers, multimodality, skewness, etc. Therefore, other methods, such as statistical tests or visualizations, may be needed to complement the summary statistics trends and provide a more comprehensive analysis of the feature drift123 References:

• Monitoring Feature Drift - Databricks
• Monitor feature attribution skew and drift | Vertex AI | Google Cloud
• Monitoring Model Drift - 6 Different Methodologies - Qualdo

The log_metric function is used to log a numeric metric value to the current MLflow run. It takes the name of the metric and the value as arguments. In this case, the name of the metric is “rmse” and the value is the variable rmse. The log_metric function can be used multiple times to log different metrics or update the same metric. The log_artifact function is used to log a file or directory as an artifact to the current MLflow run. The log_model function is used to log a model with a specified flavor to the current MLflow run. The log_param function is used to log a key-value pair as a parameter to the current MLflow run. There is a way to store values like this using the log_metric function, so option E is incorrect.https://docs.databricks.com/

To view the data visualizations that are logged with each run, you can go to the Artifacts section of the MLflow Run page in Databricks. The Artifacts section shows the files and directories that are logged as artifacts for a run. You can browse the artifact hierarchy and preview the files, such as images, text, or HTML1. You can also download the artifacts or copy their URIs for further use2. The other options are incorrect because:

• Option A: The MLflow Model Registry Model page shows the information and metadata of a registered model, such as its name, description, versions, stages, and lineage. It does not show the data visualizations that are logged with each run3.
• Option B: The Artifacts section of the MLflow Experiment page shows the artifacts that are logged for an experiment, not for a specific run. It does not allow you to preview the files or browse the artifact hierarchy4.
• Option C: Logged data visualizations can be viewed in Databricks using the Artifacts section of the MLflow Run page1.
• Option E: There is no Figures section of the MLflow Run page in Databricks. The Figures section is only available in the open source MLflow UI, which shows the plots that are logged as figures for a run5. References: View run artifacts, Log, list, and download artifacts, Manage models, View experiment artifacts, Logging Visualizations with MLflow

The client.delete_registered_model method of the MlflowClient class can be used to delete a model from the MLflow Model Registry. The method takes the following parameter:

• name: The name of the registered model to delete.

The method deletes the registered model and all its versions from the Model Registry. The method does not return any value1

The other options are incorrect because:

• A. The client.transition_model_version_stage method can be used to transition a model version to a different stage, such as None, Staging, Production, or Archived. The method does not delete the model or the model version from the Model Registry2
• B. The client.delete_model_version method can be used to delete a specific model version from the Model Registry. The method does not delete the registered model or all its versions from the Model Registry3
• C. The client.update_registered_model method can be used to update the metadata of a registered model, such as its name or description. The method does not delete the model or its versions from the Model Registry4
• D. There is no method called client.delete_model. To delete a model from the Model Registry, the client.delete_registered_model method should be used.

References:

• mlflow.tracking.client.MlflowClient.delete_registered_model — MLflow 2.9.1 documentation
• mlflow.tracking.client.MlflowClient.transition_model_version_stage — MLflow 2.9.1 documentation
• mlflow.tracking.client.MlflowClient.delete_model_version — MLflow 2.9.1 documentation
• mlflow.tracking.client.MlflowClient.update_registered_model — MLflow 2.9.1 documentation

The DESCRIBE HISTORY command can be used to view the commit history of a Delta table, including the schema changes, operations, and timestamps. This command can help identify when a column was dropped from the feature set and by which operation. The other commands are either invalid or do not provide the required information. References:

• Delta Lake - View Commit History
• Databricks Certified Machine Learning Professional Exam Guide - Section 1: Experimentation - Data Management