Видео

Is there a Part 2? Great video 🤘

More to come!

I added the GitHub link to the description

Can you reach out to your account team? I don't know whats going on.

@@stephanieamrivera Thanks for replying. The issue was resolved actually. In shared mode, it does not support some of the APIs and spark context according to the documentation. So we used single-user, multi-node cluster and it's all working fine. Thanks.

To configure Service Principal Names (SPN) for accessing resources in Azure Databricks, you typically need to define the SPN as an admin in Databricks, but it isn't required to obtain an access token. Here's the typical flow to use an SPN to access data in Databricks: 1. Create an SPN: Firstly, you need to create an SPN in Azure Active Directory (AAD) and provide the necessary permissions for accessing the resources you require. 2. Assign Permissions: Assign the appropriate permissions to the SPN, such as the necessary roles or access policies to access the Databricks workspace and other resources. 3. Configure Databricks: As an admin in Databricks, you will configure the SPN by creating a secret scope and storing the SPN credentials securely. This can be done using the Databricks CLI or the Databricks UI. 4. Access Tokens: To obtain an access token for the SPN, you can use the Azure Active Directory authentication flow, such as OAuth2 or client credentials, to authenticate and generate the access token. This token will be used to authenticate the SPN when accessing the Databricks resources.

Is this what you were looking for? databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/8599738367597028/2070341989008551/3601578643761083/latest.html.

The Customer VPC NACL is a security feature in AWS that functions as a virtual firewall for controlling inbound and outbound traffic at the subnet level. It resembles a set of rules that determine what traffic is allowed or denied in a VPC. Here are some key aspects and characteristics of the Customer VPC NACL: 1. Associations: A VPC NACL is associated with one or more subnets within a VPC. By default, each subnet in a VPC is associated with the default VPC NACL, but you can associate a custom NACL with your subnets. 2. Numbering: Each VPC NACL rule is assigned a rule number that determines the order in which rules are evaluated. Rule numbers can be either explicit (specified by the user) or implicit (automatically assigned by AWS). 3. Inbound and outbound rules: VPC NACLs have separate sets of rules for inbound and outbound traffic. Inbound rules control incoming traffic to the subnet, while outbound rules control outgoing traffic from the subnet. 4. Allow and deny rules: VPC NACLs can have rules that either allow or deny traffic. The rules are evaluated in order, and the first matching rule determines whether the traffic is allowed or denied. 5. Stateless: VPC NACLs are stateless, which means that responses to allowed inbound traffic are not automatically allowed outbound. Separate rules must be created for inbound and outbound traffic. 6. Default rules: By default, a VPC NACL allows all inbound and outbound traffic. You can modify the default rules to tighten security or create custom rules to fit your specific requirements. 7. Logging: VPC NACLs can be configured to log accepted and denied traffic, which helps in monitoring and analyzing network traffic patterns. It's important to note that the VPC NACL operates at the subnet level and provides a basic level of security. For more granular control and advanced security features, it is recommended to use Network Security Groups (NSGs) in conjunction with VPC NACLs. Reference: AWS Documentation on VPC NACLs - docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html. Hope this helps!

Unfortunately, RUclips doesn't let me upload files or images. It might be easier for you to take screenshots from the video. Sorry about that!

The performance of DLT compared to a regular workflow depends on various factors, including the specific use case, data volume, query patterns, and optimization techniques used. While DLT introduces some overhead due to its real-time change data capture capabilities, the benefits it provides may still make it worth considering, even without utilizing all of its enhanced features. 1. Performance: DLT may have some additional processing overhead compared to regular workflows due to change tracking and maintaining transactional integrity. However, DLT's optimizations, such as indexing, caching, and predicate pushdown, can help mitigate this impact. If you leverage these optimizations effectively, the performance difference might be minimal. 2. Enhanced features: Unity catalog, with its lineage and other capabilities, can provide valuable insights into data connections and lineage. These features can enhance data understanding, data governance, and debugging processes. 3. Optimize and Z-ordering: Delta Lake provides various optimization techniques, including optimizing layout and improving data locality by leveraging Z-ordering. If you can incorporate these optimizations effectively into your regular workflow without using DLT, you can still achieve performance benefits without incurring the additional cost associated with DLT. In summary, using DLT depends on the specific requirements and constraints of your use case.

If you are experiencing issues with the personal access token (PAT) not being recognized or not having the right permissions, there are a few troubleshooting steps you can try: 1. Verify token permissions: Confirm that the PAT has the necessary permissions assigned within the Databricks workspace. Although you mentioned that the PAT has admin permissions, make sure it has the required permissions specifically for the actions you are trying to perform. For example, if you are accessing Delta tables, ensure that the PAT has the necessary permissions for table operations. 2. Check workspace configuration: Verify that token-based authentication is enabled in your Databricks workspace and that there are no restrictions or configurations that could prevent the use of tokens. Contact your workspace administrator to confirm the token settings and make sure there are no conflicts or restrictions. 3. Try with a new token: If all else fails, you can try revoking the existing PAT and generating a new one. Sometimes, there can be issues with specific tokens, so generating a fresh token may resolve the problem.

I have asked Hobbs to reply :)

Hi @Pixelements. If you're using an Import approach, you will set a refresh schedule in the Power BI Service and your model will then refresh itself as often as the schedule dictates. If you're using DirectQuery, each time any given report is opened, it re-runs the query its based on and retrieves the results, so there's no need to set a refresh schedule there. You can also turn on a report setting in DirectQuery reports that says "once the report is open, go ahead and re-run your query every X minutes." In either case, your PBI Semantic Model (previously known as PBI Datasets) is using whatever connector you used when you made it to reach from the Power BI Service to Databricks and retrieve new data.

Much appreciated!

Thanks! 👍

I asked Robert to reply :)

It depends on how much control you want. I have some customers that explicitly create every table before loading into it, but that's not necessary. You can create it ad-hoc at the time it's loaded. Chances are, many columns will be inferred as strings, so you may find that you want to specifically create the table before you begin loading into it.

Thank you@@robertmosley4577

Thank you@@stephanieamrivera

Thanks for the question. Not really. I see customers use workflows unless their company already uses airflow.

Happy its helpful!

Happy its helpful!

Will get back to you shortly on this!

The connectivity of the VPC will depend from deployment to deployment. In this case, there is a private subnets with route tables to an S3 gateway endpoint, the local VPC CIDR, and 0.0.0.0/0 to a NAT gateway. The public subnet then routes all traffic to an internet gateway. The traffic from the EC2 instance to the PrivateLink endpoint for Databricks is covered in the local VPC CIDR range route table entry. The traffic finds it's way to the endpoint using DNS resolution. Hope this helps!

​@@stephanieamrivera It doesn't. The training is vague in some areas. What I would like to see is is an explanation of: (1) sample NACL for the traffic into the subnet. (2) sample security group that can be attached to the cross account IAM role that would work for private link purposes (3) configuring access for access to s3 for example and other IGW public services. I have been battling with this for 4 days now. I need a resolution asap. Documentation is taking me all over the place. There are members of my team that are calling for other platforms but I am adamant that this is the best platform for our purposes. Thanks.

@@OPopoola Please reach out to your Databricks team for more details. NACLs remain standard, they should be unchanged. This is standard AWS networking. S3 Gateway Endpoint for in-region buckets, NAT Gateway to Internet Gateway for public services, with a WAF if needed.

There are 2 different authentication methods that can be used: 1. SQL Authentication: This method involves providing a username and password to authenticate against the Azure SQL Server. It requires a login and password that are configured on the Azure SQL Server. 2. Azure Active Directory (AD) authentication: Azure SQL supports using Azure AD identities to authenticate and authorize database access. This method enables you to use Azure AD accounts or groups to authenticate and manage access to your Azure SQL Database or Azure Synapse Analytics. Azure SQL currently does not directly support authenticating with Service Principals. However, you can use Azure AD credentials associated with a Service Principal to authenticate and access Azure SQL by creating a SQL login mapped to the Service Principal's Azure AD identity.

You mean connecting Databricks to dataflow?

I have asked JD to respond :)

@@stephanieamrivera thanks

Thanks Jim!

I'll ask Ashley or one of our DAB SMEs!

Glad it was helpful!

done

The link is in the description

​@@stephanieamriverathank you so much

The local disk size in the executor is determined by the instance type of the worker nodes in your Databricks cluster. The size of the local disk is a characteristic of the instance type and is fixed for each instance. Local disk is used for storing temporary data, intermediate results, and executor-related files during the processing of jobs and tasks.

Thanks!

Our pleasure!

@@stephanieamrivera Hey, would it be possible to get the code too?

.

We no longer host the sessions live

Glad you enjoyed it! I will ask JD if he can respond :)