![]() ![]() If it does, then we will leverage the substring() function to return only the second index (non-zero) of that string. Using the function, startswith() we can nest this logic inside an if() condition function that will check if the value starts with 0. Split(split(pipeline().parameters.interval, ‘:’), ‘.’)Īnother challenge is the leading 0 as we will need the values to be of type integers. To capture only the Day part, we can nest the original split function inside another split function and reference a specific index of the array returned: Notice that the Day and Hour parts both fell into the same index because the delimiter differs. Using the split() function, we can split the string into an array given some delimiter.įor 01.12:05:02, if we use split(‘01.12:05:02’, ‘:’), our result is going to be an array: We want to isolate each segment (days, hours, minutes, seconds), convert to seconds, then aggregate out results that we can leverage within the addseconds() function. The parameter Interval is of type String, allowing us to parse the value into segments. Within the Data Pipeline Expression Builder, we have access toīecause our smallest increment is Seconds, we will be leveraging the addseconds() function. For this example, we need to add 1 day, 12 hours, 5 minutes, and 2 seconds to some date value we have. This Interval is used to determine a future date and comes into the pipeline as dd.hh:mm:ss (e.g. We have a Data Pipeline that has an Interval parameter (string) being passed in. ![]() Part 1: How to convert a time interval (dd.hh:mm:ss) into seconds Scenario By the end of this series, we will cover all of the tips and tricks needed to achieve these performance gains. ![]() The first two parts of this series are designed to ease you into two technical processes while the third and final part will bring everything together. Using a ForEach Activity (Sequential = False, Batch Count = 50) and iterating over each range, I was able to execute many copy activities in parallel, taking processing time down from 6.5 hours to under 8 minutes. To provide some context, I’ve used this on a large SQL Table that had a datetime column by taking the Min and Max date then creating sub-time ranges. This method can also be extended to any scenario when provided with some boundary condition. We can improve performance by creating many sub-time ranges based on the StartIndex, EndIndex, and Interval, then for each sub-time range, we call a child pipeline containing a Copy Activity, allowing multiple Copy Activities to execute at the same time, all handling a subset of data. In the current design, the pipeline will take a considerable amount of time using a single Copy Activity if we provide a StartIndex and EndIndex spanning multiple years with a small Interval and Pagecount. ![]() Pagecount – number of records per file (limited by the source).Interval – Granularity of data, if interval is set to 1 minute, if provided a Start and End index for a 1-hour span, we will get back 60 records.There, we covered how to move parquet files from a REST API to Microsoft Fabric Lakehouse in a semi-single threaded method given these parameters: In this series, I will extend upon my previous blog post on ingesting files into a Lakehouse from a REST API with pagination ft. You should know the maximum number of concurrent connections and other factors, then design your solution to utilize them to your advantage. In many cases, we can logically partition the source data into buckets then copy each bucket over to the destination.Īn important tip to remember is for you, as the engineer, to take the time and understand your source and destination. It doesn’t matter if the source is a REST API, Blob Storage, or a Transactional Database. To improve performance, instead of using a single Copy Activity to move a large volume of data, we can have multiple Copy Activities moving smaller volumes in parallel. While this works great, you might face a scenario where you need to improve the performance by reducing the time it takes to move data into your Fabric Lakehouse. Often, we see solutions leveraging a single Copy Activity to move large volumes of data. Welcome to this short series where we’ll be discussing the technical methods used to improve Data Pipeline Copy activity performance through parallelization by logically partitioning any source. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |