Processing Methods
Duplicate Manager offers different methods to run your Jobs, tailored to your specific requirements and situation. Due to technical limitations within the Dataverse Platform—especially when searching through all records of an entity—some jobs may not be executable directly and automatically inside the Dataverse environment. This depends on factors like the number of records in the entity and the complexity of certain matching algorithms.
This section explains the benefits and restrictions of each processing method available, helping you decide which method is best suited for your scenario.
Dataverse
This method processes the job automatically within the Dataverse environment.
Choose this method when detecting duplicates in entities with a low total record count and rules that do not involve fuzzy matching.
Benefits
- Easy to use: No need to install or use the external application.
- Convenient: Fully integrated into the Dynamics & Power Apps experience.
- Fast: Ideal for entities with a low number of records and simple matching rules (e.g., Exact Match, Starts With).
Restrictions
- 2-minute runtime limit: Microsoft enforces a strict execution limit of 2 minutes. Jobs that require longer processing time due to high record count or complex matching (e.g. fuzzy matching) will fail.
- Record and algorithm limitations:
- If the combined number of records across all entities that the job needs to search (based on the selected rules) exceeds 250,000, or if fuzzy matching is used and the total exceeds 10,000 records, you will be prompted to confirm that the job may fail.
- This method is not available if:
- the total number of records exceeds 750,000, or
- fuzzy matching is used and the total exceeds 20,000 records
- These hard limits exist because jobs of this scale almost always fail within Dataverse due to the time constraint.
- Retry possible: Failed jobs can be rerun with the same or adjusted settings via the Rerun option in the header bar or context menu, including the possibility to switch the processing method.
Desktop
This method uses the Duplicate Manager Desktop App in order to process jobs. It is the most powerful processing method and comes with significantly fewer restrictions than the Power Apps processing method. See Running a Job with the Duplicate Manager Desktop App for instructions on how to process jobs locally on your device.
Choose this method when detecting duplicates in entities with medium to large record counts or when using complex matching methods, including fuzzy matching.
Benefits
- Can handle any number of records and all matching methods without limitations.
- Ideal for large-scale and complex duplicate detection scenarios.
Restrictions
- Requires you to download and run the Duplicate Manager Desktop App.
- Performance depends on your local system and network speed.