Before scanning you can scope your scan to specific folders or subfolders by choosing the appropriate items in the list. Once the data source is registered and scanned, the Data map extracts information about the structure (hierarchical namespace) of the data source. This information is used to build the browsing experience for data discovery.
Note:
- All future assets under a certain parent will be automatically selected if the parent is fully or partially checked
- After a successful scan, there may be delay before newly scanned assets appear in the browse experience. This delay may take up to a
few hours.
While searching the catalog for assets, operators can be used to compose a search query.
Specifically you can use the Boolean operators NOT in all caps to specify what an asset can't contain as a keyword to the right of the clause or use '*' a wildcard that matches on one to many characters so that your query does not return assets that have properties with (.wmk) in them.
Example: Expense NOT wmk NOT *.wmk
(Operators can be combined as many times as need in a single query.)
Concept of resource sets:
- A resource set is a single object in the catalog that represents a
large number of assets in storage.
- To address the challenge of mapping large numbers of data assets to a
single logical resource, Azure Purview uses resource sets.
- Azure Purview automatically detects resource sets when scanning. This
feature looks at all of the data that's ingested via scanning and
compares it to a set of defined patterns. Then it switches from a
full scan to a sample scan.
- In a sample scan, it opens only a subset of the files that it thinks
are in the resource set. For each file it does open, it uses its
schema and runs its classifiers.
- Azure Purview then finds the newest resource among the opened
resources and uses that resource's schema and classifications in the
entry for the entire resource set in the catalog. Stores aggregate
information about the partition resources that make up the resource
set.
To customize or override how Azure Purview detects which assets are grouped as resource sets and how they are displayed within the catalog, you can define pattern rules in the management center.
Create resource set pattern rules:
- Go to the management center. Select Pattern rules from the menu
under the Resource sets heading. Select + New to create a new rule
set.
- Enter the scope of your resource set pattern rule. (Folder path)
- Update the fields appropriately, in your case mainly, Qualified
name and Do not group as resource set
Note: After a pattern rule is created, all new scans will apply the
rule during ingestion. Existing assets in the data catalog will be
updated via a background process which can take up to a few hours.
Example: Don't group .wmk files into resource sets
Input Files:
https://myazureblob.blob.core.windows.net/bar/raw/Expense-7/01-01-2020/22:33:22-001.xls
https://myazureblob.blob.core.windows.net/bar/raw/Expense-8/01-01-2020/22:33:22-002.wmk
Pattern rule
Scope: https://myazureblob.blob.core.windows.net/bar/
Display name: Expense-{{Fileid}}
Qualified Name: raw/Filename-{{Fileid:int}}/{{:date}}/{{:time}}-{{:int}}.wmk
Resource Set: false
Output individual assets
Asset 1
Display name: Expense-7
Qualified Name: https://myazureblob.blob.core.windows.net/bar/raw/Expense-7/01-01-2020/22:33:22-001.xls
Additionally, if you feel this is not helpful, you can share your Feedback so the product team can look into this idea. ✌
