FME can search user data with minimum input parameters. A primary identifier such as an email address is mandatory for the search to begin. We have two more optional parameters, Tag Name and Potential Key Candidates. The Tag Name e.g., ‘PII’, when provided, shall narrow down the search to PII buckets only, else it will try to look for user data in all the buckets. Potential Key Candidate parameter expects the substring like ‘id’, ‘key’ etc., for example substring ‘id’ will use all the column names having ‘id’ like ‘EMP_ID’ ‘ADDRESS_ID’ to join two different files.
Sounds easy to use, right? Wait, FME provides another level of customization. Based on input parameters, an intermediate editable metadata file is generated. The metadata file has all the bucket names, file names, and column names where FME shall search for user data. This intermediate metadata file can be changed to have more column names or file names. FME joins search results for a single user from the data lake and Redshift and generates an HTML file. The HTML file has complete search results with bucket name, file name and location along with an editable field called ‘TO_BE_DELETED’.