Migration utility

The data source import file should have its columns in the following order:

1. SourceName
2. SourceDescription
3. SourceType
4. SourcePath
5. Source_HDFS_URL
6. Source_HIVE_URL
7. Source_JDBC_URL
8. Source_JDBC_USER
9. Source_JDBC_PASSWD
10. Source_JDBC_DRIVER

Use the following samples as a guide.

• Sample JDBC data source import file:

  1. "OracleHR","Corporate Oracle
        HR","jdbc","/HR",,,"jdbc:oracle:thin:@172.31.38.71:1521:XE","h
        r","rO0ABXcIAAABYv7KAz5zcgAZamF2YXguY3J5cHRvLlNlYWxlZE9iamVjdD42PabDt1RwAgAEWwANZW5jb2RlZFB
        hcmFtc3QAAltCWwAQZW5jcnlwdGVkQ29udGVudHEAfgABTAAJcGFyYW1zQWxndAASTGphdmEvbGFuZy9TdHJpbmc7TA
        AHc2VhbEFsZ3EAfgACeHBwdXIAAltCrPMXAYIVOACAAB4cAAAABDFgkyIzH3cWC5fAYIVOACAAB4cAAAABDFgkyIzH3
        cWC5fr32fVXHycHQAFEFFUy9FQ0IvUEtDUzVQYWRkaW5nr32fVXHycHQAFEFFUy9FQ0IvUEtDUzVQYWRkaW5n","ora
        cle.jdbc.OracleDriver"

• Sample HIVE data source import file:

  1. "demoHive","Claims HIVE database",source_type_hive,"/default",,"jdbc:hive2:// ip-172-31-31
     -141.ec2.internal:10000",,,,"org.apache.hive.jdbc.HiveDriver"

• Sample HDFS data source import file:

  1. "sfo-airport2","Local HDFS sfo",source_type_hdfs,"/sfo-airport3","hdfs:// ip-172-31-31-141
     .ec2.internal:8020",,,,,

After importing data sources, make sure to:

• Create and connect agents.
• Profile the data before importing any tag associations or data objects.

The columns for the roles metadata import file are in the following order:

1. RoleName
2. RoleDescription
3. RoleAccessLevel
4. RoleVirtualFolders
5. RoleDomains
6. MetadataAccess
7. DataAccess
8. MakeDefaultRole
9. AllowJobExecution

Sample role import file:

1. "Analyst","Analyst Role",Analyst,"[]","[]",”FALSE”, “METADATA”, “FALSE”, “TRUE”
2. "SFOAnalyst","Curates tags and discovers data in the SFO Open Dataset.",Analyst,"[SFOAirpo
   rt]","[Aviation]", ”TRUE”, “NATIVE”, “FALSE”, “TRUE”

Before importing any users, make sure you have imported all custom roles that are assigned to the users.

The columns for the user metadata import file are in the following order:

1. USER_NAME
2. USER_DESCRIPTION
3. USER_ROLE
4. USER_FAVORITES: Include favorites in an array with each favorite separated with "::".
   1. ReviewSource
   2. ReviewResource
5. USER_REVIEWS: Include reviews in an array with each review separated with "::".
   1. ReviewSource
   2. ReviewResource
   3. ReviewRating
   4. ReviewTitle
   5. ReviewDescription
6. USER_ATTRIBUTES: The following attributes are to be passed as a comma-separated list enclosed in '{}'.
   1. firstname
   2. lastname
   3. email
   4. user_last_login [Timestamp]

Sample user import file:

1. "lara_analyst","","[SFOAnalyst]","[]","[]","{firstName=Lara,lastName=Analyst,email=lara@
   xyz.com}"
2. "ldcuser","","[Guest]","[]","[]","{TimeStamp}"

The columns for the tag domain metadata import file are in the following order:

1. DomainName
2. DomainDescription
3. DomainColor

Sample domain import file:

1. "Aviation","SFO open data. Air traffic landing and passenger statistics, tail (aircrafts) 
   numbers and models","#50AAE7"
2. "Built-in_Tags","Built-in_Tags","#30BAE7"

The columns for the tags metadata import file are in the following order:

1. TAG_DOMAIN
2. TAG_NAME
3. TAG_DESCRIPTION
4. TAG_METHOD
5. TAG_REGEX
6. TAG_MINSCORE
7. TAG_MINLEN
8. TAG_MAXLEN
9. TAG_ENABLED
10. TAG_LEARNING_ENABLED
11. EXTERNAL_ID
12. EXTERNAL_SOURCE_NAME
13. SYNONYMS

Use the following samples as guides.

• Sample REGEX tag import file

  1. "Built-in_Tags","Email","Email",REGEX,"[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(?:.[a-zA-Z0-9!#$%&'
     *+/=?^_`{|}~-]+)*@(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?.)+[a-zA-Z0-9](?:[a-zA-Z0-9-]
     *[a-zA-Z0-9])?",0.4,5,64,true,true,,,""

• Sample Value tag import file

  1. "Built-in_Tags","First_Name","First_Name",VALUE,,0.4,0,0,true,true,,,"" 
  2. "Sales","Customers","Tag for Customers",VALUE,,0.6,0,0,true,true,,,""   
  3. "Sales","Location",,VALUE,,0.8,0,0,true,true,,,""   
  4. "Sales","Orders","Tag for Orders",VALUE,,0.0,0,0,true,true,,,""   
  5. "Sales","Regions","Tag for Regions",VALUE,,0.8,0,0,true,true,,,""   
  6. "Sales","Sales","Tag for Sales",VALUE,,0.6,0,0,true,true,,,""  
  7. "Sales","Stores",,VALUE,,0.8,0,0,true,true,,,

You can use the tag name to show hierarchy in tags, using the dot notation for naming the tag as follows:

1. "Sales","Orders.Jan.Week1","Tag for Orders",VALUE,,0.0,0,0,true,true,,,""

The example above identifies a tag Week1 with parent tag Jan and grandparent tag Orders under the Sales domain.

The columns for the tag associations metadata import file are in the following order:

1. TagDomain
2. TagName
3. DataSourceName
4. ResourceName
5. FieldName
6. TagPropagation: Possible values: SEED or OTHER. SEED is used in Discovery Tags (Marked with dot(.) in front of tag name).
7. TagAssociationState

• Sample tag associations import file

  1. "Sales","COGS","HIVE","/foodmart.sales_fact_dec_1998","store_cost",SAMPLE,ACCEPTED
  2. "Global","Customer_Master_GDPR","HIVE","/foodmart.customer","customer_id",OTHER,REJECTED
  3. "Sales","Sales.Sales_location","HDFS","/user/ldcuser/Pass1/raw/nyc_open/r at_sightings_nyc_2014.csv","LOCATION_TYPE",SAMPLE,ACCEPTED   
  4. "Sales","Sales.Sales_location","MySQL","/HR.employee","location",SAMPLE,ACCEPTED</div>

The columns for the reviews metadata import file are in the following order:

1. USER_NAME
2. USER_DESCRIPTION
3. USER_ROLE (enclose roles in array of strings)
4. USER_FAVORITES: Include favorites in an array, with the following parts of each favorite separated with "::".
   1. Review/Source
   2. Review/Resource
5. USER_REVIEWS: Include the following parts of a review in an array, with the following parts of each review separated with "::".
   1. Review/Source
   2. Review/Resource
   3. Review/Rating
   4. Review/Title
   5. Review/Description
6. USER_ATTRIBUTES: Pass the following attributes in comma-separated list enclosed in '{}'.
   1. firstname
   2. lastname
   3. email
   4. user_last_login [Timestamp]

   Use the following sample as a guide.

   • Sample CSV file for user reviews:

     1. ldcuser,Predefined administrator user,"[Administrator]","[]","[MyHIVE::/default.tabl e3::4::Database_review::
     This is one of the nicest database, 
     2. MyHIVE::/default.table3::4::::]","{firstName=LDC, lastName=Service, email=build+ldcuser@wldata.com}"

The Export Comments/Description option can export the custom field comments, label, and description as a _comments.csv file. file.

The columns of the field comments or descriptions metadata import file are in the following order:

1. VFName
2. Resource Path
3. Field name
4. Field Label
5. User Comment

Use the following sample as a guide.

• Sample comments import file

  DO_Test,/user/EmpSkill_Analytics/Employee.csv,employee_id,EMPID,This is a sample comment to test re-profiling effects of user comments

The columns for the virtual folder metadata import file are in the following order:

1. VFName
2. VFDescription
3. VFParent
4. VFPath
5. VFPInclude
6. VFPathExclude
7. VF_IsRoot

Use the following sample as a guide.

• Sample virtual folder import file

  "foodmart4","foodmart4","foodMart","/foodmart","(.customer_s1.*|.sales.*)",,false
        "foodMartChild","food mart","foodMart","/foodmart",".customer.*",,false

• Imported the corresponding data sources and virtual folders
• Profiled the resources
• Imported the tag domains, tags, and tag associations

The column sequence in the data object metadata import file is a multi-level nested sequence as follows:

1. Data Object Name
2. Description
3. Join Conditions: Multiple join conditions are separated with ':#;' as in: join_condition1:#;join_condition2:#;join_condition3
   • Where each join condition is constructed with:
     1. left_field_info
     2. right_field_info
     3. join_info

     with each entity in join_condition separated as left_field_info;;right_field_info##join_info

     • Each join_info is built as follows:
       1. Left Column info
       2. Right Column info
       3. Join Cardinality
       4. Join Order
       5. Join Operation

       with each entity in join_info separated as follows:

       left_col_info;;right_col_info##Join_Cardinality,Join_Order,Join_Operation

       • Each column info is built as follows:
         1. Column Origin
         2. Column Data Source Name
         3. Column path
         4. Column resolved name
         5. Column name

         with each entity in col_info separated with a comma (,):

         Left_col_origin,Left_col_source_name,Left_col_path,Left_col_res_name,Left_col_name

Use the following sample as a guide.

• Sample data objects import file

  1. ProjectDO,Lara's Projects,HdfsDS::HdfsDS::/data/DocData/Lara/DO/Data/ProjectReq.csv::Proje
        ctReq.csv::man_id;;HdfsDS::HdfsDS::/data/DocData/Joe/DO_Demo/UseCase1/Data/Employ.csv::Emp
        loy.csv::emp_id##MANY_MANY::ALTERNATE::JoinOpEQ

The columns for the datasets metadata import file are in the following order:

1. Dataset Name
2. Description
3. ID
4. Schema Version
5. Virtual Folder
6. Path Specifications in the format
   1. Exclude Regex Pattern
   2. Include Regex Pattern
   3. Source Path

      With each entity separated by :: (two colons) and multiple path specifications separated by ;; (two semicolons). For example:

      <Exclude Regex1>::<Include Regex1>::<Source Path1>;;<Exclude Regex2>::<Include Regex2>::<Source Path2>

7. Fields
8. Origin

Use the following sample as a guide.

• Sample dataset import file

  1. Dset1,,Dset1,10,HdfsDS,::.*::/data/data_NYPD;;,,HdfsDS 
  2. EmpDataSet,,EmpDataSet01,10,HdfsDS,::.*::/data/DocData;;,,HdfsDS 
  3. Sample,Sample Description,ID:01,9,DocDataHdfs,.*json::.*csv::/data/DocData/Joe;;.*xml::.*j
        son::/data/DocData/Lara;;,,HdfsDS

The columns for the custom properties metadata import file are in the following order:

1. Name
2. Display Name
3. Data Type
4. Description
5. Property Group
6. Property Group Description
7. Is property custom property
8. Access Level, where each join condition is constructed as:
   • pg1 = View: Everyone/Set: Analyst & Higher
   • pg2 = View: Everyone/Set: Steward & Higher
   • pg3 = View: Everyone/Set: Admin Only
   • pg4 = View: Everyone/Set: Nobody
   • pg5 = View: Analyst & Higher/Set: Admin Only
   • pg6 = View: Steward & Higher/Set: Admin Only
   • pg7 = View: Admin Only/Set: Admin Only
9. Is property case sensitive
10. Searchable
11. Facetable

Use the following sample custom property import file as a guide.

1. DataGrp,Data Group,string,Data Group,,,true,pg1,false,true,true 
2. Department,Department,string,,,,true,pg1,false,true,true 
3. test_property,Test Property,string,Test Description,TestGroup,,true,pg7,false,false,true 
4. test_property2,Test Property 2,string,Test Property 2 Description,TestGroup,,true,pg4,fa
    lse,true,false