Requirements for MapR

Capture	Hub	Integrate

This section describes the requirements, access privileges, and other features of HVR when using MapR for replication. HVR supports the WebHDFS API for reading and writing files from and to MapR.

Location Connection

This section lists and describes the connection details required for creating MapR location in HVR.

Hive ODBC Connection

HVR allows you to create Hive External Tables above HDFS which are only used during compare. You can enable/disable the Hive configuration for HDFS in location creation screen using the Hive External Tables field.

SSL Options

MapR Client

MapR locations can only be accessed through HVR running on Linux or Windows, and it is not required to run HVR installed on the Namenode although it is possible to do so. The MapR client should be present on the server from which HVR will access the MapR (Namenode). For more information about installing MapR client, refer to MapR Documentation.

MapR Client Configuration

The following are required on the server from which HVR connects to MapR:

• Install MapR Client

• Install Java Development Kit (JDK), version 1.7 or later

• Install Java Runtime Environment (JRE), version 7 or later

• Set the environment variable $JAVA_HOME to the Java installation directory. Ensure that this is the directory that has a bin folder, e.g. if the Java bin directory is d:\java\bin, $JAVA_HOME should point to d:\java.

• Set the environment variable $MAPR_HOME to the MapR installation directory, or the hadoop command line client should be available in the path.

  Since the binary distribution available in Hadoop website lacks Windows-specific executables, a warning about unable to locate winutils.exe is displayed. This warning can be ignored for using Hadoop library for client operations to connect to a HDFS server using HVR. However, the performance on integrate location would be poor due to this warning, so it is recommended to use a Windows-specific Hadoop distribution to avoid this warning. For more information about this warning, refer to Hadoop issue HADOOP-10051.

Verifying MapR Client Installation

To verify the MapR client installation,

1. The MAPR_HOME/bin directory in MapR installation location should contain the MapR executables in it.

2. Execute the following commands to verify MapR client installation:

   $JAVA_HOME/bin/java -version
   
   $MAPR_HOME/bin/hadoop version
   
   $MAPR_HOME/bin/hadoop classpath
   

3. If the MapR client installation is verified successfully then execute the following command to verify the connectivity between HVR and MapR:

   $MAPR_HOME/bin/hadoop fs -ls hdfs://cluster/
   

Client Configuration Files

Client configuration files are not required for HVR to perform replication, however, they can be useful for debugging. Client configuration files contain settings for different services like HDFS, and others. If the HVR integrate server is not part of the cluster, it is recommended to download the configuration files for the cluster so that the MapR client knows how to connect to HDFS.

Hive External Table

HVR allows you to create Hive external tables above HDFS files which are only used during compare. The Hive ODBC connection can be enabled for MapR in the location creation screen by selecting the Hive External Tables field (see section Location Connection).

ODBC Connection

HVR uses ODBC connection to the MapR cluster for which it requires the MapR ODBC driver for Hive installed on the server (or in the same network). For more information about using ODBC to connect to HiveServer 2, refer to MapR Documentation.

Channel Configuration

For the file formats (CSV and AVRO) the following action definitions are required to handle certain limitations of the Hive deserialization implementation during Bulk or Row-wise Compare:

• For CSV,

  Group	Table	Action
  HDFS	*	FileFormat/NullRepresentation=\\N
  HDFS	*	TableProperties/CharacterMapping="\x00>\\0;\n>\\n;\r>\\r;">\""
  HDFS	*	TableProperties/MapBinary=BASE64

• For Avro,

  Group	Table	Action
  HDFS	*	FileFormat/AvroVersion=v1_8

  v1_8 is the default value for FileFormat /AvroVersion, so it is not mandatory to define this action.

The JSON file format is not supported in MapR.

Connecting to MapR

HVR can connect to MapR with or without using the MapR user impersonation. The configuration/setup requirements differ on each scenarios mentioned below.

Without MapR User Impersonation

• When hub connects to MapR server using HVR remotelistener installed on the MapR server.
  1. Log in as MapR user and start the remotelistener on the MapR server.
• When hub connects directly to the MapR server or if the integrate server (this is separate from MapR server) connects to the MapR server.

  1. Create a MapR user on the hub/integrate server. Login as root user and execute the following commands.

     groupadd -g2000 mapr
     

     useradd -gmapr -m -u2000 mapr
     

     passwd mapr
     

  2. Login as MapR user and start the remotelistener on hub/integrate server.

With MapR User Impersonation

For more information about MapR user impersonation, refer to MapR Documentation.

• When the hub connects to the MapR server using the remotelistener on the MapR server, the MapR user impersonation is not required.
• When the hub connects directly to the MapR server or if the integrate server (this is separate from MapR server) connects to the MapR server.

  1. Login as root user on hub/integrate server and modify the core-site.xml file available in /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/ directory as shown below:

     <property>
             <name>hadoop.proxyuser.mapr.hosts</name>
             <value>*</value>
     </property>
     <property>
             <name>hadoop.proxyuser.mapr.groups</name>
             <value>*</value>
     </property>
     <property>
         <name>fs.mapr.server.resolve.user</name>
         <value>true</value>
     </property>
     

     For more information, refer to MapR Documentation.

  2. Create a file in /opt/mapr/conf/proxy/ that has name of the mapr superuser or any other user. This file can also be copied as shown below,

     sudo cp /opt/mapr/conf/proxy/mapr /opt/mapr/conf/proxy/hvrremotelistener_os_user
     

  3. Export MapR impersonation,

     export MAPR_IMPERSONATION_ENABLED=true
     

  4. Start hvrremotelistener as hvrremotelistener_os_user

  5. On the MapR server, execute the following using the config command,

     maprcli config save -values {cldb.security.resolve.user:1};
     

     For more information, refer to MapR Documentation.

  6. To verify this update, execute:

     maprcli config load -keys cldb.security.resolve.user
     

  7. Restart the MapR CLDB.