HBase Client Issues after Upgrade

The Problem:
After upgrade application libraries (spring 3.2 to 4.2, mahout, hbase 0.98 to 1.1, ibatis 2 to mybatis, …..) the application was not able to connect to HBase anymore. After adding and excluding some the missing and conflicting dependencies it was still stuck.

The Debuging:
Narrow down the problem I decided to use a simple test application
Setup test table:

bin/hbase shell
hbase(main):001:0> create ‘test’, ‘cf’
0 row(s) in 3.8890 seconds

hbase(main):002:0> put ‘test’, ‘row1’, ‘cf:a’, ‘value1’
0 row(s) in 0.1840 seconds

Demo Application:

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;

public class HBaseClient {

    public static void main(String[] arg) throws IOException {
        Configuration config = HBaseConfiguration.create();
        config.set("hbase.zookeeper.quorum", "127.0.0.1");
        config.set("hbase.zookeeper.property.clientPort", "2182");
        config.set("hbase.hconnection.threads.core","1");
        config.set("hbase.hconnection.threads.max","10");
        config.set("hbase.client.retries.number","3");
        config.set("hbase.rootdir","hdfs://127.0.0.1:9000/hbase");
        //config.set("","");


	long startTime0 = System.currentTimeMillis();
        HTable testTable = new HTable(config, "test");
	long stopTime0 = System.currentTimeMillis();
	System.out.println("connect in "+(stopTime0 - startTime0)+" ms");
        
	long startTime = System.currentTimeMillis();
        for (int i = 0; i < 1; i++) {
            byte[] family = Bytes.toBytes("cf");
            byte[] qual = Bytes.toBytes("a");

            Scan scan = new Scan();
            scan.addColumn(family, qual);
            scan.setMaxResultsPerColumnFamily(5);
            ResultScanner rs = testTable.getScanner(scan);
            for (Result r = rs.next(); r != null; r = rs.next()) {
	         	
                byte[] valueObj = r.getValue(family, qual);
                String value = new String(valueObj);
                System.out.println(value);
            }
        }
	long stopTime = System.currentTimeMillis();
	System.out.println("Scan finished in "+(stopTime - startTime)+" ms");

	long startTime2 = System.currentTimeMillis();
	Get g=new Get("row1".getBytes());
	Result rs = testTable.get(g);
	long stopTime2 = System.currentTimeMillis();
	System.out.println("Get finished in "+(stopTime2 - startTime2)+" ms");
        
        testTable.close();
    }
}

(Sorry about the unessary speed counters – I’m a bit a performance freak)

Now I just added the libraries needed until it worked. Then I added more util didn’t work again.

The Solution:

Exception in thread "main" java.lang.NoSuchFieldError: IBM_JAVA
at org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName(UserGroupInformation.java:339)
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:384)

hadoop-core-1.2.1.jar and hadoop-auth-2.5.1.jar are conflicting remove hadoop-core from your classpath

 

Example to exclude it in Maven from mahout:

 <dependency>
    <groupId>org.apache.mahout</groupId>
    <artifactId>mahout</artifactId>
    <version>0.8</version>
    <exclusions>
       <exclusion>
           <groupId>org.apache.hadoop</groupId>
           <artifactId>hadoop-core</artifactId>
       </exclusion>
    </exclusions>
</dependency>

 

XEN 4.4 with Oracle Linux Guest

The first post is not directly about data. Is about XEN and howto prepare it for Oracle Linux as Guest. Installing XEN is the easy and documented task.

To install the Oracle Linux 7.1 guest VM you need a proper VM configuration for installing and one for running the guest

The configuration for installing:

name = “oralin01”
kernel = “/media/cdrom/images/pxeboot/vmlinuz”
ramdisk = “/media/cdrom/images/pxeboot/initrd.img”
extra = “inst.vnc inst.stage2=hd:LABEL=OL-7.1\\x20Server.x86_64 earlyprintk=xen console=hvc0”
memory = 4096
vcpus = 2
vif = [ ‘bridge=xenbr0’ ]
disk = [ ‘/dev/vg0/oralin01,raw,xvda,rw’,’file:/pathtoiso/oracle_linux_7_1_V74844-01.iso,xvdc:cdrom,r’ ]
vfb = [ ‘type=vnc, vncdisplay=1, vnclisten=127.0.0.1’ ]
on_poweroff = ‘destroy’
on_reboot = ‘restart’
on_crash = ‘restart’

connect with the consloe or vnc

xl console oralin01

Setup the Oracle Linux 7.1. Then change the configuratin, destroy the domU (your guest vm) and recreate it with the run configuratin:

name = “oralin01″
bootloader=”pygrub”
memory = 4096
vcpus = 2
vif = [ ‘mac=00:16:3e:02:0a:79,bridge=xenbr0’ , ‘mac=00:16:3e:02:0a:80,bridge=xenbr0’]
disk = [ ‘/dev/vg0/oralin01,raw,xvda,rw’,’file:/pathtoiso/oracle/oracle_linux_7_1_V74844-01.iso,xvdc:cdrom,r’ ]
vfb = [ ‘type=vnc, vncdisplay=1, vnclisten=127.0.0.1’ ]
on_poweroff = ‘destroy’
on_reboot = ‘restart’
on_crash = ‘restart’

Notice we have two network interfaces (eth0, eth1) both on the xenbr0 which should allow us to configure the public and private network if we will install Oracle RAC on Xen later.