java - Mongo Hadoop Connector Issue -

-

i trying run mapreduce job: pull mongo , write hdfs, cannot seem job run. not find example, issues having if set input path of mongo loos output path of mongo. , getting authentication error when mongodb instance not have authentication.

final configuration conf = getconf(); final job job = new job(conf, "sort"); mongoconfig config = new mongoconfig(conf); mongoconfigutil.setinputformat(getconf(), mongoinputformat.class); fileoutputformat.setoutputpath(job, new path("/trythisdir")); mongoconfigutil.setinputuri(conf,"mongodb://localhost:27017/fake_data.file"); //conf.set("mongo.output.uri", "mongodb://localhost:27017/fake_data.file"); job.setjarbyclass(imageextractor.class); job.setmapperclass(imageextractormapper.class);  job.setoutputkeyclass(text.class); job.setoutputvalueclass(text.class);  job.setinputformatclass( mongoinputformat.class );  // execute job , return status return job.waitforcompletion(true) ? 0 : 1;

edit: current error having:

exception in thread "main" java.lang.illegalargumentexception: couldn't connect , authenticate collection     @ com.mongodb.hadoop.util.mongoconfigutil.getcollection(mongoconfigutil.java:353)     @ com.mongodb.hadoop.splitter.mongosplitterfactory.getsplitterbystats(mongosplitterfactory.java:71)     @ com.mongodb.hadoop.splitter.mongosplitterfactory.getsplitter(mongosplitterfactory.java:107)     @ com.mongodb.hadoop.mongoinputformat.getsplits(mongoinputformat.java:56)     @ org.apache.hadoop.mapred.jobclient.writenewsplits(jobclient.java:1079)     @ org.apache.hadoop.mapred.jobclient.writesplits(jobclient.java:1096)     @ org.apache.hadoop.mapred.jobclient.access$600(jobclient.java:177)     @ org.apache.hadoop.mapred.jobclient$2.run(jobclient.java:995)     @ org.apache.hadoop.mapred.jobclient$2.run(jobclient.java:948)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1408)     @ org.apache.hadoop.mapred.jobclient.submitjobinternal(jobclient.java:948)     @ org.apache.hadoop.mapreduce.job.submit(job.java:566)     @ org.apache.hadoop.mapreduce.job.waitforcompletion(job.java:596)     @ com.orbis.image.extractor.mongo.imageextractor.run(imageextractor.java:103)     @ org.apache.hadoop.util.toolrunner.run(toolrunner.java:70)     @ com.orbis.image.extractor.mongo.imageextractor.main(imageextractor.java:78)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:606)     @ org.apache.hadoop.util.runjar.main(runjar.java:208) caused by: java.lang.nullpointerexception     @ com.mongodb.mongouri.<init>(mongouri.java:148)     @ com.mongodb.mongoclient.<init>(mongoclient.java:268)     @ com.mongodb.hadoop.util.mongoconfigutil.getcollection(mongoconfigutil.java:351)     ... 22 more

late answer.. may helpul people. encountered same problem while playing apache spark.

i think should set correctly mongo.input.uri , mongo.output.uri used hadoop , input , output formats.

/*correct input , output uri setting on spark(hadoop)*/ conf.set("mongo.input.uri", "mongodb://localhost:27017/dbname.inputcolname"); conf.set("mongo.output.uri", "mongodb://localhost:27017/dbname.outputcolname");  /*set input , output formats*/ job.setinputformatclass( mongoinputformat.class ); job.setoutputformatclass( mongooutputformat.class )

btw, if "mongo.input.uri" or "mongo.output.uri" strings have typos causes same error.


Comments

Post a Comment

Popular posts from this blog

java - Oracle EBS .ClassNotFoundException: oracle.apps.fnd.formsClient.FormsLauncher.class ERROR -

-
i need in configuring java on machine because, when try loading forms, loader goes in circles , gives error: java.lang.classnotfoundexception: oracle.apps.fnd.formsclient.formslauncher.class @ sun.plugin2.applet.applet2classloader.findclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass0(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass(unknown source) @ java.lang.classloader.loadclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadcode(unknown source) @ sun.plugin2.applet.plugin2manager.createapplet(unknown source) @ sun.plugin2.applet.plugin2manager$appletexecutionrunnable.run (unknown source) @ java.lang.thread.run(unknown source) i have tried re-installing firefox, older versions of java runtime environments, disabling proxy. what missing? application working on other machines. i spent around 100 hours finding s
Read more

c# - how to use buttonedit in devexpress gridcontrol -

-
Image
i trying create gridcontrol buttonedit in 1 of columns. when user clicks button edit, popups form select product. when selection done on popup, returns datarow selection main grid below. but when column loses focus, value wrote column disappear. here code creates gridcontrol's data , buttonedit's click event. private void frmsiparisnew_load(object sender, eventargs e) { dt = new datatable(); dt.columns.add("malzeme_kodu",typeof(string)); dt.columns.add("malzeme_aciklama", typeof(string)); dt.columns.add("adet", typeof(decimal)); dt.columns.add("birim", typeof(string)); dt.columns.add("fiyat", typeof(decimal)); dt.columns.add("kur", typeof(string)); dt.columns.add("tutar", typeof(decimal)); datarow dr = dt.newrow(); dt.rows.add(dr); gc.datasource = dt; } private void repositoryitembuttoneditmal
Read more

nvd3.js - angularjs-nvd3-directives setting color in legend as well as in chart elements -

-
good morning i using angularjs-nvd3-directives, great, having difficulties when setting color of multi bar chart ( http://cmaurer.github.io/angularjs-nvd3-directives/multi.bar.chart.html ). the show legend works well: <div ng-controller="examplectrl"> <nvd3-multi-bar-chart data="exampledata" id="showlegendexample" width="550" height="300" showxaxis="true" showyaxis="true" xaxistickformat="xaxistickformatfunction()" showlegend="true"> <svg></svg> </nvd3-multi-bar-chart> </div> but when add option in conjunction color="colorfunction()" where, var colorarray = ['#ff0000', '#0000ff', '#ffff00', '#00ffff']; $scope.colorfunction = function() { return function(d, i) { return colorarray[i]; }; } the color of cha
Read more