batch file - Slow performance in spark streaming -

-

i using spark streaming 1.1.0 locally (not in cluster). created simple app parses data (about 10.000 entries), stores in stream , makes transformations on it. here code:

def main(args : array[string]){      val master = "local[8]"     val conf = new sparkconf().setappname("tester").setmaster(master)     val sc = new streamingcontext(conf, milliseconds(110000))      val stream = sc.receiverstream(new myreceiver("localhost", 9999))      val parsedstream = parse(stream)      parsedstream.foreachrdd(rdd =>          println(rdd.first()+"\nrule starts "+system.currenttimemillis()))      val result1 = parsedstream        .filter(entry => entry.symbol.contains("walking")         && entry.symbol.contains("true") && entry.symbol.contains("id0"))        .map(_.time)      val result2 = parsedstream        .filter(entry =>         entry.symbol == "disappear" && entry.symbol.contains("id0"))        .map(_.time)      val result3 = result1       .transformwith(result2, (rdd1, rdd2: rdd[int]) => rdd1.subtract(rdd2))      result3.foreachrdd(rdd =>      println(rdd.first()+"\nrule ends "+system.currenttimemillis()))     sc.start()    sc.awaittermination() }  def parse(stream: dstream[string]) = {      stream.flatmap { line =>         val entries = line.split("assert").filter(entry => !entry.isempty)         entries.map { tuple =>              val pattern = """\s*[(](.+)[,]\s*([0-9]+)+\s*[)]\s*[)]\s*[,|\.]\s*""".r              tuple match {               case pattern(symbol, time) =>               new data(symbol, time.toint)             }          }     } }  case class data (symbol: string, time: int)

i have batch duration of 110.000 milliseconds in order receive data in 1 batch. believed that, locally, spark fast. in case, takes 3.5sec execute rule (between "rule starts" , "rule ends"). doing wrong or expected time? advise

so using case matching in allot of jobs , killed performance, more when introduced json parser. try tweaking batch time on streamingcontext. made quite bit of difference me. how many local workers have?


Comments

Post a Comment

Popular posts from this blog

java - Oracle EBS .ClassNotFoundException: oracle.apps.fnd.formsClient.FormsLauncher.class ERROR -

-
i need in configuring java on machine because, when try loading forms, loader goes in circles , gives error: java.lang.classnotfoundexception: oracle.apps.fnd.formsclient.formslauncher.class @ sun.plugin2.applet.applet2classloader.findclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass0(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass(unknown source) @ java.lang.classloader.loadclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadcode(unknown source) @ sun.plugin2.applet.plugin2manager.createapplet(unknown source) @ sun.plugin2.applet.plugin2manager$appletexecutionrunnable.run (unknown source) @ java.lang.thread.run(unknown source) i have tried re-installing firefox, older versions of java runtime environments, disabling proxy. what missing? application working on other machines. i spent around 100 hours finding s...
Read more

c# - how to use buttonedit in devexpress gridcontrol -

-
Image
i trying create gridcontrol buttonedit in 1 of columns. when user clicks button edit, popups form select product. when selection done on popup, returns datarow selection main grid below. but when column loses focus, value wrote column disappear. here code creates gridcontrol's data , buttonedit's click event. private void frmsiparisnew_load(object sender, eventargs e) { dt = new datatable(); dt.columns.add("malzeme_kodu",typeof(string)); dt.columns.add("malzeme_aciklama", typeof(string)); dt.columns.add("adet", typeof(decimal)); dt.columns.add("birim", typeof(string)); dt.columns.add("fiyat", typeof(decimal)); dt.columns.add("kur", typeof(string)); dt.columns.add("tutar", typeof(decimal)); datarow dr = dt.newrow(); dt.rows.add(dr); gc.datasource = dt; } private void repositoryitembuttoneditmal...
Read more

How do you convert a timestamp into a datetime in python with the correct timezone? -

-
you naively expect work: import pytz datetime import datetime def tz_datetime_from_timestamp(timestamp): """convert timestamp datetime """ tz = pytz.timezone('australia/perth') server_time = datetime.utcnow() delta = tz.utcoffset(server_time) value = datetime.utcfromtimestamp(timestamp).replace(tzinfo=tz) return value + delta print tz_datetime_from_timestamp(1416387060) and convert timestamp 1416387060 wed nov 19 16:51:00 2014 gmt+8:00. ...but not. prints: 2014-11-19 16:51:00+07:43 the timezone of australia/perth not gmt+7:43. it gmt+8:00. unequivocally stated on australian government website http://www.australia.gov.au/about-australia/our-country/time : awst equal coordinated universal time plus 8 hours (utc +8). so, heck pytz pulling 7.43 from? well, turns out pytz pulls data http://www.iana.org/time-zones time zone database, , database states: # western australia # # rule name type in on @...
Read more