hadoop - Hive table reading from GZIP contains meta information like file name in the first row -

-

i have created external table in hive pointing gzip file

create external table if not exists raw_cn ( column1                        string, column2                       string, column3            string, column4       string, column5            string, column6          string, column7            string, column8           string, column9                        string, column10        string

) partitioned (day_id string, file_type string) row format delimited fields terminated '|' stored textfile;

added partition:

alter table raw_cn add partition (day_id = '20140815' , file_type = 'daily' ) location    '/mapr/mapr.cluster/cn/20140501/daily';

placed gzip file @ above location

however when query table, first row gives me file level information (there no header in file). how resolve issue first row (rest of rows fine):

vendor1_617_cn_daily.201408150000664000202600020260243475554512373676764017202 0ustar  fworksfworks4f06c1a123456|82910|26|espn2|espn2|2014/08/15 01:09:42|2014/08/15     01:10:13|233|53066|jefferson-walworth (jefferson), wi 123456|82910|8|wmlw|wmlw|2014/08/15 03:16:53||233|53066|jefferson-walworth (jefferson), wi 123456|82910|3|witi|witi|2014/08/15 14:34:13|2014/08/15 14:35:20|233|53066|jefferson-walworth (jefferson), wi 123456|82910|43|hgtv|home & garden television (east)|2014/08/15 14:35:20|2014/08/15 14:37:00|233|53066|jefferson-walworth (jefferson), wi

that depends on version of hive using.

for hive version 13 , above:

there table property tblproperties ("skip.header.line.count"="1") can use while creating table. skip no of lines.

for hive version 12 , below:

you need remove line/header manually or using shell/python script.

hope helps...!!!


Comments

Post a Comment

Popular posts from this blog

java - Oracle EBS .ClassNotFoundException: oracle.apps.fnd.formsClient.FormsLauncher.class ERROR -

-
i need in configuring java on machine because, when try loading forms, loader goes in circles , gives error: java.lang.classnotfoundexception: oracle.apps.fnd.formsclient.formslauncher.class @ sun.plugin2.applet.applet2classloader.findclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass0(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadclass(unknown source) @ java.lang.classloader.loadclass(unknown source) @ sun.plugin2.applet.plugin2classloader.loadcode(unknown source) @ sun.plugin2.applet.plugin2manager.createapplet(unknown source) @ sun.plugin2.applet.plugin2manager$appletexecutionrunnable.run (unknown source) @ java.lang.thread.run(unknown source) i have tried re-installing firefox, older versions of java runtime environments, disabling proxy. what missing? application working on other machines. i spent around 100 hours finding s...
Read more

c# - how to use buttonedit in devexpress gridcontrol -

-
Image
i trying create gridcontrol buttonedit in 1 of columns. when user clicks button edit, popups form select product. when selection done on popup, returns datarow selection main grid below. but when column loses focus, value wrote column disappear. here code creates gridcontrol's data , buttonedit's click event. private void frmsiparisnew_load(object sender, eventargs e) { dt = new datatable(); dt.columns.add("malzeme_kodu",typeof(string)); dt.columns.add("malzeme_aciklama", typeof(string)); dt.columns.add("adet", typeof(decimal)); dt.columns.add("birim", typeof(string)); dt.columns.add("fiyat", typeof(decimal)); dt.columns.add("kur", typeof(string)); dt.columns.add("tutar", typeof(decimal)); datarow dr = dt.newrow(); dt.rows.add(dr); gc.datasource = dt; } private void repositoryitembuttoneditmal...
Read more

How do you convert a timestamp into a datetime in python with the correct timezone? -

-
you naively expect work: import pytz datetime import datetime def tz_datetime_from_timestamp(timestamp): """convert timestamp datetime """ tz = pytz.timezone('australia/perth') server_time = datetime.utcnow() delta = tz.utcoffset(server_time) value = datetime.utcfromtimestamp(timestamp).replace(tzinfo=tz) return value + delta print tz_datetime_from_timestamp(1416387060) and convert timestamp 1416387060 wed nov 19 16:51:00 2014 gmt+8:00. ...but not. prints: 2014-11-19 16:51:00+07:43 the timezone of australia/perth not gmt+7:43. it gmt+8:00. unequivocally stated on australian government website http://www.australia.gov.au/about-australia/our-country/time : awst equal coordinated universal time plus 8 hours (utc +8). so, heck pytz pulling 7.43 from? well, turns out pytz pulls data http://www.iana.org/time-zones time zone database, , database states: # western australia # # rule name type in on @...
Read more