2022-07-28 15:13:27.162 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2022-07-28 15:13:27.171 [main] INFO Engine - the machine info => osInfo: Oracle Corporation 1.8 25.262-b10 jvmInfo: Linux amd64 3.10.0-957.el7.x86_64 cpu num: 128 totalPhysicalMemory: -0.00G freePhysicalMemory: -0.00G maxFileDescriptorCount: -1 currentOpenFileDescriptorCount: -1 GC Names [PS MarkSweep, PS Scavenge] MEMORY_NAME | allocation_size | init_size PS Eden Space | 256.00MB | 256.00MB Code Cache | 240.00MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB PS Survivor Space | 42.50MB | 42.50MB PS Old Gen | 683.00MB | 683.00MB Metaspace | -0.00MB | 0.00MB 2022-07-28 15:13:27.193 [main] INFO Engine - { "content":[ { "reader":{ "name":"hdfsreader", "parameter":{ "column":[ "*" ], "defaultFS":"hdfs://hadoop03:8020/", "encoding":"UTF-8", "fieldDelimiter":"\t", "fileType":"text", "path":"/user/hive/warehouse/user_info/user_info_data.txt" } }, "writer":{ "name":"hdfswriter", "parameter":{ "column":[ { "name":"user_id", "type":"string" }, { "name":"area_id", "type":"string" }, { "name":"age", "type":"int" }, { "name":"occupation", "type":"string" } ], "compress":"", "defaultFS":"hdfs://hadoop03:8020/", "fieldDelimiter":"\t", "fileName":"user_info_data_2.txt", "fileType":"text", "path":"/user/hive/warehouse/user_info/", "writeMode":"append" } } } ], "setting":{ "speed":{ "channel":"1" } } } 2022-07-28 15:13:27.214 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null 2022-07-28 15:13:27.216 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0 2022-07-28 15:13:27.217 [main] INFO JobContainer - DataX jobContainer starts job. 2022-07-28 15:13:27.219 [main] INFO JobContainer - Set jobId = 0 2022-07-28 15:13:27.244 [job-0] INFO HdfsReader$Job - init() begin... 2022-07-28 15:13:27.681 [job-0] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":[]} 2022-07-28 15:13:27.682 [job-0] INFO HdfsReader$Job - init() ok and end... 2022-07-28 15:13:28.899 [job-0] INFO JobContainer - jobContainer starts to do prepare ... 2022-07-28 15:13:28.900 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] do prepare work . 2022-07-28 15:13:28.901 [job-0] INFO HdfsReader$Job - prepare(), start to getAllFiles... 2022-07-28 15:13:28.902 [job-0] INFO HdfsReader$Job - get HDFS all files in path = [/user/hive/warehouse/user_info/user_info_data.txt] 2022-07-28 15:13:29.892 [job-0] INFO HdfsReader$Job - [hdfs://hadoop03:8020/user/hive/warehouse/user_info/user_info_data.txt]是[text]类型的文件, 将该文件加入source files列表 2022-07-28 15:13:29.894 [job-0] INFO HdfsReader$Job - 您即将读取的文件数为: [1], 列表为: [hdfs://hadoop03:8020/user/hive/warehouse/user_info/user_info_data.txt] 2022-07-28 15:13:29.895 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] do prepare work . 2022-07-28 15:13:29.986 [job-0] INFO HdfsWriter$Job - 由于您配置了writeMode append, 写入前不做清理工作, [/user/hive/warehouse/user_info/] 目录下写入相应文件名前缀 [user_info_data_2.txt] 的文件 2022-07-28 15:13:29.986 [job-0] INFO JobContainer - jobContainer starts to do split ... 2022-07-28 15:13:29.986 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels. 2022-07-28 15:13:29.987 [job-0] INFO HdfsReader$Job - split() begin... 2022-07-28 15:13:29.987 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] splits to [1] tasks. 2022-07-28 15:13:29.987 [job-0] INFO HdfsWriter$Job - begin do split... 2022-07-28 15:13:29.991 [job-0] INFO HdfsWriter$Job - splited write file name:[hdfs://hadoop03:8020//user/hive/warehouse/user_info__cae0010e_452e_4ee5_b9c2_91963c7817ce/user_info_data_2.txt__49706284_96d9_4ab0_b32e_56c90cf5c66e] 2022-07-28 15:13:29.992 [job-0] INFO HdfsWriter$Job - end do split. 2022-07-28 15:13:29.992 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] splits to [1] tasks. 2022-07-28 15:13:30.003 [job-0] INFO JobContainer - jobContainer starts to do schedule ... 2022-07-28 15:13:30.009 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups. 2022-07-28 15:13:30.011 [job-0] INFO JobContainer - Running by standalone Mode. 2022-07-28 15:13:30.139 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks. 2022-07-28 15:13:30.152 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated. 2022-07-28 15:13:30.152 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated. 2022-07-28 15:13:30.171 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started 2022-07-28 15:13:30.231 [0-0-0-reader] INFO HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]} 2022-07-28 15:13:30.233 [0-0-0-reader] INFO Reader$Task - read start 2022-07-28 15:13:30.233 [0-0-0-reader] INFO Reader$Task - reading file : [hdfs://hadoop03:8020/user/hive/warehouse/user_info/user_info_data.txt] 2022-07-28 15:13:30.239 [0-0-0-writer] INFO HdfsWriter$Task - begin do write... 2022-07-28 15:13:30.240 [0-0-0-writer] INFO HdfsWriter$Task - write to file : [hdfs://hadoop03:8020//user/hive/warehouse/user_info__cae0010e_452e_4ee5_b9c2_91963c7817ce/user_info_data_2.txt__49706284_96d9_4ab0_b32e_56c90cf5c66e] 2022-07-28 15:13:30.265 [0-0-0-reader] INFO UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":"\t","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null] 2022-07-28 15:13:30.271 [0-0-0-reader] INFO Reader$Task - end read source files... 2022-07-28 15:13:30.539 [0-0-0-writer] INFO HdfsWriter$Task - end do write 2022-07-28 15:13:30.573 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[404]ms 2022-07-28 15:13:30.574 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks. 2022-07-28 15:13:40.160 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 79 bytes | Speed 7B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00% 2022-07-28 15:13:40.161 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks. 2022-07-28 15:13:40.162 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] do post work. 2022-07-28 15:13:40.162 [job-0] INFO HdfsWriter$Job - start rename file [hdfs://hadoop03:8020//user/hive/warehouse/user_info__cae0010e_452e_4ee5_b9c2_91963c7817ce/user_info_data_2.txt__49706284_96d9_4ab0_b32e_56c90cf5c66e] to file [hdfs://hadoop03:8020//user/hive/warehouse/user_info/user_info_data_2.txt__49706284_96d9_4ab0_b32e_56c90cf5c66e]. 2022-07-28 15:13:40.176 [job-0] INFO HdfsWriter$Job - finish rename file [hdfs://hadoop03:8020//user/hive/warehouse/user_info__cae0010e_452e_4ee5_b9c2_91963c7817ce/user_info_data_2.txt__49706284_96d9_4ab0_b32e_56c90cf5c66e] to file [hdfs://hadoop03:8020//user/hive/warehouse/user_info/user_info_data_2.txt__49706284_96d9_4ab0_b32e_56c90cf5c66e]. 2022-07-28 15:13:40.177 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://hadoop03:8020/user/hive/warehouse/user_info__cae0010e_452e_4ee5_b9c2_91963c7817ce] . 2022-07-28 15:13:40.188 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://hadoop03:8020/user/hive/warehouse/user_info__cae0010e_452e_4ee5_b9c2_91963c7817ce] . 2022-07-28 15:13:40.189 [job-0] INFO JobContainer - DataX Reader.Job [hdfsreader] do post work. 2022-07-28 15:13:40.190 [job-0] INFO JobContainer - DataX jobId [0] completed successfully. 2022-07-28 15:13:40.192 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /root/yili-datax/datax/hook 2022-07-28 15:13:40.296 [job-0] INFO JobContainer - [total cpu info] => averageCpu | maxDeltaCpu | minDeltaCpu -1.00% | -1.00% | -1.00% [total gc info] => NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime PS MarkSweep | 2 | 2 | 2 | 0.132s | 0.132s | 0.132s PS Scavenge | 2 | 2 | 2 | 0.105s | 0.105s | 0.105s 2022-07-28 15:13:40.297 [job-0] INFO JobContainer - PerfTrace not enable! 2022-07-28 15:13:40.298 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 79 bytes | Speed 7B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00% 2022-07-28 15:13:40.300 [job-0] INFO JobContainer - 任务启动时刻 : 2022-07-28 15:13:27 任务结束时刻 : 2022-07-28 15:13:40 任务总计耗时 : 13s 任务平均流量 : 7B/s 记录写入速度 : 0rec/s 读出记录总数 : 4 读写失败总数 : 0