2022-08-08 17:19:09.160 [main] INFO VMInfo - VMInfo# operatingSystem class => com.sun.management.internal.OperatingSystemImpl 2022-08-08 17:19:09.164 [main] INFO Engine - the machine info => osInfo: Ubuntu 11 11.0.16+8-post-Ubuntu-0ubuntu120.04 jvmInfo: Linux amd64 5.13.0-52-generic cpu num: 8 totalPhysicalMemory: -0.00G freePhysicalMemory: -0.00G maxFileDescriptorCount: -1 currentOpenFileDescriptorCount: -1 GC Names [G1 Young Generation, G1 Old Generation] MEMORY_NAME | allocation_size | init_size CodeHeap 'profiled nmethods' | 117.21MB | 2.44MB G1 Old Gen | 1,024.00MB | 970.00MB G1 Survivor Space | -0.00MB | 0.00MB CodeHeap 'non-profiled nmethods' | 117.22MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB Metaspace | -0.00MB | 0.00MB G1 Eden Space | -0.00MB | 54.00MB CodeHeap 'non-nmethods' | 5.57MB | 2.44MB 2022-08-08 17:19:09.175 [main] INFO Engine - { "content":[ { "reader":{ "name":"mysqlreader", "parameter":{ "connection":[ { "jdbcUrl":[ "jdbc:mysql://192.168.2.2:3306/gene" ], "querySql":[ "select * from gene.user_info_transfer" ] } ], "password":"*********", "username":"root" } }, "writer":{ "name":"hdfswriter", "parameter":{ "column":[ { "name":"user_id", "type":"string" }, { "name":"area_id", "type":"string" }, { "name":"age", "type":"int" }, { "name":"occupation", "type":"string" } ], "compress":"", "defaultFS":"hdfs://hadoop03:8020/", "fieldDelimiter":"\t", "fileName":"user_info_data_3.txt", "fileType":"text", "path":"/user/hive/warehouse/user_info/", "writeMode":"append" } } } ], "setting":{ "speed":{ "channel":"2" } } } 2022-08-08 17:19:09.183 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null 2022-08-08 17:19:09.184 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0 2022-08-08 17:19:09.184 [main] INFO JobContainer - DataX jobContainer starts job. 2022-08-08 17:19:09.185 [main] INFO JobContainer - Set jobId = 0 2022-08-08 17:19:09.371 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.2.2:3306/gene?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. 2022-08-08 17:19:09.899 [job-0] INFO JobContainer - jobContainer starts to do prepare ... 2022-08-08 17:19:09.899 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work . 2022-08-08 17:19:09.900 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] do prepare work . 2022-08-08 17:19:09.960 [job-0] INFO HdfsWriter$Job - 由于您配置了writeMode append, 写入前不做清理工作, [/user/hive/warehouse/user_info/] 目录下写入相应文件名前缀 [user_info_data_3.txt] 的文件 2022-08-08 17:19:09.960 [job-0] INFO JobContainer - jobContainer starts to do split ... 2022-08-08 17:19:09.960 [job-0] INFO JobContainer - Job set Channel-Number to 2 channels. 2022-08-08 17:19:09.961 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks. 2022-08-08 17:19:09.962 [job-0] INFO HdfsWriter$Job - begin do split... 2022-08-08 17:19:09.965 [job-0] INFO HdfsWriter$Job - splited write file name:[hdfs://hadoop03:8020//user/hive/warehouse/user_info__d08e04a5_91f2_4610_ad2f_fec59a7bc14d/user_info_data_3.txt__b7ee6106_5144_4078_be54_6aed202d6de3] 2022-08-08 17:19:09.965 [job-0] INFO HdfsWriter$Job - end do split. 2022-08-08 17:19:09.965 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] splits to [1] tasks. 2022-08-08 17:19:09.973 [job-0] INFO JobContainer - jobContainer starts to do schedule ... 2022-08-08 17:19:09.975 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups. 2022-08-08 17:19:09.976 [job-0] INFO JobContainer - Running by standalone Mode. 2022-08-08 17:19:09.979 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks. 2022-08-08 17:19:09.983 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated. 2022-08-08 17:19:09.983 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated. 2022-08-08 17:19:09.989 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started 2022-08-08 17:19:09.991 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from gene.user_info_transfer ] jdbcUrl:[jdbc:mysql://192.168.2.2:3306/gene?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-08-08 17:19:10.003 [0-0-0-writer] INFO HdfsWriter$Task - begin do write... 2022-08-08 17:19:10.003 [0-0-0-writer] INFO HdfsWriter$Task - write to file : [hdfs://hadoop03:8020//user/hive/warehouse/user_info__d08e04a5_91f2_4610_ad2f_fec59a7bc14d/user_info_data_3.txt__b7ee6106_5144_4078_be54_6aed202d6de3] 2022-08-08 17:19:10.006 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from gene.user_info_transfer ] jdbcUrl:[jdbc:mysql://192.168.2.2:3306/gene?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-08-08 17:19:10.139 [0-0-0-writer] INFO HdfsWriter$Task - end do write 2022-08-08 17:19:10.189 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[201]ms 2022-08-08 17:19:10.190 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks. 2022-08-08 17:19:19.989 [job-0] INFO StandAloneJobContainerCommunicator - Total 8 records, 158 bytes | Speed 15B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00% 2022-08-08 17:19:19.990 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks. 2022-08-08 17:19:19.991 [job-0] INFO JobContainer - DataX Writer.Job [hdfswriter] do post work. 2022-08-08 17:19:19.992 [job-0] INFO HdfsWriter$Job - start rename file [hdfs://hadoop03:8020//user/hive/warehouse/user_info__d08e04a5_91f2_4610_ad2f_fec59a7bc14d/user_info_data_3.txt__b7ee6106_5144_4078_be54_6aed202d6de3] to file [hdfs://hadoop03:8020//user/hive/warehouse/user_info/user_info_data_3.txt__b7ee6106_5144_4078_be54_6aed202d6de3]. 2022-08-08 17:19:20.006 [job-0] INFO HdfsWriter$Job - finish rename file [hdfs://hadoop03:8020//user/hive/warehouse/user_info__d08e04a5_91f2_4610_ad2f_fec59a7bc14d/user_info_data_3.txt__b7ee6106_5144_4078_be54_6aed202d6de3] to file [hdfs://hadoop03:8020//user/hive/warehouse/user_info/user_info_data_3.txt__b7ee6106_5144_4078_be54_6aed202d6de3]. 2022-08-08 17:19:20.006 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://hadoop03:8020/user/hive/warehouse/user_info__d08e04a5_91f2_4610_ad2f_fec59a7bc14d] . 2022-08-08 17:19:20.018 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://hadoop03:8020/user/hive/warehouse/user_info__d08e04a5_91f2_4610_ad2f_fec59a7bc14d] . 2022-08-08 17:19:20.019 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work. 2022-08-08 17:19:20.019 [job-0] INFO JobContainer - DataX jobId [0] completed successfully. 2022-08-08 17:19:20.021 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /home/zhouze/PycharmProjects/yili/yili-portal/app/datax/hook 2022-08-08 17:19:20.128 [job-0] INFO JobContainer - [total cpu info] => averageCpu | maxDeltaCpu | minDeltaCpu -1.00% | -1.00% | -1.00% [total gc info] => NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime G1 Young Generation | 2 | 2 | 2 | 0.014s | 0.014s | 0.014s G1 Old Generation | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s 2022-08-08 17:19:20.129 [job-0] INFO JobContainer - PerfTrace not enable! 2022-08-08 17:19:20.130 [job-0] INFO StandAloneJobContainerCommunicator - Total 8 records, 158 bytes | Speed 15B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00% 2022-08-08 17:19:20.132 [job-0] INFO JobContainer - 任务启动时刻 : 2022-08-08 17:19:09 任务结束时刻 : 2022-08-08 17:19:20 任务总计耗时 : 10s 任务平均流量 : 15B/s 记录写入速度 : 0rec/s 读出记录总数 : 8 读写失败总数 : 0