Home Explore Help
Register Sign In
chenguilong
/
backend
1
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Tree: 20d2becaf2
Branches Tags
backend
/
app
/
datax
/
log
/
2022-08-07
/
2hive1659860410_json-16_20_10.526.log

2hive1659860410_json-16_20_10.526.log 10 KB
History Raw

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153 
  1. 2022-08-07 16:20:10.708 [main] INFO  VMInfo - VMInfo# operatingSystem class => com.sun.management.internal.OperatingSystemImpl
    2. 

  2. 2022-08-07 16:20:10.713 [main] INFO  Engine - the machine info  => 
    3. 

  3. 

  4. 	osInfo:	Ubuntu 11 11.0.16+8-post-Ubuntu-0ubuntu120.04
    5. 

  5. 	jvmInfo:	Linux amd64 5.13.0-52-generic
    6. 

  6. 	cpu num:	8
    7. 

  7. 

  8. 	totalPhysicalMemory:	-0.00G
    9. 

  9. 	freePhysicalMemory:	-0.00G
    10. 

  10. 	maxFileDescriptorCount:	-1
    11. 

  11. 	currentOpenFileDescriptorCount:	-1
    12. 

  12. 

  13. 	GC Names	[G1 Young Generation, G1 Old Generation]
    14. 

  14. 

  15. 	MEMORY_NAME                    | allocation_size                | init_size                      
    16. 

  16. 	CodeHeap 'profiled nmethods'   | 117.21MB                       | 2.44MB                         
    17. 

  17. 	G1 Old Gen                     | 1,024.00MB                     | 970.00MB                       
    18. 

  18. 	G1 Survivor Space              | -0.00MB                        | 0.00MB                         
    19. 

  19. 	CodeHeap 'non-profiled nmethods' | 117.22MB                       | 2.44MB                         
    20. 

  20. 	Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
    21. 

  21. 	Metaspace                      | -0.00MB                        | 0.00MB                         
    22. 

  22. 	G1 Eden Space                  | -0.00MB                        | 54.00MB                        
    23. 

  23. 	CodeHeap 'non-nmethods'        | 5.57MB                         | 2.44MB                         
    24. 

  24. 

  25. 

  26. 2022-08-07 16:20:10.723 [main] INFO  Engine - 
    27. 

  27. {
    28. 

  28. 	"content":[
    29. 

  29. 		{
    30. 

  30. 			"reader":{
    31. 

  31. 				"name":"hdfsreader",
    32. 

  32. 				"parameter":{
    33. 

  33. 					"column":[
    34. 

  34. 						"*"
    35. 

  35. 					],
    36. 

  36. 					"defaultFS":"hdfs://hadoop03:8020/",
    37. 

  37. 					"encoding":"UTF-8",
    38. 

  38. 					"fieldDelimiter":"\t",
    39. 

  39. 					"fileType":"text",
    40. 

  40. 					"path":"/user/hive/warehouse/user_info/user_info_data.txt"
    41. 

  41. 				}
    42. 

  42. 			},
    43. 

  43. 			"writer":{
    44. 

  44. 				"name":"hdfswriter",
    45. 

  45. 				"parameter":{
    46. 

  46. 					"column":[
    47. 

  47. 						{
    48. 

  48. 							"name":"user_id",
    49. 

  49. 							"type":"string"
    50. 

  50. 						},
    51. 

  51. 						{
    52. 

  52. 							"name":"area_id",
    53. 

  53. 							"type":"string"
    54. 

  54. 						},
    55. 

  55. 						{
    56. 

  56. 							"name":"age",
    57. 

  57. 							"type":"int"
    58. 

  58. 						},
    59. 

  59. 						{
    60. 

  60. 							"name":"occupation",
    61. 

  61. 							"type":"string"
    62. 

  62. 						}
    63. 

  63. 					],
    64. 

  64. 					"compress":"",
    65. 

  65. 					"defaultFS":"hdfs://hadoop03:8020/",
    66. 

  66. 					"fieldDelimiter":"\t",
    67. 

  67. 					"fileName":"user_info_data_2.txt",
    68. 

  68. 					"fileType":"text",
    69. 

  69. 					"path":"/user/hive/warehouse/user_info/",
    70. 

  70. 					"writeMode":"append"
    71. 

  71. 				}
    72. 

  72. 			}
    73. 

  73. 		}
    74. 

  74. 	],
    75. 

  75. 	"setting":{
    76. 

  76. 		"speed":{
    77. 

  77. 			"channel":"2"
    78. 

  78. 		}
    79. 

  79. 	}
    80. 

  80. }
    81. 

  81. 

  82. 2022-08-07 16:20:10.734 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
    83. 

  83. 2022-08-07 16:20:10.735 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
    84. 

  84. 2022-08-07 16:20:10.736 [main] INFO  JobContainer - DataX jobContainer starts job.
    85. 

  85. 2022-08-07 16:20:10.737 [main] INFO  JobContainer - Set jobId = 0
    86. 

  86. 2022-08-07 16:20:10.750 [job-0] INFO  HdfsReader$Job - init() begin...
    87. 

  87. 2022-08-07 16:20:10.979 [job-0] INFO  HdfsReader$Job - hadoopConfig details:{"finalParameters":[]}
    88. 

  88. 2022-08-07 16:20:10.979 [job-0] INFO  HdfsReader$Job - init() ok and end...
    89. 

  89. 2022-08-07 16:20:11.575 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
    90. 

  90. 2022-08-07 16:20:11.576 [job-0] INFO  JobContainer - DataX Reader.Job [hdfsreader] do prepare work .
    91. 

  91. 2022-08-07 16:20:11.576 [job-0] INFO  HdfsReader$Job - prepare(), start to getAllFiles...
    92. 

  92. 2022-08-07 16:20:11.576 [job-0] INFO  HdfsReader$Job - get HDFS all files in path = [/user/hive/warehouse/user_info/user_info_data.txt]
    93. 

  93. 2022-08-07 16:20:11.994 [job-0] INFO  HdfsReader$Job - [hdfs://hadoop03:8020/user/hive/warehouse/user_info/user_info_data.txt]是[text]类型的文件, 将该文件加入source files列表
    94. 

  94. 2022-08-07 16:20:11.995 [job-0] INFO  HdfsReader$Job - 您即将读取的文件数为: [1], 列表为: [hdfs://hadoop03:8020/user/hive/warehouse/user_info/user_info_data.txt]
    95. 

  95. 2022-08-07 16:20:11.996 [job-0] INFO  JobContainer - DataX Writer.Job [hdfswriter] do prepare work .
    96. 

  96. 2022-08-07 16:20:12.035 [job-0] INFO  HdfsWriter$Job - 由于您配置了writeMode append, 写入前不做清理工作, [/user/hive/warehouse/user_info/] 目录下写入相应文件名前缀  [user_info_data_2.txt] 的文件
    97. 

  97. 2022-08-07 16:20:12.036 [job-0] INFO  JobContainer - jobContainer starts to do split ...
    98. 

  98. 2022-08-07 16:20:12.036 [job-0] INFO  JobContainer - Job set Channel-Number to 2 channels.
    99. 

  99. 2022-08-07 16:20:12.036 [job-0] INFO  HdfsReader$Job - split() begin...
    100. 

  100. 2022-08-07 16:20:12.036 [job-0] INFO  JobContainer - DataX Reader.Job [hdfsreader] splits to [1] tasks.
    101. 

  101. 2022-08-07 16:20:12.037 [job-0] INFO  HdfsWriter$Job - begin do split...
    102. 

  102. 2022-08-07 16:20:12.041 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://hadoop03:8020//user/hive/warehouse/user_info__e6d84b6f_0e53_4fc0_b8b4_b063711e54b6/user_info_data_2.txt__031bc8a3_7c7c_484f_af81_f539ac8c46a2]
    103. 

  103. 2022-08-07 16:20:12.041 [job-0] INFO  HdfsWriter$Job - end do split.
    104. 

  104. 2022-08-07 16:20:12.041 [job-0] INFO  JobContainer - DataX Writer.Job [hdfswriter] splits to [1] tasks.
    105. 

  105. 2022-08-07 16:20:12.046 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
    106. 

  106. 2022-08-07 16:20:12.049 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
    107. 

  107. 2022-08-07 16:20:12.050 [job-0] INFO  JobContainer - Running by standalone Mode.
    108. 

  108. 2022-08-07 16:20:12.053 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
    109. 

  109. 2022-08-07 16:20:12.057 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
    110. 

  110. 2022-08-07 16:20:12.057 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
    111. 

  111. 2022-08-07 16:20:12.062 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
    112. 

  112. 2022-08-07 16:20:12.084 [0-0-0-reader] INFO  HdfsReader$Job - hadoopConfig details:{"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
    113. 

  113. 2022-08-07 16:20:12.084 [0-0-0-reader] INFO  Reader$Task - read start
    114. 

  114. 2022-08-07 16:20:12.084 [0-0-0-reader] INFO  Reader$Task - reading file : [hdfs://hadoop03:8020/user/hive/warehouse/user_info/user_info_data.txt]
    115. 

  115. 2022-08-07 16:20:12.089 [0-0-0-writer] INFO  HdfsWriter$Task - begin do write...
    116. 

  116. 2022-08-07 16:20:12.089 [0-0-0-writer] INFO  HdfsWriter$Task - write to file : [hdfs://hadoop03:8020//user/hive/warehouse/user_info__e6d84b6f_0e53_4fc0_b8b4_b063711e54b6/user_info_data_2.txt__031bc8a3_7c7c_484f_af81_f539ac8c46a2]
    117. 

  117. 2022-08-07 16:20:12.096 [0-0-0-reader] INFO  UnstructuredStorageReaderUtil - CsvReader使用默认值[{"captureRawRecord":true,"columnCount":0,"comment":"#","currentRecord":-1,"delimiter":"\t","escapeMode":1,"headerCount":0,"rawRecord":"","recordDelimiter":"\u0000","safetySwitch":false,"skipEmptyRecords":true,"textQualifier":"\"","trimWhitespace":true,"useComments":false,"useTextQualifier":true,"values":[]}],csvReaderConfig值为[null]
    118. 

  118. 2022-08-07 16:20:12.100 [0-0-0-reader] INFO  Reader$Task - end read source files...
    119. 

  119. 2022-08-07 16:20:12.202 [0-0-0-writer] INFO  HdfsWriter$Task - end do write
    120. 

  120. 2022-08-07 16:20:12.263 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[202]ms
    121. 

  121. 2022-08-07 16:20:12.263 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
    122. 

  122. 2022-08-07 16:20:22.060 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4 records, 79 bytes | Speed 7B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
    123. 

  123. 2022-08-07 16:20:22.060 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
    124. 

  124. 2022-08-07 16:20:22.060 [job-0] INFO  JobContainer - DataX Writer.Job [hdfswriter] do post work.
    125. 

  125. 2022-08-07 16:20:22.060 [job-0] INFO  HdfsWriter$Job - start rename file [hdfs://hadoop03:8020//user/hive/warehouse/user_info__e6d84b6f_0e53_4fc0_b8b4_b063711e54b6/user_info_data_2.txt__031bc8a3_7c7c_484f_af81_f539ac8c46a2] to file [hdfs://hadoop03:8020//user/hive/warehouse/user_info/user_info_data_2.txt__031bc8a3_7c7c_484f_af81_f539ac8c46a2].
    126. 

  126. 2022-08-07 16:20:22.072 [job-0] INFO  HdfsWriter$Job - finish rename file [hdfs://hadoop03:8020//user/hive/warehouse/user_info__e6d84b6f_0e53_4fc0_b8b4_b063711e54b6/user_info_data_2.txt__031bc8a3_7c7c_484f_af81_f539ac8c46a2] to file [hdfs://hadoop03:8020//user/hive/warehouse/user_info/user_info_data_2.txt__031bc8a3_7c7c_484f_af81_f539ac8c46a2].
    127. 

  127. 2022-08-07 16:20:22.072 [job-0] INFO  HdfsWriter$Job - start delete tmp dir [hdfs://hadoop03:8020/user/hive/warehouse/user_info__e6d84b6f_0e53_4fc0_b8b4_b063711e54b6] .
    128. 

  128. 2022-08-07 16:20:22.079 [job-0] INFO  HdfsWriter$Job - finish delete tmp dir [hdfs://hadoop03:8020/user/hive/warehouse/user_info__e6d84b6f_0e53_4fc0_b8b4_b063711e54b6] .
    129. 

  129. 2022-08-07 16:20:22.079 [job-0] INFO  JobContainer - DataX Reader.Job [hdfsreader] do post work.
    130. 

  130. 2022-08-07 16:20:22.079 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
    131. 

  131. 2022-08-07 16:20:22.080 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /home/zhouze/PycharmProjects/yili/yili-portal/app/datax/hook
    132. 

  132. 2022-08-07 16:20:22.182 [job-0] INFO  JobContainer - 
    133. 

  133. 	 [total cpu info] => 
    134. 

  134. 		averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
    135. 

  135. 		-1.00%                         | -1.00%                         | -1.00%
    136. 

  136.                         
    137. 

  137. 

  138. 	 [total gc info] => 
    139. 

  139. 		 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
    140. 

  140. 		 G1 Young Generation  | 5                  | 5                  | 5                  | 0.046s             | 0.046s             | 0.046s             
    141. 

  141. 		 G1 Old Generation    | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
    142. 

  142. 

  143. 2022-08-07 16:20:22.182 [job-0] INFO  JobContainer - PerfTrace not enable!
    144. 

  144. 2022-08-07 16:20:22.182 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4 records, 79 bytes | Speed 7B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
    145. 

  145. 2022-08-07 16:20:22.183 [job-0] INFO  JobContainer - 
    146. 

  146. 任务启动时刻                    : 2022-08-07 16:20:10
    147. 

  147. 任务结束时刻                    : 2022-08-07 16:20:22
    148. 

  148. 任务总计耗时                    :                 11s
    149. 

  149. 任务平均流量                    :                7B/s
    150. 

  150. 记录写入速度                    :              0rec/s
    151. 

  151. 读出记录总数                    :                   4
    152. 

  152. 读写失败总数                    :                   0
    153. 

  153.