How data is constructed in files created by teragen in Hadoop?-CodePudding

I'm trying to understand the details of terasort. After using the Hadoop's teragen, I can successfully create the input data files.

Many online documents say that each row contains data in the following format:

<10 bytes key><10 bytes rowid><78 bytes filler>\r\n

Therefore, I downloaded the first part-file part-m-00000, and then use the following command to cut the rowid out of it:

cut -nb 11-20 part-m-00000 | od -c -w10 -Ad

However, here is what I got:

0000000  \0 021   0   0   0   0   0   0   0   0
0000010  \n   0   0   0   0   0   0   0   0   0
0000020   0  \n 021   0   0   0   0   0   0   0
0000030   0   0  \n   0   0   0   0   0   0   0
0000040   0   0   0  \n 021   0   0   0   0   0
0000050   0   0   0   0  \n 021   0   0   0   0
0000060   0   0   0   0   0  \n   0   0   0   0
0000070   0   0   0   0   0   0  \n 021   0   0
0000080   0   0   0   0   0   0   0  \n   0   0
0000090   0   0   0   0   0   0   0   0  \n   0
0000100   0   0   0   0   0   0   0   0   0  \n

Apparently, there is nothing like the increased row ids, that I expect in the result. I wonder what did I do wrong?

CodePudding user response：

I am not sure what online documents you're quoting, but the comment in TeraGen source code clearly states the format of generated records:

/**
 * Generate the official GraySort input data set.
 * The user specifies the number of rows and the output directory and this
 * class runs a map/reduce program to generate the data.
 * The format of the data is:
 * <ul>
 * <li>(10 bytes key) (constant 2 bytes) (32 bytes rowid) 
 *     (constant 4 bytes) (48 bytes filler) (constant 4 bytes)
 * <li>The rowid is the right justified row id as a hex number.
 * </ul>
 *
 :

And, teragen produces rows that fully comply with the above format. In hex, they look like this (32-byte long rowids are marked by me with a "<<" :))

[hdfs@mygateway ~]# xxd part-m-00000 | head -40
0000000: 4a69 6d47 7261 7952 4950 0011 3030 3030  JimGrayRIP..0000  << rowid 0
0000010: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
0000020: 3030 3030 3030 3030 3030 3030 8899 aabb  000000000000.... 
0000030: 3232 3232 3030 3030 3232 3232 3030 3030  2222000022220000
0000040: 3232 3232 3030 3030 3232 3232 3030 3030  2222000022220000
0000050: 3232 3232 3030 3030 3030 3030 3131 3131  2222000000001111
0000060: ccdd eeff 95e0 e482 62b3 edfe 0447 0011  ........b....G..
0000070: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000 << rowid 1 
0000080: 3030 3030 3030 3030 3030 3030 3030 3031  0000000000000001
0000090: 8899 aabb 3939 3939 3434 3434 3838 3838  ....999944448888
00000a0: 3535 3535 4343 4343 3737 3737 3535 3535  5555CCCC77775555
00000b0: 3535 3535 4242 4242 3636 3636 3434 3434  5555BBBB66664444
00000c0: 3636 3636 ccdd eeff 4597 9353 dbed 5329  6666....E..S..S)
00000d0: 365c 0011 3030 3030 3030 3030 3030 3030  6\..000000000000 << rowid 2
00000e0: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
00000f0: 3030 3032 8899 aabb 4646 4646 3737 3737  0002....FFFF7777 
0000100: 3939 3939 3535 3535 4646 4646 3131 3131  99995555FFFF1111
0000110: 3232 3232 3939 3939 3838 3838 3434 3434  2222999988884444
0000120: 4444 4444 4646 4646 ccdd eeff 882a 02c3  DDDDFFFF.....*..
0000130: 1536 2b60 765f 0011 3030 3030 3030 3030  .6 `v_..00000000 << etc. etc.
0000140: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
0000150: 3030 3030 3030 3033 8899 aabb 3131 3131  00000003....1111
0000160: 3030 3030 3030 3030 3030 3030 3636 3636  0000000000006666
0000170: 3838 3838 4242 4242 3333 3333 3333 3333  8888BBBB33333333
0000180: 4141 4141 3131 3131 4343 4343 ccdd eeff  AAAA1111CCCC....
0000190: 5c90 ab38 ae52 8962 15d7 0011 3030 3030  \..8.R.b....0000
00001a0: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
00001b0: 3030 3030 3030 3030 3030 3034 8899 aabb  000000000004....
00001c0: 4141 4141 4242 4242 4242 4242 3232 3232  AAAABBBBBBBB2222
00001d0: 3434 3434 3434 3434 3939 3939 3535 3535  4444444499995555
00001e0: 4242 4242 3131 3131 3838 3838 4444 4444  BBBB11118888DDDD
00001f0: ccdd eeff 72dc 0ca5 1e33 3f32 4b7a 0011  ....r....3?2Kz..
0000200: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
0000210: 3030 3030 3030 3030 3030 3030 3030 3035  0000000000000005
0000220: 8899 aabb 3838 3838 3333 3333 4242 4242  ....88883333BBBB
0000230: 3838 3838 3838 3838 3434 3434 3737 3737  8888888844447777
0000240: 3232 3232 3737 3737 3939 3939 3030 3030  2222777799990000
0000250: 3232 3232 ccdd eeff 1043 1af6 a0d8 47b8  2222.....C....G.
0000260: c55f 0011 3030 3030 3030 3030 3030 3030  ._..000000000000
0000270: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000

If you're interested, you can also inspect how the rows are generated (constants used, etc.) in GenSort.java.