In wireshark, make sure you capture the packets on local LAN. There are different interfaces, that might explain why you did not get any packets on LAN.
Here is my payload, with correct rcon message sent on my LAN, packets captured on interface with IP 127.0.0.1, i truncated first bytes, starting at 0020 each.
client to server:
0020 00 01 c6 9d 6d 38 00 26 fe 39 [B]ff ff ff ff[/B] 72 63 ....m8.& .9[B]....[/B]rc
0030 6f 6e 20 6d 79 72 63 6f 6e 70 77 20 64 65 76 6d on myrco npw devm
0040 61 70 20 6f 61 73 69 73 ap oasis
I marked the head \xFF stuff with bold. On left this is “ff ff ff ff” and on right these are just 4 points, because wireshark can’t print those (translate to ASCII).
Here is the reply of this message from server:
server to client:
0020 00 01 6d 38 c6 9d 00 e3 fe f6 ff ff ff ff 70 72 ..m8.... ......pr
0030 69 6e 74 0a 5e 32 3c 45 6e 64 47 61 6d 65 3e 0a int.^2<E ndGame>.
0040 5e 32 2d 2d 20 53 63 72 69 70 74 20 53 79 73 74 ^2-- Scr ipt Syst
0050 65 6d 20 49 6e 66 6f 20 2d 2d 0a 5e 32 43 75 72 em Info --.^2Cur
0060 72 65 6e 74 20 4d 65 6d 6f 72 79 20 55 73 61 67 rent Mem ory Usag
0070 65 20 31 2e 35 36 37 36 32 20 4d 42 0a 5e 32 53 e 1.5676 2 MB.^2S
0080 6f 66 74 20 4d 65 6d 6f 72 79 20 55 73 61 67 65 oft Memo ry Usage
0090 20 31 2e 36 20 4d 42 0a 5e 32 48 61 72 64 20 4d 1.6 MB. ^2Hard M
00a0 65 6d 6f 72 79 20 4c 69 6d 69 74 20 32 20 4d 42 emory Li mit 2 MB
00b0 0a 5e 32 53 79 73 74 65 6d 20 4d 65 6d 6f 72 79 .^2Syste m Memory
00c0 20 55 73 61 67 65 20 31 2e 37 35 35 30 34 20 4d Usage 1 .75504 M
00d0 42 0a 5e 32 46 75 6c 6c 20 43 6f 6c 6c 65 63 74 B.^2Full Collect
00e0 73 20 31 0a 5e 32 49 6e 63 20 43 6f 6c 6c 65 63 s 1.^2In c Collec
00f0 74 73 20 33 0a 5e 32 47 43 20 57 61 72 6e 69 6e ts 3.^2G C Warnin
0100 67 73 20 30 0a gs 0.
Here with bad rcon pw:
client to server:
0020 00 01 eb 69 6d 38 00 24 fe 37 ff ff ff ff 72 63 ...im8.$ .7....rc
0030 6f 6e 20 6d 79 6f 6e 70 77 20 64 65 76 6d 61 70 on myonp w devmap
0040 20 6f 61 73 69 73 oasis
server to client:
0020 00 01 6d 38 eb 69 00 24 fe 37 ff ff ff ff 70 72 ..m8.i.$ .7....pr
0030 69 6e 74 0a 42 61 64 20 72 63 6f 6e 70 61 73 73 int.Bad rconpass
0040 77 6f 72 64 2e 0a word..
Like zero said, it looks like the header is wrong. This might be a lua encoding problem. The ‘\x’ escape char is probably not translated.