[texhax] Strange behavior of \scantokens

[Oleg Katsitadze]

Dear TeX hackers,

Can anybody please explain why the following code:

  \catcode`\^^M=13 \scantokens{x\endinput}x\bye

inserts a newline between the two x's?  What makes this really
mysterious is that when I change it to

  \scantokens{x\endinput}x\bye

I don't get a space between the x's.

I'm running this through

  $ pdftex --version
  pdfTeX using libpoppler 3.141592-1.40.3-2.2 (Web2C 7.5.6)
  kpathsea version 3.5.6

TIA for any clues,
Oleg

P.S.  FWIW, when I emulate \scantokens under vanilla tex, it behaves
exactly the same, so I suspect it's something about how TeX's \input
works.  My emulation is (from Texinfo via Eplain):

  \def\tmpfileextension{.tmp}%
  \let\tmpfilebasename = \jobname
  \newwrite\eplain at tmpfile
  \def\scantokens#1{%
    \toks@={#1}%
    \immediate\openout\eplain at tmpfile=\tmpfilebasename\tmpfileextension
    \immediate\write\eplain at tmpfile{\the\toks@}%
    \immediate\closeout\eplain at tmpfile
    \input \tmpfilebasename\tmpfileextension\relax
  }%