Le problème est la position de `\small` :
\documentclass{article}
% \usepackage{lua-visual-debug}
\usepackage[paperheight=6.4in, paperwidth=4.8in, margin=.25in]{geometry}
%\usepackage{fontspec}
%\setmainfont{EBGaramond-Regular}
\usepackage{fontspec}
\setmainfont{EBGaramond-Regular}
\usepackage{tabularx}
\usepackage{multirow}
\begin{document}
\begin{center}
\small
\begin{tabularx}{1.15\paperwidth}{p{.50\paperwidth} p{2em} p{.50\paperwidth}}
%\small
\multicolumn{1}{c}{some text} & \multirow{5}{2em}{|| ||} & \multicolumn{1}{c}{some text} \\
& & \\
& & \\
& & \\
& & \\
\end{tabularx}
\end{center}
\normalsize
\end{document}
