Explainable artificial intelligence (XAI) has been widely advocated as a solution to the opacity of machine learning models in pharmaceutical prediction, yet the connection between explanation, trust, reproducibility, and scientific validity remains unresolved. A growing body of literature applies explanation methods across drug discovery, ADMET prediction, formulation design, and clinical pharmacology; however, much of this work assumes that making predictions visually or numerically interpretable inherently confers trustworthiness. This critical review examines the strengths, weaknesses, and ongoing challenges of XAI in pharmaceutical contexts, with particular focus on user trust, reproducibility of explanations, and suitability for regulated decision-making. The literature highlights persistent gaps, including a lack of human-centered evaluation, limited assessment of explanation stability, and a misalignment between common explanation outputs and regulatory expectations. While many studies present plausible explanations, far fewer demonstrate that these explanations meaningfully improve decisions. Without rigorous validation, XAI risks obscuring rather than clarifying model behavior, and in high-stakes pharmaceutical settings, intuitive but non-robust explanations may foster misplaced confidence. This review therefore proposes a framework for assessing the maturity of XAI in pharmaceutical prediction, emphasizing the need to advance from appealing explanatory artifacts toward reproducible, uncertainty-aware, and decision-tested explanation systems.