Reliable and robust defect detection is a challenging yet essential problem in ultrasonic non- destructive evaluation (NDE). The flaw echoes are usually contaminated by high-level, time- invariant, and correlated grain noise originating from the material microstructure. This phenomenon becomes even worse when inspecting coarse grain materials like stainless steels, alloys, carbon-reinforced composites and concrete, which however, forms some of the most important and commonly used industrial materials. A wide variety of techniques have been investigated to enhance flaw detection by exploring either the spatial diversity, e.g. transducer array beamforming, or the time-frequency characteristics of the broadband ultrasonic signals, e.g. matched filtering. The matched filter is known to be optimal in terms of the SNR improvement if the signal waveform and noise statistics are exactly known a priori; this is unfortunately not the case in ultrasonic NDE. Earlier studies use simulated flaw signals [1] or the superposition of real transmitted signals [2] to design the filter, but they are inevitably subject to significant errors, especially in NDE of coarse grain materials, or not straightforward to implement in practice. In this paper, we propose a robust method to optimal matched filter design. The transmit waveform is modelled as a broadband signal centered at the transducer frequency; the flaw echo is modelled as the superposition of multiple transmit waveforms with different parameters like time delay, amplitude gain, and phase shift; and an optimization paradigm is utilized to search the optimal parameters in the above model with an objective to maximize the SNR improvements over a set of training signals captured from real samples. Different from the traditional matched filters used in radar, our filter is designed to match the flaw echoes rather than the transmitted signals. As a result, the optimal matched filter depends on the material characteristics and is supposed not identical for different type of materials. The design is validated with experiments on austenitic stainless steel and nickel-steel alloy samples, which demonstrates more than 20dB improvement on the SNR of A-scan waveforms, as well as in the images created by the total focusing method (TFM) with filtered data. The additional computational cost in implementation of the matched filter is low, which makes it suitable for real-time applications.