How fast does particle 2 fly away from particle 1's viewpoint? Use the addition law of velocities to get
What is the penalty for looking at the decay in the (m/3)'s b = 0.8 frame?
which implies m = (1.333/1.809)M = 0.737034M.
Ha! The "lost mass" is the "binding energy", the mass equivalent of the kinetic energies of the particles.
Conservation of energy states
and we have M = (2/3)m*1.20185 + (1/3)m*1.6667