TOAST - why separate visibility map

Why do Toast tables have it's own visibility map and xmin, xmax columns etc?
Isn't it increasing row size in a toast table and adding more complexity?

Ideally all the vacuum cleanup on a TOAST can be done based on Primary
table xmin,xmax and VM info. Yes, that makes any cleanup on TOAST to be
glued up with the Primary table.