Found the issue.
The attachment link that is provided by clicking the link to the attachment through the attachments tab is NOT the correct url to use. That url requires login.
If you add the image to an editable text, it gives a different attachment url format that doesn't require auth.
/cms/getattachment/47f60573-166e-4b44-9983-2d9718fa45c1/half-placeholder.png.aspx?chset=68b9eddd-2998-4a09-b160-bfb31bc0fba1 <- This one requires login
/getattachment/Test-Page/half-placeholder.png.aspx?lang=en-US <- This doesn't