Thanks for the response. That was an old article but it did lead me to the fix. We had an issue with the output filter on the upgrade and support had told us to comment out the entry in the web config that matched #3 in the article. So after further investingation I found that changing that entry from:
<add name="XHtmlModule" type="CMS.CMSOutputFilter.OutputFilterModule, CMS.OutputFilter" />
to:
<add name="XHtmlModule" type="CMS.OutputFilter.OutputFilterModule, CMS.OutputFilter" />
fixed the output filter issue we were seeing and changed the tilde (~) in my urls to a double slash (//). I was still getting 404 errors on my images but this was different anyway. Then I went to Configuration > Settings > System > Output Filter and set "Excluded XHTML Output Filter Urls" to '/' so it would not filter any Urls and this seems to have fixed it. I don't know yet if this will cause us any issues down the road but it looks good so far.