Were you using this when creating new pages? If so, my best guess would be to amend the document name (as URL slug should be based on it) before inserting a new document, in a custom module:
protected override void OnInit()
{
base.OnInit();
DocumentEvents.Insert.Before += InsertOnBefore;
}
private void InsertOnBefore(object sender, DocumentEventArgs e)
{
e.Node.DocumentName = e.Node.DocumentName.Replace("ö", "oe");
}
I haven't tested it myself, but hope it should work.