To remove the "Try the new Outlook" button, you just need to create a new Registry entry called HideNewOutlookToggle.