Possible solution 1
Your footage below the transition is a huge image scaled down (i.e. a scan or wallpaper .jpeg). If this is the case, you may need to precompose your image (Layer-> Pre-compose) with these settings:
This helps because transition is then applied to an already scaled down layer so it doesn't need extra memory.
Possible solution 2
Lower your project's bit depth to 16bpc or 8bpc. Unfortunately, some transitions might be quite demanding and use a lot of memory, especially in 4k resolutions and 32bpc setttings.
Possible solution 3
Are you having problems with rendering in Media Encoder? Check out this article.