While Microsoft has done a lot around Gaming & Windows 10, the feature isnt without bugs, and performance issue. I have seen many talking about that Gaming Mode desont really do anything substantial, and now Microsoft has acknowledged FPS drops in games on the Windows 10 Creators Update. There are several different problem sources that is casuing this FPS drop, and if you can, Microsoft claims to have fixed it.In a response to Feeback Hub, Microsoft team claims that one of the issues has been fixed which means, it still isnt fixed. The issue has been fixed in one of the Fast Insider Ring build 16273 and above. The rest of the sources are being investigated.
That said, Xbox DVR feature through the Xbox App is something that still needs lot of attention. I have played a couple of games, and streaming a game from Xbox One to PC is stil terrible, and even with decent router it does not work well either. Game DRV is in built into the system as well, and while you may try turning it off, I am pretty sure its not going to fix something with multiple issues casuing the trouble.
If you are facing the same issue, expect it to sort out in Widnows 10 Fall Creators Update which is rolling out from October 17.