One could argue that it was the juice from those links that was helping you before, and while Google isn't penalizing you for having those links, they've taken away any benefit you were getting from them. If that's the case, then you're now where you'd be if you had disavowed the links.
OK, so it probably be a waste of time to disavow the links then? Gosh, it's so confusing to know what to do, and I have to do something. Should I at least disavow the cheap domains that have 100's of spammy site-wide links (and all the same anchor text) to my site? It's hard to know if I am getting SOME benefit from them, none at all, or a penalty. If I knew I could act accordingly. If I thought SOME sort of penalty was being applied then I could disavow the ones I most distrust.
WOULD YOU TRY THAT? ie at least disavow the most spammy ones? or am I just wasting time.