I'll attempt an answer that addresses the spirit of the question. You're not simply seeking the most pragmatic way to transcribe, but hoping to understand the music theory and idioms that cause musicians to make the improvisatory choices they do. Even if the end goal is transcription, this is not an unreasonable desire; I can transcribe a simple Mozart phrase more easily if I have the most rudimentary understanding of cadences and know that a IV is probably followed by a V and a I, and I don't have to painstakingly hear every single note in a chord.
Unfortunately, the best way to "reverse engineer" is simply to "forward engineer." The suggestion to "just learn all the scales, then play around with them and see what feels good" doesn't give you the analytical detail you're looking for, but it's a good place to start. Ultimately though, you'll probably benefit by identifying the chord changes (and overall tonality) that the improviser is soloing over. A solo doesn't have to constrain itself to these chords, but if it departs from them, it usually does so by conscious choice.
As mentioned, "find the key of the song" is the first step, but definitely not enough. "Long Distance Runaround" by Yes is overall in E major, but the riff that the instrumental intro is constructed around is in G natural minor (maybe. Plus some funky whole-tone stuff.). You'll also want to identify the actual chords going on at the moment of the lick in question.
Next, while one improv tool is to simply play your way up or down a scale, and while one might choose from multiple such scales (e.g. if the underlying chord is A minor, you might play an A minor, pentatonic, dorian, or blues scale), it can get boring and you might aspire to break into something more interesting. You could of course arpeggiate across chords ("Giant Steps," anyone?), or by mixing up your up-vs-down directions and the size of your leaps, before long you're just constructing a melody.
Since we're reverse-engineering here, listen for notes that "stick out" from the underlying chord. For instance, in the intro to "Heart of the Sunrise" (can you tell I like Yes?), they do simply run up and down an A pentatonic minor scale, but they fill in a few "gaps" chromatically, adding a C# and D#. Any non-chordal tones, bends, blue notes, etc. that "clash" with the underlying chord can point to what structural element they came from or are going to.
And finally, a smart solo-er will keep in mind where they're headed. You don't want to just run out of time and be left hanging at a point that doesn't mean anything. You want to aim for a final chord (maybe there's even a modulation involved!). You might want to wind up in a certain register—do you want to "go out on a high note," with a Dizzy-Gillespie squeal? Did you build up from a low register, but need to get back there to resume the rest of the song?
In the Black Sabbath lick in question, the lick is only a few notes long, and we can hear that it descends. The first thing I notice off the bat is that it covers an octave (seems to start on an A and end on the A an octave below), and to move in a mostly scalar fashion (i.e., it mostly goes down one step at a time, but not entirely, there are some intervallic skips). At that point, I hear the main notes as A G E D C A, which is pentatonic. There are some other squiggles, squeals, and bends going on, but now I have a starting point.
