FA+
Shop
[FFXI] Enspell vs. Enspell II (TEXT WALLS & LOTS OF MATH!)
15 years ago
Some folks over on my Twitter feed have requested that I spell out the exact situation that led to a very recent mathematical frustration of mine. Since the story is WAY too long for Twitter, I decided to use this (particularly since the interested parties are furries, anyway).
Anyway, in the game, Final Fantasy XI, I play a Red Mage. Or at least, I say I do. In practice it's been sitting at 30 for forever now because I slack off like a boss on my leveling (I want to get my White Mage and Paladin, each currently at 40, to 50 first, so I may use them as support jobs while leveling Red Mage to the new level cap of 99). So, admittedly, a lot of the big talk I may spout off at times is really all theory and no practice. I'm called out for this a lot on the forums I post on, so there's some motivation to get my butt to this dang "endgame" thing . . . whenever I find the time, that is. That being said, the issue I am posting about is, in my case, "theoretical," but for other Red Mages like myself that dare take up their swords, knowing which supporting spell is "better" would be very practical indeed.
Which leads me to the title of this post: A Red Mage's Enspell versus their Enspell II. The reason I got myself in to a mathematical quandary in the first place was because another "melee Red Mage," whom I highly admire and respect, said that the Enspell II line was "brokenly horrible." Naturally, calling the second-tier of a spell line inferior to its first tier didn't sit well with me, though this required me to do a lot of math because the way the damage on each spell is calculated, Enspells II are not, frustratingly enough, strictly superior to first-tier Enspells, either.
First, some conceptual groundwork before I get in to the actual math: "Enspells" refer to a line of White Magic spells exclusive to the Red Mage job. Basically, what they do is add some amount of elemental damage to one's base melee attacks. Enblizzard gives you additional Ice damage, Enfire gives you additional Fire damage, Enwater gives you additional Water damage, and so on. This general concept applies both to first-tier Enspells and Enspells II, and helps give Red Mage more of a flavor as the game's first magic knight. Having said that, let's go in to the differences between how the damage on each is calculated and where it is applied.
First-tier Enspell damage is simple enough. The base damage of the spell (that is, how much damage the spell adds on to your attack) is calculated upon casting based on a piecewise function of the Red Mage's Enhancing Magic Skill. At Skill > 200, Enspell base damage = Skill / 20 + 5. This damage is added as elemental damage to all melee attacks you make. Simple enough - the more you hit with this up, the more straight-up damage you get. This can also be boosted with (albeit rather rare) "Sword Enhancement Spell Damage" boosting equipment (though technically, you'll receive the bonus regardless of what kind of weapon you use).
Enspells II are different, to the point where some of the differences are just plain annoying. The base damage of the spell is calculated based on the same function, except each time the Enspell II damage is applied, the damage is increased by one, up to a cap of 2 times your base Enspell damage. To give further incentive for using these, an elemental resistance penalty is also applied to an enemy hit with the spells's effect for whichever element the spell is weak against (for example, Enstone II weakens resistance to Wind effects), which is nice not only for a Red Mage looking to make sure their Enfeebles stick without crippling their melee capabilities with a Staff, but is also a boon to any other magic-using job in the party looking to attack using a particular element.
So, pretty sweet, right? Except Enspells II come with a catch: they only add their damage to the *first* attack of each "attack round." In a more mundane situation, this wouldn't matter so much, as attack rounds (events after your melee weapon Delay timer counts down) normally only have one attack in them. However, for those using Warrior subjob and/or Brutal Earring for Double Attack or Joyeuse (a weapon that "Occasionally attacks twice," ~45% of the time), common old strategies of boosting damage output, this is a problem. This was, in all likelihood, done to reduce exploitation of the rare and coveted Kraken Club ("Occasionally attacks 2 to 8 times" . . . yeah), but this kinda screws everybody that uses multi-attack, all in the name of balance. Less explicitly, however, there is another, stupider catch to Enspells II, which is what I was focusing mainly on for my mathematical calculations. Basically, the "Enspell base damage" is calculated based on your Enhancing Magic Skill . . . during each individual swing. Which means that when you cast Enwater II in your Enhancing Magic gear set, when you switch to your melee gear set, you're losing a couple of points of Enspell damage to make sure your base melee damage doesn't suck, as opposed to first-tier Enspells, where your base damage is locked in upon casting. Oh, and remember that "Sword Enhancement Spell Damage" gear? It still applies to Enspells II, though it's applied as a constant and does not eventually double like the base damage does.
Based on these descriptions, I've come up with two equations to use to determine which Enspell would give more damage (for simplicity's sake, this is assuming the Red Mage is an idiot and does nothing but constantly swing his weapon for the duration of the spell; realistically, some time's going to be taken out for your regular spellcasting duties):
First-tier Enspell = (Enhancing Magic Skill/20 + 5 + Sword Enhancement Spell Damage Bonus) * Duration of the spell in seconds * 60 /Delay of your weapon * Average number of attacks per attack round
(Note that the Delay number, as listed on a melee weapon, is 60 times the number of seconds in between attack rounds. For example, a Sword with 240 delay has an attack round once every four seconds. The number is rarely this precise, though, as there are also Swords with 233 Delay, 223 Delay, and so on.)
Enspell II = ((3/2)(EMS/20 + 5) + SESDB) * (EMS/20 + 5) * Delay/60 + (2(EMS/20 + 5) + SESDB) * (Duration of the spell in seconds * 60/Delay - (EMS/20 + 5) * Delay/60)
This calculation is a crapload less intuitive, and is based on quite a few assumptions, so let me explain it: for a number of attack rounds (Delay/60) equal to your base Enspell damage (because it's increasing at a rate of 1 more damage per attack round and will eventually double, EMS/20 + 5), you will be dealing an average amount of damage equal to (3/2)(EMS/20 + 5) + SESDB. This is a mathematical shortcut. To justify it with an example, add together 18 + 36, then 19 + 35, and so on, and then divide each value by two. This should result in 18 instances of the value 18 * 3/2, and should work with any integer, assuming that the Red Mage produces enough Enspell II hits to reach the damage cap. As with the first-tier Enspell equation, it's also assuming an Enhancing Magic Skill > 200, but following the "enough hits to reach the damage cap" assumption, there is also the implicit assumption that your Enhancing Magic Skill isn't too high for that to be possible at your current Delay (Enhancing Magic Skill <= 20(Duration in seconds * (60/Delay)^2 - 5), or else the second term (the amount of damage dealt while you're at the Enspell damage cap) ends up negative, when in reality the smallest this term can actually be is 0).
Also, I just realized that I forgot to account for Haste when I first did this. Crap. Well, Haste speeds up attacks by 1/(1 - Haste%), so I suppose you'd multiply each term in the equations by that as well to get the total amount of potential damage added for the duration of the spell. This is kind of inexcusable, too, since by the time one could cast Enspell II, one can and should cast Haste as well. Ah, well, I guess this hypothetical Red Mage is an idiot anyway for assuming they're not going to be casting anything while they have the Enspell effect . . .
As I mentioned earlier, "Sword Enhancement Spell Damage" bonuses are applied independently, but I managed to work out how one can separate this term from the equations entirely and then add it where appropriate:
Total damage added from SESDB = SESDB * Duration of the spell in seconds * 60/Delay (* Average number of attacks per attack round, if this is a first-tier Enspell)
What I wanted to do first of all was figure out how much additional Enhancing Magic Skill from an Enhancing Skill set would require to be lost from a first-tier Enspell to outdo an Enspell II, assuming a single-attack weapon with 240 Delay and only Brutal Earring providing 5% Double Attack (Average number of attacks per round = 1.05) and no Sword Enhancement Spell Damage gear. I would roughly estimate this to favor the Enspell II, personally, but my goal was to use something simple yet realistic enough. (Though, again, it's not at all that realistic to assume 0% Haste, but anyway . . . ) Base durations of all Enspells is 3 minutes, though by the time one can use Enspells II one can also take advantage of Composure, which triples the duration of self-cast White and Black Enhancing Magic (and also provides +15 melee Accuracy, which is kinda nice, but I'm also implicitly assuming 100% hit rate . . .). So Duration in seconds = 540. I trying solving for one variable once, but all that was good for was revealing the failure of my Enspell II damage calculation at high values of Enhancing Magic Skill (which, by the way, for 240 Delay and no Haste, EMS < 575). So I made it two variables, X being Enhancing Magic Skill in a dedicated Enhancing Magic Skill gear set, and Y being Enhancing Magic Skill in your regular melee set, and attempted to solve for X - Y. Here's what I got:
(X/20 + 5)(540 * 60/240)(21/20) = (3/2)(Y/20 + 5)(Y/20 + 5)(240/60) + 2(Y/20 + 5)(540 * 60/240 - (Y/20 + 5)(240/60))
From here, I managed to work (with pencil, paper, and calculator) that monster to . . . this monster:
Y^2 - 19600Y/37 - 912500/37 = -14175X/37
As one might guess, I was in the middle of completing the square when I posted my frustration to Twitter, where akkeresu told me "ohai, here's some Wolfram Alpha for ya." (http://bit.ly/bwWb6V) Which was awesome! Except it told me that I basically ran in to a dead end, and cannot solve for X - Y. =/
I mean, next time I'd probably input different different theoretical values, maybe getting rid of the Brutal Earring Double Attack to simplify things, and adding at least 15% Haste for realism, but I can tell right away that it wouldn't alleviate the problem of me not being able to find what I'm looking for. =/ Too many dang variables, I swear (not even mentioning the ridiculous "100% hit rate and constantly swinging" assumptions)!
So, because the aforementioned fox-cat wanted to know precisely what I was looking for, well, now ya know. =/
Ultimately, I want to know, even if you have to assume an arbitrary 240 Delay, how high the multi-attack rate has to be and/or how great the difference in Enhancing Magic Skill counted has to be in order for a first-tier Enspell to completely outdo an Enspell II. If the parameters needed were easily attainable for a high-end Red Mage (and didn't gimp regular melee damage to heck), then Enspells II would, in fact, be "brokenly horrible." =/
Oh, and mandatory addendum: "melee Red Mage" bashing will not be condoned. I'm well a-freaking-ware there are times and places where it isn't applicable and what "I'm invited for." This is neither the time nor the place for that. The equations assume the Red Mage is a moron, anyway, so just assume he's soloing and pull your panties out of your butt, seriously. This is strictly for inquiring minds who want to know.
Addendum the second - I'm also further doubting my math on some of this, so if you see any potential conceptual mistakes, please let me know . . .
EDIT FOR LONGEVITY:
Enspell = (Enhancing Maic Skill/20 + 5) * Amount of Use in Seconds per Spell * 60/Delay * 1/(1 - Haste%) * Average Attacks Per Rround + tSESD * AAPR
Enspell II = (3/2)(EMS/20 + 5) * (EMS/20 + 6) + (2[EMS/20 + 5]) * (AUSS - [EMS/20 + 6] * Delay/60 * [1 - Haste%]) * 60/Delay * 1/(1 - Haste%) + tSESD
tSESD = "Sword Enhancement Spell Damage" * AUSS * 60/Delay * 1/(1 - Haste%)
To find added DPS for Enspell or tSESD, simply omit Duration In Seconds. Because Enspell II first builds up before staying at a constant rate, the total DPS of the spell will vary with how much it is used per cast. The total damage equation for Enspell II also assumes the second term to not be negative - that is, enough attacks connect to reach the damage cap. Naturally, all DPS values are subject to hit rate, but your results may also vary with enemy magic resistance as well as the current weather. As always as a Red Mage, use the appropriate spell as per your discretion.
I've basically given up on trying to find some simpler equation for determining which spell is "better." The best one can do now is plug in appropriate values in to each equation to determine what the situation at hand calls for.
Anyway, in the game, Final Fantasy XI, I play a Red Mage. Or at least, I say I do. In practice it's been sitting at 30 for forever now because I slack off like a boss on my leveling (I want to get my White Mage and Paladin, each currently at 40, to 50 first, so I may use them as support jobs while leveling Red Mage to the new level cap of 99). So, admittedly, a lot of the big talk I may spout off at times is really all theory and no practice. I'm called out for this a lot on the forums I post on, so there's some motivation to get my butt to this dang "endgame" thing . . . whenever I find the time, that is. That being said, the issue I am posting about is, in my case, "theoretical," but for other Red Mages like myself that dare take up their swords, knowing which supporting spell is "better" would be very practical indeed.
Which leads me to the title of this post: A Red Mage's Enspell versus their Enspell II. The reason I got myself in to a mathematical quandary in the first place was because another "melee Red Mage," whom I highly admire and respect, said that the Enspell II line was "brokenly horrible." Naturally, calling the second-tier of a spell line inferior to its first tier didn't sit well with me, though this required me to do a lot of math because the way the damage on each spell is calculated, Enspells II are not, frustratingly enough, strictly superior to first-tier Enspells, either.
First, some conceptual groundwork before I get in to the actual math: "Enspells" refer to a line of White Magic spells exclusive to the Red Mage job. Basically, what they do is add some amount of elemental damage to one's base melee attacks. Enblizzard gives you additional Ice damage, Enfire gives you additional Fire damage, Enwater gives you additional Water damage, and so on. This general concept applies both to first-tier Enspells and Enspells II, and helps give Red Mage more of a flavor as the game's first magic knight. Having said that, let's go in to the differences between how the damage on each is calculated and where it is applied.
First-tier Enspell damage is simple enough. The base damage of the spell (that is, how much damage the spell adds on to your attack) is calculated upon casting based on a piecewise function of the Red Mage's Enhancing Magic Skill. At Skill > 200, Enspell base damage = Skill / 20 + 5. This damage is added as elemental damage to all melee attacks you make. Simple enough - the more you hit with this up, the more straight-up damage you get. This can also be boosted with (albeit rather rare) "Sword Enhancement Spell Damage" boosting equipment (though technically, you'll receive the bonus regardless of what kind of weapon you use).
Enspells II are different, to the point where some of the differences are just plain annoying. The base damage of the spell is calculated based on the same function, except each time the Enspell II damage is applied, the damage is increased by one, up to a cap of 2 times your base Enspell damage. To give further incentive for using these, an elemental resistance penalty is also applied to an enemy hit with the spells's effect for whichever element the spell is weak against (for example, Enstone II weakens resistance to Wind effects), which is nice not only for a Red Mage looking to make sure their Enfeebles stick without crippling their melee capabilities with a Staff, but is also a boon to any other magic-using job in the party looking to attack using a particular element.
So, pretty sweet, right? Except Enspells II come with a catch: they only add their damage to the *first* attack of each "attack round." In a more mundane situation, this wouldn't matter so much, as attack rounds (events after your melee weapon Delay timer counts down) normally only have one attack in them. However, for those using Warrior subjob and/or Brutal Earring for Double Attack or Joyeuse (a weapon that "Occasionally attacks twice," ~45% of the time), common old strategies of boosting damage output, this is a problem. This was, in all likelihood, done to reduce exploitation of the rare and coveted Kraken Club ("Occasionally attacks 2 to 8 times" . . . yeah), but this kinda screws everybody that uses multi-attack, all in the name of balance. Less explicitly, however, there is another, stupider catch to Enspells II, which is what I was focusing mainly on for my mathematical calculations. Basically, the "Enspell base damage" is calculated based on your Enhancing Magic Skill . . . during each individual swing. Which means that when you cast Enwater II in your Enhancing Magic gear set, when you switch to your melee gear set, you're losing a couple of points of Enspell damage to make sure your base melee damage doesn't suck, as opposed to first-tier Enspells, where your base damage is locked in upon casting. Oh, and remember that "Sword Enhancement Spell Damage" gear? It still applies to Enspells II, though it's applied as a constant and does not eventually double like the base damage does.
Based on these descriptions, I've come up with two equations to use to determine which Enspell would give more damage (for simplicity's sake, this is assuming the Red Mage is an idiot and does nothing but constantly swing his weapon for the duration of the spell; realistically, some time's going to be taken out for your regular spellcasting duties):
First-tier Enspell = (Enhancing Magic Skill/20 + 5 + Sword Enhancement Spell Damage Bonus) * Duration of the spell in seconds * 60 /Delay of your weapon * Average number of attacks per attack round
(Note that the Delay number, as listed on a melee weapon, is 60 times the number of seconds in between attack rounds. For example, a Sword with 240 delay has an attack round once every four seconds. The number is rarely this precise, though, as there are also Swords with 233 Delay, 223 Delay, and so on.)
Enspell II = ((3/2)(EMS/20 + 5) + SESDB) * (EMS/20 + 5) * Delay/60 + (2(EMS/20 + 5) + SESDB) * (Duration of the spell in seconds * 60/Delay - (EMS/20 + 5) * Delay/60)
This calculation is a crapload less intuitive, and is based on quite a few assumptions, so let me explain it: for a number of attack rounds (Delay/60) equal to your base Enspell damage (because it's increasing at a rate of 1 more damage per attack round and will eventually double, EMS/20 + 5), you will be dealing an average amount of damage equal to (3/2)(EMS/20 + 5) + SESDB. This is a mathematical shortcut. To justify it with an example, add together 18 + 36, then 19 + 35, and so on, and then divide each value by two. This should result in 18 instances of the value 18 * 3/2, and should work with any integer, assuming that the Red Mage produces enough Enspell II hits to reach the damage cap. As with the first-tier Enspell equation, it's also assuming an Enhancing Magic Skill > 200, but following the "enough hits to reach the damage cap" assumption, there is also the implicit assumption that your Enhancing Magic Skill isn't too high for that to be possible at your current Delay (Enhancing Magic Skill <= 20(Duration in seconds * (60/Delay)^2 - 5), or else the second term (the amount of damage dealt while you're at the Enspell damage cap) ends up negative, when in reality the smallest this term can actually be is 0).
Also, I just realized that I forgot to account for Haste when I first did this. Crap. Well, Haste speeds up attacks by 1/(1 - Haste%), so I suppose you'd multiply each term in the equations by that as well to get the total amount of potential damage added for the duration of the spell. This is kind of inexcusable, too, since by the time one could cast Enspell II, one can and should cast Haste as well. Ah, well, I guess this hypothetical Red Mage is an idiot anyway for assuming they're not going to be casting anything while they have the Enspell effect . . .
As I mentioned earlier, "Sword Enhancement Spell Damage" bonuses are applied independently, but I managed to work out how one can separate this term from the equations entirely and then add it where appropriate:
Total damage added from SESDB = SESDB * Duration of the spell in seconds * 60/Delay (* Average number of attacks per attack round, if this is a first-tier Enspell)
What I wanted to do first of all was figure out how much additional Enhancing Magic Skill from an Enhancing Skill set would require to be lost from a first-tier Enspell to outdo an Enspell II, assuming a single-attack weapon with 240 Delay and only Brutal Earring providing 5% Double Attack (Average number of attacks per round = 1.05) and no Sword Enhancement Spell Damage gear. I would roughly estimate this to favor the Enspell II, personally, but my goal was to use something simple yet realistic enough. (Though, again, it's not at all that realistic to assume 0% Haste, but anyway . . . ) Base durations of all Enspells is 3 minutes, though by the time one can use Enspells II one can also take advantage of Composure, which triples the duration of self-cast White and Black Enhancing Magic (and also provides +15 melee Accuracy, which is kinda nice, but I'm also implicitly assuming 100% hit rate . . .). So Duration in seconds = 540. I trying solving for one variable once, but all that was good for was revealing the failure of my Enspell II damage calculation at high values of Enhancing Magic Skill (which, by the way, for 240 Delay and no Haste, EMS < 575). So I made it two variables, X being Enhancing Magic Skill in a dedicated Enhancing Magic Skill gear set, and Y being Enhancing Magic Skill in your regular melee set, and attempted to solve for X - Y. Here's what I got:
(X/20 + 5)(540 * 60/240)(21/20) = (3/2)(Y/20 + 5)(Y/20 + 5)(240/60) + 2(Y/20 + 5)(540 * 60/240 - (Y/20 + 5)(240/60))
From here, I managed to work (with pencil, paper, and calculator) that monster to . . . this monster:
Y^2 - 19600Y/37 - 912500/37 = -14175X/37
As one might guess, I was in the middle of completing the square when I posted my frustration to Twitter, where akkeresu told me "ohai, here's some Wolfram Alpha for ya." (http://bit.ly/bwWb6V) Which was awesome! Except it told me that I basically ran in to a dead end, and cannot solve for X - Y. =/
I mean, next time I'd probably input different different theoretical values, maybe getting rid of the Brutal Earring Double Attack to simplify things, and adding at least 15% Haste for realism, but I can tell right away that it wouldn't alleviate the problem of me not being able to find what I'm looking for. =/ Too many dang variables, I swear (not even mentioning the ridiculous "100% hit rate and constantly swinging" assumptions)!
So, because the aforementioned fox-cat wanted to know precisely what I was looking for, well, now ya know. =/
Ultimately, I want to know, even if you have to assume an arbitrary 240 Delay, how high the multi-attack rate has to be and/or how great the difference in Enhancing Magic Skill counted has to be in order for a first-tier Enspell to completely outdo an Enspell II. If the parameters needed were easily attainable for a high-end Red Mage (and didn't gimp regular melee damage to heck), then Enspells II would, in fact, be "brokenly horrible." =/
Oh, and mandatory addendum: "melee Red Mage" bashing will not be condoned. I'm well a-freaking-ware there are times and places where it isn't applicable and what "I'm invited for." This is neither the time nor the place for that. The equations assume the Red Mage is a moron, anyway, so just assume he's soloing and pull your panties out of your butt, seriously. This is strictly for inquiring minds who want to know.
Addendum the second - I'm also further doubting my math on some of this, so if you see any potential conceptual mistakes, please let me know . . .
EDIT FOR LONGEVITY:
Enspell = (Enhancing Maic Skill/20 + 5) * Amount of Use in Seconds per Spell * 60/Delay * 1/(1 - Haste%) * Average Attacks Per Rround + tSESD * AAPR
Enspell II = (3/2)(EMS/20 + 5) * (EMS/20 + 6) + (2[EMS/20 + 5]) * (AUSS - [EMS/20 + 6] * Delay/60 * [1 - Haste%]) * 60/Delay * 1/(1 - Haste%) + tSESD
tSESD = "Sword Enhancement Spell Damage" * AUSS * 60/Delay * 1/(1 - Haste%)
To find added DPS for Enspell or tSESD, simply omit Duration In Seconds. Because Enspell II first builds up before staying at a constant rate, the total DPS of the spell will vary with how much it is used per cast. The total damage equation for Enspell II also assumes the second term to not be negative - that is, enough attacks connect to reach the damage cap. Naturally, all DPS values are subject to hit rate, but your results may also vary with enemy magic resistance as well as the current weather. As always as a Red Mage, use the appropriate spell as per your discretion.
I've basically given up on trying to find some simpler equation for determining which spell is "better." The best one can do now is plug in appropriate values in to each equation to determine what the situation at hand calls for.
First-tier Enspell = (Enhancing Magic Skill/20 + 5 + Sword Enhancement Spell Damage Bonus) * Duration of the spell in seconds * 60 /Delay of your weapon * Average number of attacks per attack round
Enspell II = ((3/2)(EMS/20 + 5) + SESDB) * (EMS/20 + 5) * Delay/60 + (2(EMS/20 + 5) + SESDB) * (Duration of the spell in seconds * 60/Delay - (EMS/20 + 5) * Delay/60)
Total damage added from SESDB = SESDB * Duration of the spell in seconds * 60/Delay (* Average number of attacks per attack round, if this is a first-tier Enspell)
Here's the variables:
EMS = ?
spellTime = ?
Delay = 240
avrAR = ?
We'll take the equation to this...
totalSESDB = SESDB * Duration of the spell in seconds * 60/Delay (* Average number of attacks per attack round, if this is a first-tier Enspell)
Enspell1 = (EMS/20 + 5 + SESDB) * spellTime * 60 /Delay * avrAR
Enspell2 = ((3/2)(EMS/20 + 5) + SESDB) * (EMS/20 + 5) * Delay/60 + (2(EMS/20 + 5) + SESDB) * (spellTime * 60/Delay - (EMS/20 + 5) * Delay/60)
And
Your change in damage between Enspel2 and Enspell1 (E2 - E1 = deltaD) is the change in damage.
So, give me some valid values to work with; I might stab at this more with a fresh mind later. XD
Also, Haste. Kind of a big deal in this game, and there's zero reason why a meleeing Red Mage shouldn't have at least 15% at all times from casting the Haste spell on themselves (though with gear, 20% seems actually realistically on the low side, but 5/4 might be a friendlier number to work with than, say, 4/3, or 25% Haste).
Furthermore, the purpose of the totalSESDB equation was to demonstrate that the only difference it has between the two Enspells is a proportional increase with avrAR in a first-tier Enspell.
And I think (buried somewhere in those walls of text . . . ) I had mentioned that spellTime was 540 seconds.
If I were to attempt a more accurate rewrite each of the first three equations (with similar assumptions being made for the purpose of showing each spell's maximum damage potential) using your terms, they'd look like:
totalSESDB = SESDB * spellTime * 60/Delay * 1/(1 - Haste%)
Enspell1 = (EMS/20 + 5) * spellTime * 60/Delay * avrAR * 1/(1 - Haste%) + totalSESDB * avrAR
Enspell2 = (3/2)(EMS/20 + 5)(EMS/20 + 5) + 2(EMS/20 + 5)(spellTime * 60/Delay * 1/(1 - Haste%) - (EMS/20 + 5) * Delay/60 * (1 - Haste%)) + total SESDB
Even then, I'm still kinda iffy on the Enspell II damage formula concerning time in the second term, namely whether ((EMS/20 + 5) * Delay/60 * (1 - Haste%)), a term that's still technically expressed in seconds, should be subtracted from spellTime before or after getting divided by the delay. Actually, I'm starting to think it's before now, which would make the second term look more like
2(EMS/20 + 5)((spellTime - (EMS/20 + 5) * Delay/60 * (1 - Haste%)) * 60/Delay * 1/(1 - Haste%))
As for variables, Delay, Haste%, and spellTime are all being arbitrarily assigned 240, 20%, and 540, respectively (the 540 is arbitrary for, like I said, assuming that the Red Mage is going to spend that entire time doing nothing but swinging, which is, as I said, actually rather stupid). That leaves, as intended in the original post, only EMS and avrAR in question (and remember that EMS is implicitly assumed to not be high enough to make the second term in E2 negative!).
Also, didn't think about subtracting one equation from the other to get deltaD. On one hand, that would've made quite a few things easier, but on the other hand it probably would've let quite a few more mistakes go through unchecked, so in a way I'm kind of glad I gave this as much thought as I did. At least I can say I got as accurate a result as I could! XD
Well let me take a crack at this.
En1 damage = BaseEn Damage*Delay+ MultiHit / 540sec
En2 damage = BaseEn+Increasing damage up to double*Delay / 540 sec
Ok, well personally at 269 Ehn Skill, my Base Dmg is going to be... 18.45
Using my 2 normal swords, I'd have a combined delay of 462 minus dualwield reduction (15%) 393. Including haste and hastegear at 31%, making my delay roughly 272. So I attack twice every 4.5 seconds. (Actually, I use Joyeuse and Brutal Earring but I'll add those in as theoretical damage last).
4.5x =540, 540/4.5 = x = number of attack rounds, 120
120x2(dualwield) x18 En1 damage= 4320 damage... hmm that seems abit high...
Assuming the Joyeuse kicks in 45% of the time, and the Brutal Earring 5%, 6 extra attacks main hand, 60~ on Joyeuse. 240+66 = 306*18 = 5508
Ok, so yeah lots of damage from En1.
En2 Starting at 18 increasing up to double (36), using the same main hand sword, 238, at 31% haste, 165 Delay/ 60, 1 attack ever 2.7 sec.
2.7x =540, 540/2.7 = 200 Attack rounds
18+19+20+[...]+36(200-18) **takes 18 attack rounds to max out**
477+36(182)
477+6552 = 7029 *eep*
So unless I really screwed something up, yes En2 are indeed horribly broken, and that 18 base damage wasn't even in my Enhancing magic setup... this is assuming perfect accuracy and not needing to cast anything during a 9 minute time span, 2 things that typically do not happen.
Estimating Attack rounds in Ehn Gears...
590+ 40(180-20)
590+ 40(160)
590+ 6400
6990
Not too far off... I'll come back to this once I figure out how much Enh I can stack and how much Haste I'd have to sacrifice.
However, please don't take offense if I question some of your calculations.
Most glaringly, why did you use a different Delay value for your Enspell II calculation? It should be constant between the two, with first-tier Enspells getting a multiplier for your average number of attacks per round. If you used the same Dual-Wield setup with roughly 272 "net" delay, your total Enspell II damage would look more like
19(3/2)(18) + 36(120-19)
19 * 27 is a shortcut for counting all the values from 18-36, including the first "36" hit itself. My own equation would've made this term 18(3/2)(18), so I guess I need to fix that with a + 1 in there to correct my off-by-one error.
513 + 3636 = 4149 total damage
I'd also need to give slight correction to your first-tier Enspell calculations, namely how you factored in multi-attack. Dual-wield lets you swing twice, so you'd need to figure out the average number of attacks per round for each hand, and then add them to get your total average number of attacks per round.
For the non-Joyeuse weapon, with Brutal Earring average attacks per round is simply 1.05. Joyeuse, however, is different: the Double Attack trait cannot activate when Joyeuse's "Occasionally attacks twice" effect activates, so you need to further sub-divide Joyeuse's average attacks per round in to percentages. Since Joyeuse activates about 45% of the time, multiply 2 by .45, and then multiply the other .55 by 1.05 for Brutal Earring.
1.05 + 2(.45) + 1.05(.55) = 1.05 + .9 + 0.5775 = 2.5275
Multiply that by 120 attack rounds and your 20 Enspell damage (from casting it in your Enhancing Magic set).
2.5275 * 120 * 20 = 6066 total damage
While this is obviously just a theoretical value begotten from a decimal number of attacks per round (or else it would be divisible by 20), this still tells us, as common sense would suggest, that Dual-Wield plainly favors first-tier Enspells. Let us recalculate for both for single-wielding Joyeuse (w/Brutal, why not) at 31% Haste. The net Delay between attacks rounds is roughly
224/60 * (.69) = 1 attack every 2.576 seconds
540 seconds/2.576 seconds/attack is roughly 209 attacks before Enspell wears off.
209 attack rounds * 20 Enspell damage * 1.4775 average attacks per round = 6175.95 total damage
Again a theoretical value, but one that suggests that first-tier Enspells are favored even more by single-wielding Joyeuse than by dual-wielding it with whatever you're dual-wielding it with.
For Enspells II:
19(3/2)(18) + 36(209 - 19) = 513 + 36(190) = 513 + 6840 = 7353 total damage
So, single-wielding even Joyeuse seems to favor Enspells II even more. Bonus points for being able to use an EX Sword WS sub, since you're not Dual-Wielding (I'm heavily partial to the idea of /PLD myself, Auto Refresh gogo!).
I heavily appreciate you sharing some practical data to work with, but ideally I'd find how many attacks per round and/or how much extra Enhancing Magic Skill it takes to get to the point where first-tier Enspells start out-doing Enspells II. I think I'll work on finding a deltaDamage equation, as Akkeresu suggested earlier.
I don't normally main hand Joyeuse(low damage, not great for WS) so I used the delay of my Machaera.
I think it may be possible with a Magian Multihit sword + Joyeuse to get near the damage output of En2 using En1, but that's alot of work for something that has already been achived.
In short, make a nice hybrid set with enough haste, acc and enh skill to do the job, get a High damage Low delay sword and use En2.
What I've ultimately concluded about the Enspells is that one basically has to plug in every applicable variable on a case-by-case basis to see which one is better for their specific instance. Since I intend to go after a Magic Accuracy-build Shamshir as my weapon of choice by the time I can even do Magians, I'll definitely want to be using Enspells II (not to mention how they let me capitalize further on the net Magic Accuracy Bonuses).
I was just perturbed at how a Red Mage I highly respect has said that Enspells II are "brokenly horrible." And after all the much more complicated math they put me through, I must say that I now agree. >:(