Instead of having a "mastery of Element X" passive skill, the character has "Element Affinity" which is sum of points invested in skills of that element.
Example:A wizard with L10 fireball L1 lightning and L3 Sparkle and L12 Inferno has 10+12=22 affinity to Fire and 1+3=4 Affinity to Lightning.
Elemental Affinity can provide resistance/absorb to element used against player and/or boost damage for the skill using the element.
It works as if all points in skills of the element create synergy for it.
Same concept could be applied to parts of skill tree which are specific to
specific weapon class(staffs,swords, bows) creating e.g. Bow Affinity which boost bow skills and reduces damage from Bow attacks on player.