You need to calibrate with sources of known intensities at known distances.
You should expect that your webcam is non-linear in its brightness response. If it is a cheap webcam, there is also a good possibility that there are notable non-linearities across the imaging sensor, and there is an even better chance that the lens does not focus linearly.
You should also not trust the higher-end or lower-end readings: at the lower end you will get quantum noise producing false pixels, and at the upper end brightnesses past a certain point will saturate the sensor. Don't be surprised if the saturation effects start below full brightness.
Remember to let everything warm up for at least 4 hours (24 preferred) and remember to re-do the calibrations against all the environmental temperatures the equipment might be exposed to, and remember to include a digital thermometer as one of the sensors so that you know which temperature profile to use.
And keep in mind that unless you know the exact distance to the object you are trying to measure the brightness of, you will only get relative brightness and not absolute luminosity.
In summary: if you want the results to be more than a vague approximation, you probably want to use better equipment.