Construction of objective function in optimization problem
Many optimization procedures require a single objective function. How to construct an efficient objective function in optimization problem is a critical. Sometimes we have multiple goals, how to create a single function to integrate all the goals together? Sometimes, even though we have a single goal, but we have specific aspect of requests for the goal. There are many factors to impact the efficiency of objective functions.
Recently I encountered a problem when I trained a neural network model to fit a single target with 4 correlated input variables. To find a good solution for the given neural network structure is not a easy thing when I adopted traditional objective function – root mean square error (RMSE) between the observed and predicted target values. The real problem is the observed target values are between 0 and 1. The estimated RMSE will be very small, especially after square. One simple solution to solve the problem is to multiply the observed and predicted target values by 100 and then calculate RMSE. A more sophisticated and general solution is to construct a objective function based on properties of the target variable.
Here is an examples. We keep the original RMSE objective function as one component of the final composite objective function. Since we want the predicted values are as close as possible to the observed values, we want all data points in a scatter plot (observed vs predicted) aligning in 1:1 line. Therefore, we want the slope and R2 to be 1 and the constant to be 0 if we get the linear regression equation. So, the slope and R2 can be another component of the final composite objective function. So the final objective function should look like:
RMSE + (Slope – 1) ^2 + (R2 – 1) ^2
We want the minimize the value. Even though the function is complicated, good solutions were achieved much fast than the previous RMSE.
In addition, we can consider the ratio between observed and predicted value as an criteria. Our goal is to minimize the ratio. So the objective function could be:
Sum(abs(Predicted – Observed) / Observed)
All in all, we can construct simple or sophisticated objective functions for a specific optimization problem. What you have to do is to analyze your problem thoroughly and consider all possible solutions.
