Heavily constrained optimization via SinglePeriodOptimization() question

[dlevanoni]

I'm setting up a fairly constrained optimization problem using SinglePeriodOptimization() and I'm getting back weights that aren't quite what I expected.

In particular, for an 11 asset universe:

1. The Var-Cov matrix is set to the identity matrix (i.e. DiagonalCovariance(sigma_squares=1.)
2. Linear TCost (i.e. StocksTransactionCost(a=constant))
3. Constraints are only min_weight and max_weight (i.e. MinWeights() and MaxWeights() with the min weights set to 0. and the max weights = 2. * benchmark)
4. Benchmark is a fairly non-equal vector (varies from a low of 0.04 to a high of 0.50 depending on the asset)
5. The market_data is a UserProvidedMarketData()...returns are actual asset returns (one day) as are the prices and the volumes are set to a large number (1e9). since I just need the dates and the weights (not translated to $ amounts)

[Note that I have a worry that using prices at 1. is a bad idea as it may force some odd trade size behavior...haven't dug into before asking here]

I can provide a lot more detail on the setup (both inputs and outputs, including example data), but if that's sufficient, here's what I'm concerned about.

When I look at the optimal weights, I get, for some of the very small markets (i.e. those with small benchmark weights and hence relatively small ranges) have very small negative values...which would be below the minimum weight constraint. Further, those assets with wide ranges (i.e. max-min) tend to hug the benchmark while those with tight ranges (i.e. small markets) tend to swing quite a bit.

I suspect I'm missing something...

[enzbus]

I guess you should use TransactionCost, StocksTransactionCost (bad name, on me) has default b=1 for the non-linear term, and has a small per-share cost; it was designed for use in the examples but definitely go with the base one for your usecase.

Once you drop that, you can avoid passing volumes and prices to the market data; use MarketSimulator, and add costs if you wish manually (you can add a transaction cost with the same parameter).

For the min/max weights you can use the min/max benchmark deviation constraints, which do the same thing.

If you have small numerical errors and are concerned about them, you can increase the target accuracy of the numerical solver CVXPY calls, they are listed in the "advanced" page of cvxpy.org; for example passing solver='SCS', eps=1e-12 to the single period constructor. In general interior point solvers tend to be more accurate, you might want to set solver=CLARABEL.

I suspect the main issue was the transaction cost term adding a dependence on volume. If these don't solve it, post a minimal script to reproduce 👍

[enzbus]

Another point: the weights plotted in the default plot are the open weights, not the target weights, so it's normal that they drift a bit from whatever constraint you have, because of market returns. Have a look at both the w and w_plus attributes of the result object.

[dlevanoni]

Thanks for that!

I'll switch to TransactionCost() and see how it goes. I was going straight to the Optimization() rather than through the simulator...since I have both pre- and post- trade processing to do for each iteration (some funny path dependent stuff in play). Hence I was just looking at the first item (u) of the results tuple from SinglePeriodOptimization()...

As a side note, switching to CLARABEL fails (even with default params)...but as I think CVXOPT is also an interior point method solver, that one seems to work for some reason. With verbose=true, CVXPY is just giving me a "solver failed" message from CLARABEL...not sure what the error is.

[dlevanoni]

By the way, I use cvxportfolio instead of just building it via cvxpy directly as I do like the clarity of the cvxportfolio implementation, particularly given that other folks may be using my code and explaining QP to developers isn't my favorite thing in the world...

[dlevanoni]

I'll try to build a simple reproducible script and post it...

[dlevanoni]

python 3.9.18
cvxportfolio 1.3.1
cvxpy 1.3.1
pandas 1.4.4
numpy 1.21.5
scipy 1.10.1

[dlevanoni]

In the meantime...I'm putting in a final step of "inbounding" (rounding isn't technically correct) since the deviations are tiny it can just push the offending values to the boundary (min or max) and then adjust the remaining portfolio weights (if necessary) on a proportional basis.

One thing I still note is that for assets where the bounds (max-min) are relatively tight it tends corner more, whereas for markets where those markets where looser markets don't nearly as much...I'm assuming that the resulting portfolio is still optimal, but I would have expected equivalent usage of the active range for all markets as the forecasts are generally similarly Gaussian...

That could still be an impact of TCs, so I may try running it with TCs set to zero and see if the behavior changes...

[enzbus]

Just tried this. So, anything below, say 1e-12, I usually accept as numerical error. Solvers are very complex codes and do a lot of roundings and heuristics internally. The reason why OSQP gives you small negative values on the zeros, rather than small positive ones, has to do with the algorithm it uses, at the end of its loop it does a linear system solve using the active set it found. Interior point solvers should give you always feasible solutions, with the numerical errors going inside the feasible set rather than outside (although, some transformation might invalidate that).

The errors anyway do not accumulate; every period when you solve you get that small $\epsilon$ error on the bounds that you impose for that period.

I was able to run your script using all solvers including CLARABEL, on:

Linux x86_64
Python 3.11 (Debian testing)
Numpy 1.26.4 (Debian testing)
Scipy 1.13.1 (Debian testing)
fresh pip versions of other packages.

I know CLARABEL is a bit newer and might have transient issues, especially on non-Linux platforms.

A simple fix, to me, seems to hard limit after you solve

    # hard thresholding on weight limits
    opt_wgt = np.minimum(opt_wgt, data['max'])
    opt_wgt = np.maximum(opt_wgt, data['min'])

I'm not sure about other issues, here. Dependency of the error on how tight the limits are seems unlikely, although it might have to do, again, with scaling heuristics. If you're concerned, definitely go with more mature solvers. CVXOPT is a good choice.

[enzbus]

PS I didn't know, CVXOPT is available in Debian, sudo apt install python3-cvxopt, another good reason for picking it.

[dlevanoni]

I updated the packages (slight update to CVXPY and CLARABEL) and it seemed to move things to a better place...so using CLARABEL now with default tolerances seems to work...I added in an inbounding set of rules (had to be a little more complex that just the np.min() or np.where() versions...), so that should work as a "belt and suspenders" approach.

It turns out that the transaction cost is a bit impact than I would have guessed (even though they're set quite low)...setting them to zero cleaned up quite a bit of the issues as well. To that end, it occurred to me that the initial loop through (from zero to the first portfolio) has particularly large trades and so the TCs might be too high a hurdle just for that one time...so I added in some logic for the first day's trading to assume zero TCs and then for all subsequent days assume the normal TCs...that seemed to work well and as expected. I've done that in the past, so it made sense to try it here...