Polygon Backtesting does not include dividends

[grzesir]

When Backtesting using the polygon data provider, dividends are not included in the calculation.

Steps to replicate: run a Backtest using a symbol that only has dividends (eg. USFR). Notice that the Backtest does not include the dividends

[grzesir]

@jimwhite I figure you would find this relevant since you're working on the polygon stuff now, and this can make a pretty big difference in backtests. It's on my list of things to do but I won't be able to get to it for a few months given my workload.

[jimwhite]

Yeah, I reported this. Wanted to let you know and folks should be aware. I have family stuff that will keep me busy for a while so I don't know when I'll get to it. If someone wants to work on this it should be very straightforward to extend what I did in #397 to options.

[grzesir]

I believe @davidlatte and I had a discussion about how to implement this a few months ago. From what I remember it actually was not a lot of work, only a few files needed to be changed.

By the way @jimwhite let me know if you'd like free access to our pro plan for the work you have been putting into lumibot. We have an algorithm library that I think you would really appreciate and a very active community with regular events.

[jimwhite]

Oh, sorry I didn't actually read this issue, I thought you were cc'ing me on the issue I reported because of the timing.

I will take you up on the free Pro plan because I would like to chat with folks using lumibot. I have plans for many improvements to lumibot but I am preoccupied with family stuff atm. Hopefully I'll get back to this within the next few weeks.

[grzesir]

@jimwhite sounds great. I agree, family first. I just had a baby 3 weeks ago myself, so I definitely understand. I wish you and your family all the best.

I'm happy to give you a complimentary pro plan. You will love our community, algorithm library and other resources. To set it up, I think it would be easiest if we jump on a call. You can book a time on my calendar here: https://calendly.com/lumi-rob/30min

[jimwhite]

I'm looking in the code and I see various functions for getting dividends and strategy executor calling Strategy._update_cash_with_dividends but I don't find any data sources that actually provide dividend values, nor are there tests for dividends.

A few questions:

1. Are there any backtesting configurations that track the portfolio value correctly wrt dividends? If so, which one?

2. Are there any data sources that populate Bars with dividend values? It looked to me as though not only does the Polygon Pandas data source not get dividends but neither do Alpaca, Tradier, or IBKR. What I see in that code are stubs filling in zero for dividend values.

3. Why is the portfolio cash value updated by this strategy executor logic? Shouldn't this be maintained by the broker, and in the case of backtesting, by backtesting/backtesting_broker.py? IOW, don't we want backtesting to follow the same logic as trading when interacting with the broker?

lumibot/lumibot/strategies/strategy_executor.py

Line 756 in 7a5df2d

self.strategy._update_cash_with_dividends()

lumibot/lumibot/strategies/strategy_executor.py

Line 768 in 7a5df2d

if not has_data_source or (has_data_source and self.broker.data_source.SOURCE != "PANDAS"):

[jimwhite]

Answer to q1 from Discord: Dividend calculations should work for Yahoo backtesting.

A first step here would be to implement a test demonstrating that.

[grzesir]

To answer your questions: 1. I believe yahoo finance Backtesting does this. 2. Data sources to brokers such as alpaca, ibkr, etc should not process dividends. The brokers themselves will update the cash value, we don’t need to touch it. Only Backtesting data sources should handle dividends. 3. Maybe moving it to Backtesting broker would be appropriate since only Backtesting brokers should be doing this. Robert Grzesik 347-635-3416

…

On Sat, Apr 6, 2024 at 3:41 PM Jim White ***@***.***> wrote: Dividend calculations reportedly work for Yahoo backtesting. A first step here would be to implement a test demonstrating that. — Reply to this email directly, view it on GitHub <#402 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIYQK3WNV54AW43V73I5T3Y4BFVHAVCNFSM6AAAAABEZZCCDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBRGE3TMNBWGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>