Keeping cards in a filtered deck

If I create a filtered deck for all the cards that I got wrong in a day, but I never finish the deck (i.e. empty the deck) and keep the cards in there. Does this mess with the SRS algorithm, because my cards never return back to their home decks?

No, that won’t mess with the algorithm. The only thing that will be impacted is those individual cards. If they stay in the Filtered deck past their next due date, then when they are returned to their home deck, they will be overdue (assuming that you have reschedule-based-on turned off for this deck?). The algorithm will deal with that just fine though – as long as you grade your answers accurately.