If I stop the oils but keep inulin will I see a relapse?

Yes, I believe you will. The olive and fish oils help stabilize the omega-3:omega-6 balance which is critical to maintaining lower levels of inflammation.