nxfilter losing users, policies regularly

[sywong2000]

Anyone experiences the polices, users settings are losing regularly?
After sometime not sure why nxfilter lost all of the polices, users. It can still log the activities, but the policies are all gone.
Any clues?

[support200]

What's 'losing user'? You created a user on NxFilter GUI and it's removed after sometime?

[sywong2000]

Yes that's correct. The users are gone, and the custom polices are gone too like sometimes after 24 hours or 36 hours. The strange thing is that it seemed the log can log with the user names, but then since all policies are gone it only applies the Default policy. I believed the free time settings are lost too. At least they do not shows in the GUI.

[support200]

We don't have such functions to delete users or policies on its own. You don't do AD importation synch regularly? Do you have any auto backup and restore system?

When it happens. You can try to restore it form your auto backup files. We keep auto backup files in /nxfilter/backup with date. If they appear again then you lost them after that. Before you do that, download your current backup on 'Config > Backup'. If you send them to us, we can compare them.

[sywong2000]

No I didn't do AD importation sync. I do have the backup. How can I restore the backup? I'm running mxfilter in a raspberry pi 3

[support200]

In the zip file, there's config.h2.db file. You stop NxFilter and then you can copy it into /nxfilter/db.

How do you stop or restart your raspberry pi? Just power off and on? That's very dangerous for database system. You could have some damage on NxFilter database. You better shutdown your system before you plug out.

[sywong2000]

I got it, let me try to restore it. thank you
No I keep my RPI runs 24x7. I wouldn't power it off without issuing sudo shutdown (but I don't shutdown the nxfilter server)

[support200]

Yeah, I was just talking about power off thing as I remember there's no proper switches for rasperberry pi when I tested it like 3 years ago. People might think they can just plug it out. You don't need to shutdown NxFilter before you shutdown your system. Its shutdown script will be called during the shutdown process.

[sywong2000]

Unfortunate the issue is still happening. I tried to inspect the nxfilter log by checking if there're any LocalTalkie.notifyUpdate. But the policy lost without any of it.
I added back the policy manually and the attached is the config.h2.db (which is after my manual added back). I'm not sure if you can see any transactions history. If cannot, I will wait for the next happening (usually it happens everyday) and I'll upload the DB again for your checking. Thank you very much

[support200]

No. There's no transaction history. You have 10 backup files in /nxfilter/backup. There's date in the filename. You know when it happened last time. Send me the one before it happened and the one after it happened. I can compare them to see if there's anything less.

[support200]

There's only one policy except Default. So, you lost Roxxxx policy and you restored it manually?

[sywong2000]

yes, that's correct.

[support200]

Why don't you create more items like users, policies, whitelist rules and other changes. To see if they all get wiped out. So, it's like everything you created have gone or just some of them? What about other changes like Upstream DNS Server settings. Do they get preserved?

And when it has gone, do you knot lose other data like traffic data?

[support200]

What's your system? Do you run it on Docket or some VM? When you lose your policy was it restarted?

[sywong2000]

I think now I'm able to see what's going on (but still dunno why) - let me try to outline the events I've encounter today. I attached the log and the DBs for inspection.
I'm running in a RPI4 box, as a system service (not docker). version is v4.6.0.3

Last night I setup a policy named "Test"
at 13:27 today I've found the policy I setup last night was lost again
at 13:31 I perform a systemctl stop of the nxfilter service, then I do a DB restoration by unzipping auto-2022031601.zip which is a autobackup in the morning (01:00), copying config.h2.db into the /nxfilter/db folder
at 13:33, I start the nxfilter service again
at 13:35, I go into the web admin and inspect and found the "Test" policy reappear again. I can see the DNS request log is running good and able to log with the "Test" policy
at 14:12, I've found the policy is lost again.

I've found an entry at 14:01 with LocalTalkie.notifyUpdate on the policy table.
and inspect all the DNS query, after 14:02 there're no more entries logged with the "Test" policy, which matches the fact that somehow the policy was lost at 14:01
So the update at 14:01 that would probably wiped the "Test" policy. I'm pretty sure that I didn't operate the UI to remove the policy.

Attachments:
nxfilter_truncated.zip contains the nxfilter.log
auto-2022031601.zip was the autobackup at 01:00 that the policy was still there.
config-20220316141013.zip contains the DB after the policy was lost


I have two raspberry Pi - one is 4 and another is 3B+. At the beginning I was running the 3B+ as master and the RPI4 as slave. All runs fine since the beginning of the year. I did regularly update the versions by dpkg -i the deb packages.

Since sometime in early March, I started experiencing lost in both the policies and users. And I also remember losing free-time settings as well. The version was 4.5.x.x
I attempted to re-install the packages again and clear the /nxfilter directories. But the problem keeps happening.

And then I stopped running nxfilter in the 3B+, and move the service to the RPI4 which is a separate hardware.
On the RPI4, I still experiences policies lost, but users profile seemed to be fine.

In both machines, the DNS queries history were intact, no lost.

I'm going to create more policies, users and rules and try to do more tests and find out.